Method and system for speech signal enhancement with reduced latency - Patents.com

Abstract

A system and method for low latency speech signal enhancement is provided. An input mixture of speech signals is divided into a sequence of overlapping frames using a first sliding window method. The first sliding window method comprises a first window function having a first width associated with the window of the corresponding frame and a shift length associated with the shifting of the window of the first sliding window method. Each frame is then processed using a first DNN, a frequency domain causal linear filter and a second DNN to generate a final enhanced overlapping frame for each of the processed frames. The final enhanced overlapping frames are then combined using a second sliding window method, the second sliding window method being associated with a second window function having a second width less than the first width and a shift length identical to the first sliding window method.

Description

[Technical field]

[0001] The present disclosure relates generally to audio signal processing, and more particularly to methods and systems for low latency enhancement of audio signals. [Background technology]

[0002] Audio signal processing is a technological concept with a variety of applications, each with its own set of requirements. Audio signal processing techniques that work well in one application area may not work well in a different application area. For example, in a closed environment, it may be important to use audio signal processing to reduce the effects of reverberation in audio signals, but in applications such as hearing aids, reducing latency in processing audio signals is more important than reducing reverberation. This is because in a closed room, an audio signal, such as a speech signal, propagates through the air and may be reflected by walls, floors, ceilings, and other objects in the room before being captured by a microphone. Reverberation is the multipath propagation of a speech signal from a source or speaker to a receiving end, such as a microphone. Such speech reverberation occurs when sound reflects off surfaces in the environment. Parts of the sound may be absorbed by those surfaces, which causes multiple attenuations of the speech signal. The reflection and absorption of sound by those surfaces may generate multiple attenuated and delayed copies of the speech signal. These multiple attenuated and delayed copies can degrade the quality of the speech and hinder the performance of an Automatic Speech Recognition (ASR) system or any speech / audio processing system, for example, an ASR can produce inaccurate output due to the degraded speech quality of the audio input.

[0003] On the other hand, in many application scenarios such as teleconferencing and hearing aids, low latency speech enhancement and speaker separation are more important. In modern learning-based speech signal processing systems, this is managed by using causal neural network blocks such as unidirectional Long Short-Term Memory (LSTM), causal convolutions, causal attention layers and causal normalization layers. Although these systems are considered to be causal and real-time due to the underlying causal Deep Neural Networks (DNNs), these systems are more precisely frame-online, with the amount of look-ahead determined by the frame length used in the speech signal processing. One major approach that may enable sample-level causal processing is by using models such as Wave Net, which are essentially deep generative models of the raw speech waveforms.

[0004] However, the effectiveness of these known systems in dealing with noise and reverberation in a sample-causal setup is unclear, and at run-time, such models need to perform a feedforward at each sample, making them computationally very expensive, sometimes more than necessary.

[0005] Some audio signal processing approaches, such as Short-Time Fourier Transform (STFT) and time-domain approaches, generally split the audio signal into overlapping frames before processing. In these approaches, the audio signal is multiplied by a window function that associates a processing window of a predefined length or width, which converts it into multiple overlapping frames. The processing latency in these approaches is comparable to the window length due to the use of overlap-add in the signal resynthesis and the execution time to process one frame. However, these approaches have the disadvantage of high processing latency in real-time applications such as hearing aid and teleconferencing applications. As an example, in a typical STFT-based system with a 32 millisecond (ms) window and 8 ms hop size, a new frame comes in for processing every 8 ms, so a frame-online DNN-based system meets the latency requirement if the processing of each frame can be completed within 8 ms on a given processor. The total latency in such an example is 40 ms, which is not ideal. In fact, in applications such as hearing aid design, the required algorithm latency is at most 5 ms, which is much lower than the latency in this example. Such low latency constraints require new designs and significant modifications to existing voice processing algorithms.

[0006] To overcome the above problems in the case of hearing aids, some known solutions have used 4ms windows for window functions and 1ms hop sizes in some STFT-based hearing aid studies. However, using 4ms windows results in much lower frequency resolution than 32ms windows. It is well known that in deep learning-based TF masking, for the same hop size, longer windows, and therefore higher frequency resolution, usually result in better oracle separation than shorter windows. Also, in multi-microphone processing, using short windows may be particularly poor at reliably capturing inter-channel phase patterns, because the windowed signals may be too short to exhibit significant signal delays. However, in these previous solutions, it was observed that short window lengths, and therefore low resolution, do not lead to accurate STFT-based speech signal processing.

[0007] However, some time-domain signal processing approaches, such as Conv-TasNet, which uses a linear encoder to generate a representation of the speech waveform optimized to separate individual speakers, can use significantly shorter windows and hop sizes for ultra-low latency separation. Furthermore, Conv-TasNet can leverage DNN-based end-to-end optimization to learn a set of basis over a small window of samples for the encoder and decoder, respectively, to replace traditional STFT and inverse STFT (iSTFT) operations. In this case, separation is performed in the encoded space, and the decoder is used for overlap-add-based signal resynthesis. They achieve good separation performance in mono anechoic speaker separation tasks, but less impressive performance compared to frequency-domain approaches in reverberant and multi-microphone scenarios. Also, the basis learned by TasNet is not narrowband. It is not straightforward how to directly combine TasNet with traditional STFT-domain separation algorithms such as beamforming and weighted prediction error (WPE), which rely on narrowband assumptions and can provide robust separation and dereverberation through per-frequency processing. One way to combine them is by iterating TasNet using significantly shorter windows along with STFT-domain beamforming using regular longer windows. In order to use the TasNet output to compute signal statistics for STFT-domain beamforming, the time-domain signal must first be resynthesized before extracting the STFT spectrum for beamforming.

[0008] Similarly, to apply TasNet to the beamforming results for post-filtering, we must apply iSTFT to obtain the time-domain signal before feeding them into TasNet. However, this iterative procedure will gradually accumulate algorithmic latency because the overlap-add algorithm is used multiple times in TasNet and iSTFT. Other time-domain approaches that use a typically large window size, such as 32 ms, have the same problems as STFT-based systems.

[0009] Therefore, there is a need to overcome the above problems, and more specifically, to develop a method and system for low latency processing of audio signals while overcoming reverberant conditions and non-stationary noise in the environment. Summary of the Invention

[0010] It is an object of some embodiments to develop methods and systems for efficient, accurate and low latency based processing of audio signals and the like.

[0011] Some embodiments are based on the understanding that STFT domain methods for speech signal processing, such as speaker separation tasks, generally use large window lengths, such as 32 ms, and overlap between consecutive frames of 75%. However, this incurs a latency of at least 32 mn, because the overlap-and-add algorithm used in inverse STFT (iSTFT) is also performed based on a window size of 32 ms. However, this inherent latency needs to be reduced for applications such as hearing aid design, which requires a latency as low as 5 ms. To reduce this inherent latency, some embodiments of the present disclosure provide a novel dual window size approach for STFT domain low latency speech signal processing, such as for use in hearing aid design or speaker separation tasks.

[0012] Some embodiments are based on the recognition that while STFT extraction uses a regular window size, overlap-add in iSTFT uses a much smaller window size. This approach can leverage a reasonably high frequency resolution for separation while simultaneously keeping latency low. Based on this novel STFT representation, single or multi-microphone complex spectrum mapping of the speech signal is applied for frame-online separation, and one or more DNNs are trained to predict the RI components of the target speech from the Real and Imaginary (RI) components of the received speech signal, such as the input acoustic mixture of the signal. Since the STFT method is inherently narrowband, the predicted RI components by the first DNN are used to perform frame-per-frequency online frequency-domain linear causal filtering, such as beamforming and convolutional prediction-based dereverberation. The beamforming and dereverberation results are then used as additional features of the second DNN for post-filtering. The inclusion of frequency-domain linear causal filtering, in this case beamforming and dereverberation, between the two DNNs dramatically improves performance and can be easily integrated with complex spectral mapping, whereas such inclusion is not possible in the time-domain approach without incurring processing delays.Evaluation results on the SMS-WSJ, a benchmark dataset for speech separation, demonstrate the effectiveness of the proposed approach.

[0013] Some embodiments disclosed herein reduce algorithmic latency for audio processing to 4ms and 2ms while still achieving high performance.

[0014] Some embodiments are based on the understanding that an audio signal may correspond to clean speech that exhibits spectro-temporal patterns. Such spectro-temporal patterns are unique patterns exhibited in the time-frequency domain and can provide useful clues for reverberation reduction. Some of these patterns may originate from the structure of the speech signal itself, but some patterns may also correspond to a linear filter structure of reverberation (i.e., reflections of sound waves) specific to the physical space in which the recording is made, including all objects, structures or entities present in that space, as well as the location of the source speech signal and the receiver, such as the microphone, that records the signal. This linear filter structure can be used to account for the signal originating from the source signal at the microphone location and reflections of that signal from walls and surfaces of objects or people in the space, which represents the effect of reverberation on the input signal as a linear convolution of the input signal and the Room Impulse Response (RIR). The input signal is the original source signal, also known as the dry source signal. The Room Impulse Response represents the effect of the space and everything in that space on the input signal. An estimate of the RIR between a source and a receiver location can be recorded in a physical space such as a room, for example by firing a short-duration time-domain signal, an impulse sound (e.g., a blank gun or a balloon burst), at a source location in the room and recording the resulting signal at the receiver location. The impulse excites the room and produces a reverberant impulse signal that can be used to estimate the RIR.

[0015] The reverberation of a dry source sound signal that would be played back at the same source location and recorded at the same receiver location can then be modeled by convolving the dry source signal and the estimated RIR.

[0016] Moreover, such linear filters can be exploited as regularization to improve the dereverberation process. For example, linear filters as regularization prevent overfitting of the model of the dereverberation process to the training data. Some embodiments are based on the recognition that a linear filter structure can be utilized in combination with linear prediction and deep learning for single-channel and multi-channel reverberant speaker separation and dereverberation tasks. To that end, deep learning techniques supported by convolutional prediction can be used for dereverberation in environments with noise signals, reverberation of speech signals, etc. This convolutional prediction is a linear prediction method for speech dereverberation in reverberant situations that relies on source estimates obtained by DNNs and utilizes a linear filter structure between the source estimates and the reverberant version of the source signal in the observed input signal.

[0017] To obtain the source estimate, a DNN is trained in the time-frequency domain or the time domain to predict the target speech from the reverberant speech, which corresponds to the target direct path signal between the source and a receiver such as a microphone. This approach can exploit prior knowledge of the speech patterns.

[0018] Previous studies have also attempted to utilize some form of linear filter structure to perform dereverberation. For example, weighted prediction error (WPE) may be used to dereverberate speech signals. The WPE method computes an inverse linear filter based on variance normalized delayed linear prediction. The computed linear filter is applied to past observations of a reverberated and possibly noisy mixture input signal to estimate late reverberation of a target source signal in the mixture input signal from past observations of reverberation for dereverberation. This estimated late reverberation is subtracted from the mixture of acoustic signals received from various sources to estimate a target speech signal in the mixture of acoustic signals. Also, in some embodiments, the filter may be estimated using a time-varying Power Spectral Density (PSD) of the target speech signal. The PSD is the distribution of the power of the signal over the frequency range of the signal. Such a linear filter may be iteratively estimated using WPE in an unsupervised manner. However, the iterative procedure of WPE for filter estimation may lead to suboptimal results and may be computationally expensive.

[0019] To overcome the above deficiencies of WPE, the iterative procedure for filter estimation can be replaced with a DNN-based WPE (DNN-WPE) approach. DNN-WPE uses the magnitude estimated by the DNN as the PSD of the target speech signal for filter estimation. However, DNN-WPE cannot reduce early reflections because it requires a strict non-zero frame delay to avoid trivial solutions and cannot have a mechanism for utilizing the phase estimated by the DNN for filter estimation. Also, DNN-WPE may not be robust to interference caused by noise signals. For example, DNN-WPE may estimate a filter that relates a noisy past observation to a noisy current observation, thereby limiting the filter estimation accuracy. Also, DNN-WPE directly uses the linear prediction result as its output, which may result in partial or minimal reduction in reverberation.

[0020] To that end, it is also an object of some embodiments to estimate a basic filter for approximating or modeling the RIR. In some exemplary embodiments, the RIR may be estimated based on a linear regression problem solved per frequency in the time-frequency domain. The filter estimate that models the RIR may be used to identify delayed and attenuated copies of the input signal for dereverberation of the speech signal.

[0021] In some cases, the mixture of acoustic signals may be received from a single channel, such as a single microphone, or from multiple channels, such as an array of microphones. Each different channel measures a different version of the mixture of acoustic signals. In that case, two or more DNNs may be trained to estimate the target direct path signal of the reference channel or the received signal in each channel. The training of each DNN may be based on complex spectral mapping in one or more channels, and these DNNs are trained to output an estimate in the time-frequency domain of the target direct path signal in one or more channels such that the distance between the estimate in the time-frequency domain of the target direct path signal in one or more channels and the reference is minimized.

[0022] In the case of an array of microphones, a beamforming output can be obtained. The beamforming output can be obtained based on statistical information calculated from one or a combination of the first estimate of the target direct path signal at each microphone of the array of microphones and the dereverberated mixture of the target direct path signal. The beamforming output can be input to a second DNN to generate a second estimate of the target direct path signal for each of the multiple speakers. Additionally or alternatively, the beamforming output and the dereverberation results can be used as additional features of the second DNN to perform better separation and dereverberation tasks, which can be considered to be tasks related to audio signal enhancement processing or speech enhancement processing.

[0023] To that end, some embodiments provide systems and methods for speech signal enhancement processing that use different window sizes for analysis at the input (STFT) and synthesis at the output (iSTFT). For example, some embodiments provide a 32 ms window and 1 ms hop size for the STFT, and a 4 ms window and 1 ms hop size for the overlap-and-add in the iSTFT. The 32 ms window for the STFT provides 4 ms look-ahead and 28 ms look-back.

[0024] To this end, the speech signal is processed in a frame-online configuration using two DNNs with a linear filtering module in between and an asymmetric windowing method with different window sizes at the input (analysis) and output (or synthesis) sides. In one example, the speech signal is synthesized by performing an inverse discrete Fourier transform (iDFT), after which the first 28 ms of the speech signal waveform is discarded, and then a synthesis window is applied to perform overlap-add based on the last 4 ms of the waveform in each frame. This dual window size approach using different window sizes for the STFT and overlap-add has several advantages. First, the use of a longer window for the STFT provides a higher frequency resolution, thus providing more estimated filters (or mask values) per frame, resulting in finer speech signal separation. The higher frequency resolution can also better exploit the speech sparsity property in the TF domain for separation. Second, the use of a longer window for the STFT can capture more reverberation in each frame, potentially leading to improved dereverberation. Third, using a larger window for the STFT may lead to improved spatial processing since inter-channel phase patterns may be more stable and more pronounced for longer signals. Fourth, the basis of the STFT is narrowband in nature, allowing the use of the DNN output, i.e. the estimated target RI components, to compute signal statistics for conventional frequency domain beamforming and dereverberation, which can then be used as additional features for another DNN to better predict the target speech.

[0025] Therefore, some embodiments provide a computer-implemented signal enhancement method for processing an input mixture of audio signals. The signal enhancement method includes receiving the input mixture of audio signals via an input interface, the input mixture of audio signals may be a single-channel audio signal or a multi-channel audio signal. The input mixture of audio signals is then split into a sequence of input overlapped frames using a first sliding window method, the first sliding window method having a first window function having a first width associated with a window of a corresponding frame and a shift length associated with a shifting of the window of the first sliding window method. In some embodiments, the first width corresponds to a window length, and the shift length corresponds to a hop size of the sliding window method. The shift length is less than or equal to 20% of the first width associated with the window. Each frame of the split overlapped frames is then processed using a first deep neural network (DNN) to generate an enhanced overlapped frame, the enhanced overlapped frame comprising a corresponding enhanced frame for each processed frame among the overlapped frames. Further, a frequency domain filtering output is generated for each frame of the enhanced duplicated frames. The frequency domain filtering output is then processed using a second DNN to generate a corresponding final enhanced frame for each frame of the enhanced duplicated frames. The final enhanced duplicated frames are then combined using a second sliding method, and the second sliding method is associated with a second window function having a second width less than the first width. To that end, a higher first width provides high frequency resolution at the input side of the audio signal processing, while a smaller or shorter second width provides low latency at the output side.

[0026] To that end, some embodiments provide that the second width is a multiple of the first width. Furthermore, the second sliding window method is also associated with a sliding window hop size or shift length that may be identical to the shift length of the first sliding window method.

[0027] In some embodiments, the second width is at least equal to twice the shift length.

[0028] In some embodiments, each of the first window function and the second window function is an asymmetric window function.

[0029] Some embodiments provide that the first width is at least equal to 32 milliseconds (ms), the second width is at least equal to 4 ms, and the shift width is at least equal to 2 ms.

[0030] In some embodiments, the frequency-domain filtered output for each frame of the enhanced overlapping frames is a causal-linear filtered output generated by a causal-linear filter. In some embodiments, the causal-linear filter is a Multi-Channel Weiner Filter (MCWF).

[0031] In some embodiments, generating the frequency domain filtered output for each frame of the enhanced overlap frame includes generating a beamforming output as the frequency domain filtered output for each frame of the enhanced overlap frame, and the beamforming output is fed into the second DNN to generate the corresponding final enhanced frame for each frame of the enhanced overlap frame.

[0032] To that end, some embodiments provide for generating a first estimate of an intermediate representation for each frame using the first DNN. Further, an estimation of a filter that models a room impulse response (RIR) is performed for the first estimate of the intermediate representation for each frame. This filter is used to obtain a reverberant-reduced mixture of the intermediate representation for each frame by removing the result of applying the filter to the intermediate representation for each frame from the mixture of the received audio signal. The reverberant-reduced mixture is then fed into the second DNN to generate a second estimate of the intermediate representation for each frame. The second estimate of the intermediate representation for each frame is then output via an output interface.

[0033] In some embodiments, the filter comprises a linear filter based on convolutional prediction techniques.

[0034] In some embodiments, the received multi-channel audio signal includes speech signals from multiple speakers, and the first DNN generates a plurality of outputs, each output of the plurality of outputs including the first estimate of the intermediate representation for each frame of a speaker from the plurality of speakers, whereby the multi-channel signal corresponding to an input mixture of the audio signals is received from an array of microphones connected to the input interface.

[0035] In some embodiments, receiving the multi-channel audio signal from the array of microphones further includes obtaining a beamforming output based on statistical information calculated from one or a combination of the first estimate of the intermediate representation for each frame at each microphone of the array of microphones and the reverberant reduced mixture of the intermediate representation for each frame, and feeding the beamforming output into the second DNN to generate a second estimate of the intermediate representation for each frame.

[0036] In some embodiments, the first DNN is pre-trained (offline) to obtain the first estimate of the intermediate representation for each frame from an observed mixture of acoustic signals. The pre-training of the first DNN is performed using a training dataset of a mixture of acoustic signals and a corresponding reference target direct path signal in the training dataset by minimizing a loss function. The loss function is a distance function defined based on real and imaginary (RI) components of the first estimate of the intermediate representation for each frame in the first time-frequency domain and the RI components of the corresponding reference target direct path signal in the first time-frequency domain, a distance function defined based on a magnitude obtained from the RI components of the first estimate of the intermediate representation for each frame in the first time-frequency domain and a corresponding magnitude of the reference target direct path signal in the first time-frequency domain, a reconstructed waveform obtained from the RI components of the first estimate of the intermediate representation for each frame in the first time-frequency domain by reconstruction in the time domain, and the reference target direct path signal. a distance function defined based on the RI component of the first estimate of the intermediate representation for each frame in the time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain; and a distance function defined based on the magnitude obtained from the RI component of the first estimate of the intermediate representation for each frame in the time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the corresponding magnitude of the reference target direct path signal in the second time-frequency domain, or a combination thereof.

[0037] Various embodiments provide a signal enhancement system for processing an input mixture of audio signals, the signal enhancement system comprising an input interface configured to receive the input mixture of audio signals, the input mixture of audio signals being at least one of a multi-channel audio signal or a single-channel audio signal. The signal enhancement system further comprises a memory storing computer-executable instructions and a processor configured to execute the computer-executable instructions. The computer-executable instructions are configured to divide the received input mixture of audio signals into a sequence of input overlapping frames using a first sliding window method, the first sliding window method comprising a first window function having a first width associated with a window of a corresponding frame and a shift length associated with a shifting of the window of the first sliding window protocol, the shift length being less than or equal to 20 percent of the first width associated with the window. The computer executable instructions are further configured to process the segmented sequence of overlapping frames using a first deep neural network (DNN) to generate enhanced overlapping frames, the enhanced overlapping frames comprising a corresponding enhanced frame for each of the processed frames in the input overlapping frames, generate a frequency domain filtered output for each frame of the enhanced overlapping frames, process the frequency domain filtered output for each frame of the enhanced overlapping frames using a second DNN to generate a corresponding final enhanced frame for each frame of the enhanced overlapping frames, and combine the final enhanced overlapping frames using a second sliding window method, the second sliding window method being associated with a second window function having a second width less than the first width and a shift length the same as the first sliding window method.

[0038] Further features and advantages will become more readily apparent from the following detailed description taken in conjunction with the accompanying drawings.

[0039] The present disclosure is further described in the following detailed description with reference to a number of drawings as non-limiting examples of exemplary embodiments of the present disclosure, in which like reference numerals represent like parts throughout the several views of the drawings. The drawings shown are not necessarily drawn to scale, with emphasis instead generally being placed on illustrating the principles of the embodiments disclosed herein. [Brief description of the drawings]

[0040] [Figure 1A] FIG. 1 is a block diagram of a signal enhancement system according to some embodiments of the present disclosure. [Figure 1B] 1B is a detailed block diagram of functions performed by the signal enhancement system of FIG. 1A for processing the audio signal shown in FIG. 1 according to one embodiment of the present disclosure. [Figure 1C] FIG. 2 illustrates an exemplary window function according to an embodiment of the present disclosure. [Figure 1D] FIG. 2 illustrates an example of an asymmetric window function according to an embodiment of the present disclosure. [Figure 2A] FIG. 2 is a schematic block diagram of the steps performed during splitting an input audio signal using a first sliding window method according to an embodiment of the present disclosure. [Figure 2B] FIG. 13 is a schematic block diagram of steps performed during combining enhanced audio signals using a second sliding window method according to an embodiment of the present disclosure. [Figure 2C] FIG. 1 is a schematic block diagram of the steps performed while enhancing a segmented audio signal using a deep learning based architecture according to an embodiment of the present disclosure. [Figure 3A] FIG. 2 is a schematic block diagram of an audio signal enhancement process according to an embodiment of the present disclosure. [Figure 3B]FIG. 2 is another schematic block diagram of another audio signal enhancement process according to an embodiment of the present disclosure. [Figure 3C] FIG. 1 illustrates a signal enhancement system having two DNNs according to an embodiment of the present disclosure. [Figure 3D] FIG. 1 is a schematic diagram of an architecture representation for enhancement of an audio signal according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating corresponding network architectures for each of two DNNs used in a signal enhancement system according to an embodiment of the present disclosure. [Figure 4 Continued] FIG. 1 is a schematic diagram illustrating corresponding network architectures for each of two DNNs used in a signal enhancement system according to an embodiment of the present disclosure. [Diagram 5] FIG. 1 is a block diagram of an audio processing system according to an embodiment of the present disclosure. [Figure 6] FIG. 1 illustrates a use case for enhancement of a speech signal in a teleconferencing setup, according to some exemplary embodiments of the present disclosure. [Figure 7] FIG. 2 illustrates a use case for enhancement of a speech signal in a hearing aid, according to some exemplary embodiments of the present disclosure.

[0041] While the above drawings set forth embodiments disclosed herein, other embodiments are contemplated, as noted in the description. The present disclosure presents exemplary embodiments by way of representation and not by way of limitation. Numerous other modifications and embodiments may be devised by those skilled in the art that fall within the scope and spirit of the principles of the embodiments disclosed herein. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0042] In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to one skilled in the art that the present disclosure can be practiced without these specific details. In other instances, the apparatus and methods are shown only in block diagram form to avoid obscuring the present disclosure. It is contemplated that various changes may be made to the function and arrangement of elements without departing from the spirit and scope of the disclosed subject matter as set forth in the appended claims.

[0043] As used in the specification and claims, the terms "for example," "for instance," and "such as," as well as the verbs "comprising," "having," "including," and other forms of these verbs, when used in conjunction with a list of one or more components or other items, should be construed as open-ended, meaning that the list should not be considered to exclude further components or items. The term "based on" means based at least in part on. Furthermore, it should be understood that the phraseology and terminology used herein are for purposes of description and should not be considered limiting. Any headings used in this description are for convenience only and do not have any legal or limiting effect.

[0044] In the following description, specific details are set forth to provide a thorough understanding of the embodiments. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For example, systems, processes and other elements in the disclosed subject matter may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other instances, well-known processes, structures and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments. Additionally, like reference numbers and names in the various drawings refer to like elements.

[0045] Although most of the description has been given using speech as the target source, the same methods can be applied to other types of audio signals.

[0046] Speech signals are increasingly processed using deep learning methodologies based on the use of DNNs, especially in speech signal enhancement tasks such as reverberation modeling, speaker separation, and accurate sound source identification. Some application scenarios of speech signal enhancement, such as teleconferencing and hearing aids, require low-latency speech enhancement. The present disclosure provides a method and system for performing such low-latency speech signal enhancement, especially speech enhancement, by using STFT-based signal analysis and resynthesis in conjunction with a deep learning framework. The method and system disclosed herein provide dual window size based speech signal enhancement with different window sizes at the input and output sides of a speech signal processing system. The input side window has a long length, which provides the advantage of high resolution and high accuracy during analysis. The output side window has a short length, which provides low latency of processing and synthesis at the output side. Furthermore, dereverberation and beamforming operations are used during the signal enhancement process described in various embodiments disclosed herein. These and other advantages of the present disclosure will be more apparent by the following description of the drawings.

[0047] FIG. 1A is a block diagram 100a of a signal enhancement system 104 for speech signal processing using deep learning-based speech signal enhancement according to some embodiments of the present disclosure. The signal enhancement system 104 is configured to receive an input mixture of speech signals 102(Y) as an input. The input mixture of speech signals 102 may be a single-channel speech signal or a multi-channel speech signal. The input mixture of speech signals 102 includes a target speech signal, which is a signal of interest in the mixture of speech signals. For example, in a room with multiple speakers, a listener may be speaking to one particular speaker, and there may also be other sounds in the room, such as the sound of an air conditioner. In this case, the speech signal corresponding to the one particular speaker that the listener is speaking to is the target speech signal. The examples outlined herein are merely for illustrating exemplary scenarios for the application of the signal enhancement system and should not be construed as limiting the scope of the present disclosure in any manner.

[0048] The signal enhancement system 104 performs processing of the input mixture of audio signals 102 for signal enhancement using deep learning. Such signal enhancement is performed to ensure providing a low-latency but high-quality enhanced signal 106 (S) at the output of the signal enhancement system 104. Such signal enhancement is performed by the signal enhancement system 104 using deep learning techniques, such as using Artificial Intelligence (AI) and Deep Neural Networks (DNN). The enhanced signal 106 is used in applications such as hearing aid devices and teleconferencing systems where real-time high-quality and low-latency audio signal processing is required.

[0049] To that end, the signal enhancement system 104 performs various functions that are further illustrated in the diagram of FIG. 1B.

[0050] 1B is a detailed block diagram of functions performed by the signal enhancement system 104 for audio signal processing shown in FIG. 1A according to one embodiment of the present disclosure. To that end, the signal enhancement system 104 comprises various components for performing various functions. In particular, the signal enhancement system 104 comprises an input / output (I / O) interface 108 configured to receive inputs at the signal enhancement system 104 and transmit outputs from the signal enhancement system 104. For example, the I / O interface 108 includes an input interface for receiving the mixture of audio signals (Y) 102 and an output interface for generating the enhanced signal (S) 106.

[0051] The signal enhancement system 104 also includes a memory 110 for storing computer executable instructions comprising instructions corresponding to various functions performed by the signal enhancement system 104. These instructions are executed by a processor 112, which is configured to perform the various functions as a result of execution of the computer executable instructions. For example, the computer executable instructions include instructions for configuring the processor 112 to perform one or more operations comprising splitting the input mixture 102 of received speech signals into a sequence of input overlapped frames using a first sliding window method having a first window function with a first width (114), processing the sequence of input overlapped frames using a deep neural network (DNN) architecture to achieve AI-based or deep learning-based speech signal enhancement, and combining the enhanced overlapped frames using a second sliding window method having a second window function associated with a second width less than the first width (118). The DNN architecture may include a first DNN and a second DNN. The first DNN processes the sequence of split input duplicate frames to generate enhanced duplicate frames. The enhanced duplicate frames are then passed to a frequency domain filter to generate a frequency domain filtered output for each frame of the enhanced duplicate frames after processing all frames up to the current frame (using all past knowledge of the frames up to the current time). The frequency domain filtered output for each of the processed enhanced frames is then passed to a second DNN, which generates a corresponding final enhanced frame for each frame of the enhanced duplicate frames. These final enhanced frames are then combined (118) using a second sliding window method. Each of these functions is described in detail in the embodiments disclosed herein.

[0052] As is commonly known, voice and speech signals are enhanced using a time-frequency domain approach that performs a phase resynthesis technique for enhancing the voice signal. However, with the advancement of deep learning, speech enhancement can take advantage of the high accuracy and low latency offered by time-frequency domain approaches based on deep learning. For this purpose, the voice signal is first transformed into the time-frequency domain, typically using known techniques such as STFT and other time-domain approaches that split the signal into overlapping frames with a moderately large hop length before processing.

[0053] For this purpose, the audio signal is multiplied by a window function and then a Discrete Fourier Transform (DFT) is applied to each frame of the overlapping frames. Such a window function, also called a sliding window or tapering function, is a function in which the amplitude (of the multiplied signal) gradually tapers off smoothly to zero towards the edges. Thus, each frame occupies a different period and the resulting STFT shows the spectral content of the signal in each corresponding period. By moving the sliding window, the spectral content of the signal over different time intervals can be observed. The STFT is thus a function of time and frequency that shows how the spectral content of the signal evolves over time. A complex-valued two-dimensional array called the STFT coefficients stores the results of the windowed Fourier transforms. The magnitudes of the STFT coefficients form the magnitude time-frequency spectrum and the phases of the STFT coefficients form the phase time-frequency spectrum. Thus, the window function provides the tapering effect by providing a mathematical function that is zero outside a selected interval and is usually symmetric around the center of the interval and usually has a maximum near the center. As a result, when another function (such as an audio signal) is multiplied by a window function, the product has a value of zero outside the interval, and the part to the left is where the two functions overlap, providing a "view through the window".

[0054] Some common examples of such window functions include Hann window function, Hamming window function, Blackman-Harris window function, generalized cosine window function, etc. For example, FIG. 1C illustrates an exemplary Hann window function 120. The Hann window function 120 is a bell-shaped curve with amplitude 122 and sample 124 values ​​represented along the Y-axis and X-axis, respectively. As can be seen from FIG. 1C, the amplitude 122 of the Hann window function 120 is maximum at the center and gradually tapers off towards the edges 126 and 128. Most window functions are similar bell-shaped curves. The type of window function used and the length of the window function, also referred to as the window width of the window function, affect the time-frequency resolution of the resulting STFT. In general, the window length is determined by the underlying signal to be analyzed. The window length should be small enough so that the windowed signal block / frame is essentially fixed over the window interval, yet large enough so that the Fourier transform of the windowed signal block provides a reasonable frequency resolution. For slowly evolving signals, the window length should be large, and for fast evolving signals, the window length should be small. For example, in speech signal processing, a time-domain window length of 25 ms is common.

[0055] Another parameter associated with the window function is the hop length, also referred to as the shift width or shift length, which determines the amount of overlap between adjacent frames or blocks of a signal. If the shift length is less than the window width, there is overlap between the frames. However, if the shift length is greater than the window width, there is no overlap between adjacent frames. Furthermore, during signal recombination, a convolution-and-add operation is used to sum all the samples from the overlapping frames in a particular time interval to obtain a final recombined signal sample. Furthermore, latency is defined as the delay caused in processing a signal to reach an output. In general, latency is equal to the sum of the window length and the execution time to process one frame by using the convolution-and-add during signal recombination. In one example, in a typical STFT-based system with a 32 ms window and a hop size of 8 ms, the overlap is 75%, so each quarter of the window is processed by four different windows. Therefore, the latency in processing a signal sample is 32 ms, since all four windows in which the sample falls must be processed in order to recreate the signal sample at the output.

[0056] Furthermore, in some embodiments, the latency in processing a signal or a sample of a signal is defined as processing latency, which is the sum of algorithmic latency and hardware latency. Algorithmic latency is defined as the latency caused by algorithmic reasons (such as overlap-add), and the computation time required to process one frame is defined as hardware latency. In the above example, the processing latency is 32ms+8ms=40ms, which may be too large for some applications such as hearing aids, leading to a poor listening experience.

[0057] To overcome this challenge, the signal enhancement system 104 splits a signal, such as the input mixture of speech signals 102, into frames and uses different widths for windows to reconstruct a signal from these frames, such as the enhanced signal 106. In this way, the signal enhancement system 104 can use a DNN-based architecture with linear filtering between the two DNNs to provide more information to process the split input mixture of speech signals 102. This is possible due to the larger input window size and the shorter output window size.

[0058] In the previous example, the delay is 32 ms because all four frames of the input signal need to be combined to recreate a quarter of the window of the input signal, which results in a delay of 32 ms. However, some embodiments disclosed herein are based on the recognition that it is not necessary to wait for all four frames to be processed and then use all four of them to reconstruct each sample of the output / enhanced signal 106. Instead, a smaller number of frames, for example only two frames, can be used during the overlap-and-add operation to perform the signal resynthesis to provide the enhanced signal 106. To this end, some embodiments are based on the recognition that the width of the sliding window method for combining frames should be less than the width of the window for dividing the multi-channel input (audio) signal 102 into overlapping frames.

[0059] To that end, the signal enhancement system 104 is configured to split 114 the input mixture of speech signals 102 into a sequence of input overlapping frames using a first sliding window method and combine the final enhanced frames using a second sliding window method. The first sliding window method is associated with a first window function having a first width. The second sliding window method is associated with a second window function having a second width. The second width is less than the first width. The first window function and the second window function may be selected from any of the above window functions, including but not limited to a Hann window function, a Hamming window function, a Blackman-Harris window function, a generalized cosine window function, a rectangular window function, a triangular window function, and the like. To that end, each of the first sliding window method and the second sliding window method may be any of a symmetric or asymmetric window function.

[0060] In some embodiments, the first window function and the second window function may both be asymmetric window functions, which are not symmetrical bell curves in shape, but rather some of the signals multiplied by them are more tapered and some are less tapered. To that end, if a window function is considered to be equivalent to a weighting function, an asymmetric window function is one in which the weighting function and the weighting coefficients are not symmetrical about the origin.

[0061] 1D illustrates an example of an asymmetric window function 130 that is not symmetric about the origin and where some samples are given more weight than others. Regardless of the type of window function used, symmetric or asymmetric, the signal enhancement system 104 uses two different window sizes at the input and output sides to achieve high frequency resolution and low latency in enhancing a signal, such as the input mixture of audio signals 102. The concept of two windows of different sizes is further explained below in conjunction with FIGs. 2A and 2B.

[0062] 2A is a schematic block diagram of steps performed during splitting 114 an input mixture of audio signals 102 using a first sliding window method using a first window function 200, according to an embodiment of the present disclosure. The first sliding window function 200 may include any of the window functions described above. The first window function 200 may be symmetric or asymmetric without departing from the scope of the present disclosure.

[0063] The first window function 200 is applied, such as by multiplication, to the input mixture 102 of audio signals, so that the input mixture 102 of audio signals is divided into a sequence of overlapping frames, i.e. frame 202 in interval t, frame 204 in interval t+1, frame 206 in interval t+2, and frame 208 in interval t+3. The length of each window of the window function 202, also known as the first width 210, is selected to have high frequency resolution at the input side. Furthermore, the hop length of the first sliding window method, also referred to herein as the shift length 212, which is related to the shifting of the window between the overlapping frames, is also selected so that there is an overlap between consecutive frames. To that end, the shift length 212 is selected to be equal to or less than 20% of the first width 210, in order to have at least 80% overlap between consecutive frames. For example, if the shift length 212 is equal to 20% of the first width, frame 202 overlaps with frame 204 by 80%, and similarly for other overlapping frames.

[0064] In one example, the first width 210 is selected as being at least equal to 32 milliseconds (ms) and the shift length 212 is selected as being at least equal to 2 ms. The shift length 212 is therefore 2 / 32*100=6.25% of the first width 210, providing an overlap of more than 93% (30 / 32*100) between any two overlapping frames in the sequence of frames 202-208, ensuring high frequency resolution in the splitting (114) step performed by the signal enhancement system 104. The overlapping frames are then enhanced in a step of AI enhancement 116 (using a neural network architecture) performed by the signal enhancement system 104 (the enhancement is described later in this description), so that a corresponding final enhanced frame is generated for each overlapping frame 202-208. To that end, the processing of the split input overlapping frames is performed in a streaming manner. As the input mixture 102 of acoustic signals is received, it is split into a sequence of input overlapping frames, and the sequence of input overlapping frames is buffered. Then, once each frame, such as the current frame, is completed, it is passed to an AI Enhancement 116 module having a neural network-based architecture. The current frame is processed in combination with previously formed and buffered frames (in some configurations, it is simply the last frame, but typically it is several recent frames, and sometimes it may be all past frames) to generate a corresponding enhanced frame. These corresponding enhanced frames are also passed through a filtering network to generate a frequency domain filtered output for each enhanced frame in the same streaming manner as above. The AI ​​Enhancement 116 is then repeated on this filtered output to generate a final enhanced frame for each filtered output, again in a streaming manner.

[0065] The corresponding final enhanced frames are then combined using a second sliding window method, which is illustrated in FIG. 2B.

[0066] 2B illustrates a schematic block diagram of steps performed by the signal enhancement system 104 while combining 118 the enhanced training duplicate frames 216 using a second sliding window method with a second window function 214 according to an embodiment of the present disclosure. The second window function 214 may include any of the window functions described above. The second window function 214 may be symmetric or asymmetric without departing from the scope of the present disclosure.

[0067] The second window function 214 is applied, such as by multiplying the enhanced signal by the second window function 214, to provide an enhanced training frame 216, which comprises a sequence of enhanced overlapping frames, namely frame 218 in interval t, frame 220 in interval t+1, frame 222 in interval t+2, and frame 224 in interval t+3. The length of each window of the window function 214, also known as the second length 226, is selected to have low latency at the output side. The hop length of the second sliding window method, also referred to herein as the shift length 228, which is related to the shifting of the window between the overlapping frames, is also selected so that there is an overlap between consecutive frames. To that end, the shift length 228 is selected to be less than the second width 226. In practice, the second width 226 is selected to be at least equal to twice the shift length 228 to have a discernible overlap between consecutive frames among the enhanced overlapping frames 218-224.

[0068] In one example, the first width 210 is equal to 32 ms and the second width 226 is equal to 4 ms. Thus, the first width 210 is eight times the second width. Furthermore, the shift length is at least equal to 2 ms. In this example, the total latency of the combining (118) process would be 4 ms.

[0069] In some embodiments, the shift length 228 is the same as the shift length 212 of the first window function 200 .

[0070] Furthermore, in the above example, in the combining step 118, the total latency is 4 ms, which is very suitable for low latency applications such as hearing aids.

[0071] In some examples, instead of waiting for all four frames 218-224 in the combining step 118, only two of the overlapping frames are used to synthesize or generate samples of the enhanced speech signal at the output 106. This further reduces the latency at the output 106.

[0072] Such a speech signal enhancement system 104 can achieve ultra-low latency speech enhancement. In one example, a 32 ms window is used as a window of a first width 210 with a shift length 212 of 1 ms at the input side (such as for performing STFT), and a 4 ms window is used as a window of a second width 226 with a shift length 228 of 1 ms at the output 106 (such as for using iSTFT) for overlap-add operation. The 32 ms window for STFT has 4 ms look ahead and 28 ms look back. After obtaining the separated waveform in each frame by performing an inverse discrete Fourier transform (iDFT), the first 28 ms of the waveform samples are discarded, and then a synthesis window is applied at the output to perform overlap-add based on the last 4 ms of the waveform in each frame. This example is further illustrated in FIG. 2C.

[0073] 2C illustrates an example method 230 for performing speech signal enhancement using the signal enhancement system 104 using two different window sizes at the input 102 and output 106. The method 230 is configured to provide STFT-based speech signal enhancement with algorithmic latency as low as 4 ms or 2 ms. This is achieved in part by combining STFT-domain deep learning based speech enhancement with a traditional dual window approach that typically uses a long window length in the STFT in the splitting (114) step and a short window length in the overlap-add in the combining (118) step.

[0074] Referring to Figure 2C, an input mixture 102 of speech signals at an input is first split 114 into a sequence of input overlapping frames 202-208 using a first sliding window function 200 (shown in Figure 2A). In the example of Figure 2C, a first width 210 for performing an STFT using the first window function 200 is 16 ms, and a shift length 212 for the first sliding window method associated with the first window function 200 is 2 ms. In this case, the 16 ms first width window has 4 ms to look ahead and 12 ms to look back.

[0075] After obtaining the input overlap frames, the method 230 includes processing each frame of the input overlap frames 202-208 to perform the enhancement 116 using a neural network-based architecture to generate final enhanced overlap frames 202a-208a, with the final enhanced overlap frames 202a-208a comprising a corresponding final enhanced frame for each processed frame among the overlap frames. For example, the final enhanced frame 202a corresponds to frame 202 of the overlap frames after processing, the final enhanced frame 204a corresponds to frame 204 of the overlap frames after processing, the final enhanced frame 206a corresponds to frame 206 of the overlap frames after processing, and the final enhanced frame 208a corresponds to frame 208 of the overlap frames after processing. The processing in the enhancement 116 step further includes applying an analysis window to perform STFT at 116a, and then performing DFT for each frame of the input overlapped frames 202-208 at 116b to convert the time-frequency domain signal in each frame to frequency domain for neural network-based enhancement. The frequency domain frame is split using the first sliding window function 200 with a longer length, so that the neural network-based processing and enhancement can work better with more information per frame. After obtaining the frames separated by DFT, frame-by-frame online DNN-based processing is performed at 116c.

[0076] This frame-online DNN-based processing is further described in Figures 3A-3D. The frame-online DNN-based processing at 116c results in final enhanced frames, which are subjected to an iDFT operation to transform the frequency domain signals in each frame back to the time-frequency domain for reconstruction of the samples in each frame. This reconstruction is performed, for example, using a combining (118) step performed using a second sliding window function 214 having a second width 226 of 4 ms and a shift length 228 of 2 ms. The combining (118) step further includes, at 118a, removing samples in the first 12 ms of the enhanced frames 202a-208a, and then, at 118b, applying a synthesis window according to the second window function 214 to resynthesize samples from the overlapping frames 118c obtained after removing samples from the final enhanced frames 202a-208a. Then, at 118d, the duplicated frame 118c is added to reconstruct the sample in the output signal 106. As can be seen in the sample 118e of the output signal 106, the sample 118e corresponds to a subframe in time interval t and is obtained by performing the overlap operation 118c and the addition operation 118d using only two subframes, namely the subframe in time interval t and the subframe in time interval t+1. Thus, each sample can be synthesized at the output 106 side by simply waiting for the next frame in the sequence, which has a width of 4 ms in this example. Thus, in this example, the total algorithm latency is only 4 ms at the output 106 side. This is a significant performance improvement compared to the method with the same size window width, which would have a conventional algorithm latency of 16 ms due to the window width at the input 102 side being 16 ms.

[0077] Thus, using the method 230 in conjunction with the DNN-based enhancement in 116c has several advantages. First, using a longer window in the STFT in the splitting (114) step results in a higher frequency resolution, which means that there can be more estimated filters (or mask values) per frame, resulting in a more fine-grained speech signal enhancement. Also, the higher frequency resolution allows the speech sparsity property in the TF domain to be better exploited in the enhancement performed by the signal enhancement system 104. Second, using a longer first window with a first width 210 longer than the second width 226 allows more reverberation to be captured in each frame among the overlapping frames 202-208, which may lead to improved dereverberation during the frame-online DNN-based processing in step 116c. Also, it may lead to improved spatial processing, since inter-channel phase patterns may be more stable and prominent in longer signals. Third, the STFT basis is narrowband in nature, which means that the DNN output from step 116c can be used to compute a traditional frequency domain beamformer, the results of which can be used as additional features for another DNN to better predict target speech for speech processing tasks by the signal enhancement system 104.

[0078] Therefore, using the signal enhancement system 104 performing the method 230, the conventional dual window size can be adapted to reduce the algorithmic latency of STFT-domain deep learning-based speech enhancement. Secondly, the output from the DNN in the processing step 116c is used for frequency domain frame online beamforming, and in one example (described later), the beamforming result is sent to a second DNN for better signal enhancement. Compared with using the output from the time domain model for frequency domain beamforming, the method 230 does not incur algorithmic latency, because the two DNNs and the beamformer all operate in the complex TF domain.

[0079] In one example, the method 230 can achieve performance as good or better than the popular Conv-TasNet technique with algorithmic latency as low as 4 ms or 2 ms using similar computational complexity. Also, recent dual window size work on deep learning based speaker separation is said to be mono and uses real-valued mask estimation, only reducing algorithmic latency to 8 ms, which does not compare to the time domain model that makes the method 230 performed by the signal enhancement system 104 far superior.

[0080] To achieve ultra-low latency separation, a DNN model trained using a regular 32ms window and 8ms hop size is configured to run at inference with smaller hop sizes such as 1ms, 2ms and 4ms, as well as smaller output window sizes such as 2ms, 4ms and 8ms. In other words, an appropriate hop size can be selected at runtime without retraining the model, and an appropriate output window size can also be selected.

[0081] In one example, the method 230 uses a gain normalization mechanism within the DNN in the frame-online DNN-based processing in step 116c to handle random gain in the input 102 mixture of multi-channel audio signals, and it is observed that the performance of the signal enhancement system 104 is less sensitive to changes in input gain.

[0082] The frame-online DNN-based signal enhancement performed in step 116c will be described in detail in conjunction with FIGS. 3A-3E.

[0083] 3A illustrates an example of processing 116c of each of the split overlapping frames 202-208 using an architecture with two neural networks, according to one embodiment. The architecture of FIG. 3A illustrates two neural networks as an example implementation. However, any number of equivalent neural networks greater than two may be possible without departing from the scope of the present disclosure.

[0084]

number

[0085] The signal enhancement system 104 performing the signal enhancement method 230 performs the processing 116c of each frame of the overlapping frames 202-208, taking into account previously received frames or frames stored in memory during buffering up to the present time, and is configured to perform the processing 116c using the architecture of FIG. 3A, which comprises at least a first deep neural network DNN1232, a frequency domain online linear filtering component 234, and a second deep neural network DNN2236. Each frame of the overlapping frames 202-208, represented in the form of a mixture Y shown in equation (1), is input to the DNN1232. As a result, a corresponding intermediate representation 232a is generated for each frame. The intermediate representation 232a of each frame corresponds to an enhanced overlapping frame for each corresponding input overlapping frame. The intermediate representation 232a of the enhanced overlapping frame is then passed to the frequency domain online linear filtering component 234 to generate a frequency domain filtering output 234a for each intermediate representation 232a of the enhanced overlapping frames. Further, the frequency domain filtering output 234a is input to a DNN 2236 to generate a corresponding final enhanced frame 236a for each of the input duplicated frames 202-208. The final enhanced duplicated frames 236a are then combined 118 using a second window function 214 to generate the output 106 signal.

[0086] In one example, the intermediate representation 232a corresponds to using the DNN 1232 to generate a first estimate of the intermediate representation 232a for each frame of the split input overlapped frames 202-208. This intermediate representation 232a is used to estimate a filter that models a room impulse response (RIR) for the first estimate of the intermediate representation 232a for each frame. A reduced reverberation mixture is then obtained for each frame of the intermediate representation 232a by removing the result of applying the filter to the intermediate representation 232a for each frame from the input mixture 102 of the received speech signal. This reduced reverberation mixture is then fed into the DNN 2236 to generate a second estimate of the intermediate representation 232a for each frame, which may be output via an output interface, such as the I / O interface 108 of the signal enhancement system 104 shown in FIG. 1B.

[0087] The frequency domain filtering output for each frame of the enhanced overlapping frames may be a causal linear filtering output generated by a causal linear filter, such as an estimated filter based on convolutional prediction.

[0088] The architecture shown in FIG. 3A with two DNNs, i.e., DNN1232, DNN2236 and the frequency domain online filtering 234 component, is advantageous compared to past approaches based on the use of dual window sizes in speech signal processing applications. First, in the architecture and processing 116c of FIG. 3A, the output 232a from the DNN1232 is used in frequency domain frame online beamforming using the frequency domain online filtering 234 component, and the beamforming result 234a is sent to the DNN2236 for better enhancement (i.e., post-filtering). This beamforming and subsequent post-filtering approach provides clear improvement in speech signal enhancement applications compared to using only one DNN (i.e., not using beamforming and post-filtering).

[0089] Furthermore, the process 116c is based on the recognition that since the two DNNs (232 and 236) and the beamformer (234) all operate in the complex TF domain, the process 116c does not incur additional algorithmic latency. In contrast, the known time domain model cannot be directly combined with frequency domain beamforming without incurring additional algorithmic latency. This comparison proves that one advantage of performing ultra-low latency enhancement in the STFT domain using the architecture shown in FIG. 3A is the integration with frequency domain beamforming. In one example, beamforming is performed using linear filtering.

[0090] For example, the linear filter may be a causal linear filter that produces a causal linear filtered output.

[0091] For example, the causal linear filter may be a multi-channel Wiener filter (MCWF). An architecture using the MCWF is shown in FIG. 3B.

[0092]

number

[0093] In one example, the DNN 1232, the DNN 2236 and the MCWF module 234 are all designed to be frame-online, so that the two-DNN system shown in Figures 3A and 3B can be easily plugged into the AI ​​enhancement 116 portion performing the frame-online DNN-based processing 116c of the method 230 shown in Figure 2C, achieving speech signal enhancement with very low algorithmic latency. An exemplary architecture of the signal enhancement system 104 plugged into the two-DNN system of Figures 3A and 3B is shown in Figure 3C.

[0094] In some exemplary embodiments, the signal enhancement system 104 includes an input interface 108a that is part of the I / O interface 108 shown in FIG. 1B, a memory 110 that stores a first deep neural network (DNN1) 232 and a second deep neural network (DNN2) 236, a processor 112, and an output interface 108b that is part of the I / O interface 108 shown in FIG. 1B.

[0095] The input interface 108a is configured to receive a mixture of acoustic signals including a target direct path signal and reverberation of the target direct path signal. In some exemplary embodiments, the input interface 108a may be configured to connect to at least a microphone of the signal enhancement system 104 or an array of microphones of the signal enhancement system 104.

[0096] The processor 112 feeds the mixture of acoustic signals, including the target direct path signal 102 and reverberation, into the DNN 1232. The DNN 1232 outputs a first estimate of the target direct path signal, which corresponds to the intermediate representation 232a. In a multi-speaker scenario generating sound signals in an environment, the target direct path signal corresponding to each of the multiple speakers is estimated by the DNN 1232. The DNN 1232 may determine the corresponding estimates 232a of the target direct path signal for each of the multiple speakers one by one or simultaneously.

[0097] The first estimate 232a of the target direct path signal is used together with the mixture of received acoustic signals 102 to estimate a filter, such as filter 234, that models a room impulse response (RIR) for the first estimate 232a of the target direct path signal. The RIR is the impulse response of a room, e.g., the environment, between the sound source (e.g., a talker) and the microphone.

[0098] In some embodiments, the filter 234 that models the RIR may be output via the output interface 108b.

[0099] In some embodiments, the filter 234 modeling the RIR for the first estimate 232a of the target direct path signal corresponding to the input mixture 102 of acoustic signals is estimated such that when it is applied to the first estimate 232a of the target direct path signal, the corresponding result is closest to the residual between the mixture 102 of acoustic signals and the first estimate 232a of the target direct path signal according to a distance function. In some embodiments, the distance function may correspond to a weighted distance with a weight at each time-frequency point in the time-frequency domain. The weight may be determined by one or a combination of the received mixture 102 of acoustic signals and the first estimate 232a of the target direct path signal. In an exemplary embodiment, the distance function may be based on a least square distance. As a result of applying the filter 234 modeling the RIR, the dereverberated output corresponding to the result is closest to the residual between the mixture 102 of acoustic signals and the first estimate 232a of the target direct path signal according to the distance function. In one embodiment, this dereverberated output based on such a weighted distance function is obtained using a weighted prediction error method.

[0100] In another embodiment, the dereverberated output is obtained based on convolutional prediction, which is a linear prediction method for speech dereverberation in reverberant situations that relies on source estimates obtained by a deep neural network (DNN) and utilizes a linear filter structure between the source estimates and the reverberated version of the source signal in the observed input signal.

[0101]

number

[0102]

number

[0103] In some embodiments, the dereverberation output is generated using a weighted prediction error (WPE) method, which computes an inverse linear filter based on variance-normalized delayed linear prediction. The computed linear filter is applied to past observations of the reverberant and possibly noisy mixture input signal to estimate late reverberation of a target source signal in the mixture input signal from past observations of reverberation for dereverberation. This estimated late reverberation is subtracted from the mixture of acoustic signals received from various sources to estimate a target speech signal in the mixture of acoustic signals. Also, in some embodiments, the filter may be estimated using a time-varying Power Spectral Density (PSD) of the target speech signal, which is the distribution of the power of the signal over the frequency range of the signal. Such a linear filter may be iteratively estimated using WPE in an unsupervised manner.

[0104] When the result of applying the filter 234 to the first estimate 232a of the target direct path signal is removed from the mixture of acoustic signals 102 by any of the methods described above, a reduced reverberation mixture of the target direct path signal, such as 234a, is obtained. The reduced reverberation mixture of the target direct path signal 234a is provided as an input to the DNN 2236. The DNN 2236 generates a second estimate 236a of the target direct path signal. In one example, the second estimate 236a of the target direct path signal is output via the output interface 108b.

[0105] In one example, the second estimates for all frames are combined using the second window method described in FIG. 2B.

[0106] Similarly, the above process may be repeated for each of multiple speakers in the environment.

[0107] In one example, the architecture of FIG. 3C is used for dereverberation of a speech signal. For example, the output of the DNN 1232, such as the first estimate 232a of the target direct path signal of the input 102 and the reverberation-reduced mixture obtained using the filter 234, can be utilized for dereverberation of the speech signal. To that end, the first estimate 232a and the reverberation-reduced mixture obtained using the filter 234 can be input to the DNN 2236 to output a second estimate of the target direct path signal. The output generated by the DNN 2236, such as the second estimate, can be better than the output of the DNN 1232. This is because the input to the DNN 2236 (i.e., the first estimate 232a and the reverberation-reduced mixture obtained using the filter 234) is more refined than the input of the DNN 1232. For example, the first estimate 232a output by the DNN 1232 and the reverberation-reduced mixture obtained using the filter 234 may have less interference. When these first estimates 232a and the reverberation-reduced mixture obtained using the filter 234 with less interference are processed by the DNN 2236, the corresponding output 236a (i.e., the second estimate) may be better than the output of the DNN 1232. Therefore, the second estimate generated by the DNN 2236 may be used to perform another iteration of convolutional prediction to obtain a second filter and a reverberation-reduced second mixture, and the reverberation-reduced second mixture may be input to the DNN 2236 together with the second estimate to generate a refined output.

[0108] In some exemplary embodiments, the corresponding RIR of each speaker may be estimated by solving a per-frequency linear regression problem in the time-frequency or time domain. To that end, a filter that models the RIR may be used to identify delayed and attenuated copies of the speaker's target direct path signal in the input 102. The delayed and attenuated copies, which are repetitive patterns due to reverberation, may be removed from the received mixture of acoustic signals 102. To that end, the filter that models the RIR may be equivalent to a filter 234 that is applied to the first estimate 232a to output a result 234a. The result 234a may be closest to the residual between the mixture of acoustic signals 102 and the first estimate 232a of the target direct path signal based on a distance function, such as a weighted least squares distance function. When the result 234a is removed from the mixture of acoustic signals 102, a mixture with reduced reverberation is obtained.

[0109] The delayed and attenuated copies may correspond to late and early reflections of the target direct path signal, which may be identified from the RIR modeled by the filter estimate 232a.

[0110] 3D is a schematic diagram of an architecture representation 238 for enhancing a speech signal according to an embodiment of the present disclosure. As shown in FIG. 3D, the architecture representation 238 includes a DNN 1 232, a DNN 2 236, and a frequency domain online beamforming 234 component of processing. The architecture 238 corresponds to the processing step 116c shown in FIG. 3A.

[0111] The DNN 1232 provides a DNN-estimated target RI component in the form of a first estimate 232a, based on which the frequency-domain online beamforming component estimates an online multi-channel Wiener filter (MCWF) 234a to enhance the target speech received as the multi-channel audio input signal 102 (Y). The MCWF 234a is calculated per frequency, exploiting the narrowband property of the STFT. The beamforming result of such a filter usually does not score better in terms of enhancement metrics than the immediate DNN output, but can provide supplemental information to help the DNN 2236 obtain a better enhancement result.

[0112] Furthermore, the architecture of FIG. 3D (and equivalently FIG. 3A and FIG. 3B) can be used to easily integrate frame-online frequency domain beamforming with STFT domain DNN to improve the overall signal enhancement performed by the signal enhancement system 104 without incurring any algorithmic latency.

[0113] In some embodiments, more advanced beamformers or dereverberation algorithms such as WPE may be used to achieve even better enhancement than MCWF without departing from the scope of this disclosure.

[0114]

number

[0115] Some embodiments are based on the recognition that both the magnitude and phase estimated by the DNN can be used in the calculations of the beamformer 234, and the DNN is used to estimate the target speech at the reference microphone q.

[0116]

number

[0117]

number

[0118]

number

[0119] Therefore, the signal enhancement system 104 can perform DNN-based low-latency speech signal enhancement. The architecture of two DNNs used in the signal enhancement performed by the signal enhancement system 104 is exemplarily shown in FIG. 4 (FIG. 4, FIG. 4Continued).

[0120] 4 is a schematic diagram illustrating a network architecture 400 for each of two DNNs, DNN1 232 and DNN2 236, used in the signal enhancement system 104 according to some other embodiments of the present disclosure. This network architecture is only one of the example architectures used for illustrative purposes. Other suitable neural network architectures may be equally used without departing from the scope of the present disclosure.

[0121] Each of the two DNNs, DNN1232 and DNN2236, is trained to perform complex spectral mapping, where the real and imaginary (RI) components of multiple signals are concatenated as inputs for the DNN to predict the target RI component at the reference microphone. Additionally, each of the two DNNs, DNN1232 and DNN2236, is trained through a dual window size approach to define a loss function on the resynthesized signal and operate with dramatically reduced computational effort per frame over previously known DNN models.

[0122] Each of the two DNNs, namely DNN1232 and DNN2236, is based on the long-short-term memory (LSTM) and ResUNet components shown in Figure 4.

[0123]

number

[0124] The encoder 401 includes one two-dimensional (2D) convolution 404 followed by causal layer normalization (cauLN) for each input signal, and six convolution blocks, namely convolution block 405a, convolution block 405b, convolution block 405c, convolution block 405d, convolution block 405e, and convolution block 405f (hereinafter referred to as convolution blocks 405a-405f). Each convolution block includes a 2D convolution, a parameterized ReLU (PReLU) nonlinearity, and a batch normalization (BN) for downsampling. The LSTM 402 includes three layers, each with 300 units.

[0125] The decoder 403 includes six 2D deconvolution blocks with PReLU and BN, namely, deconvolution block 406a, deconvolution block 406b, deconvolution block 406c, deconvolution block 406d, deconvolution block 406e and deconvolution block 406f (hereinafter referred to as deconvolution blocks 406a-406f), and one 2D deconvolution block 407 for upsampling.

[0126] Each remaining block in the encoder 401 and the decoder 403 includes five depthwise separable 2D convolution (denoted as dsConv2D) blocks, namely, dsConv2D block 408a, dsConv2D block 408b, dsConv2D block 408c, dsConv2D block 408d, and dsConv2D block 408e (hereinafter referred to as dsConv2D blocks 408a to 408e). In these dsConv2D blocks 408a to 408e, the dilation ratios along the time are 1, 2, 4, 8, and 16, respectively. Linear activation is used in the output layer to obtain the predicted RI components.

[0127] In the architecture of DNN1232 shown in Figure 4, all convolutional and normalization layers are causal at run time (i.e., frame-online). DNN1232 uses 1X3 or 1X4 kernels along time and frequency for downsampling and upsampling convolutions. The remaining blocks use 2X3 causal convolutions.

[0128] The architecture of DNN1232 shown in Figure 4 has several advantages. This architecture dramatically reduces the amount of computation and the number of trainable parameters, which is a very effective and superior computational performance parameter realized by the architecture disclosed herein.

[0129] In some exemplary embodiments, the mixture of audio signals 102 may be from a single microphone or an array of microphones. To that end, DNNs such as DNN1232 and DNN2236 may be trained based on a spectral mapping corresponding to a single microphone and an array of microphones. This spectral mapping trains the DNN1232 to predict real and imaginary (RI) components (i.e., frequencies) of an estimate, such as the first estimate 232a of the input mixture of audio signals 102 from the RI components of the input mixture of audio signals 102. The RI components of the input mixture of audio signals 102 and the RI components of the first estimate 232a may be input to the DNN2236 to predict a second estimate of the target audio signal input mixture 102. The DNN1232 may be pre-trained using a training dataset of mixtures of acoustic signals and corresponding reference target direct path signals in the training dataset.

[0130] In some embodiments, the pre-training of the DNN 1232 may be performed by minimizing a loss function. The loss function may include one or a combination of distance functions defined based on the RI components of the target direct path speech signal input mixture 102 in the first time-frequency domain and the RI components of the reference target direct path signal in the first time-frequency domain. The reference target direct path signal may be obtained from a training dataset of speech, and the corresponding reverberant mixture may be obtained by convolving the reference target direct path signal with the recorded or synthetic RIR and summing with other interference signals. The distance function may be defined based on the magnitude obtained from the RI components of the estimated target direct path signal in the first time-frequency domain and the corresponding magnitude of the reference target direct path signal.

[0131] In an alternative embodiment, the distance function may be defined based on a reconstructed waveform obtained from the RI component of the estimated target direct path signal in the first time-frequency domain by reconstruction in the time domain and the waveform of the reference target direct path signal. The distance function may also be defined based on the RI component in the complex time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain. The distance function may also be defined based on a magnitude obtained from the RI component in the second time-frequency domain obtained by transforming the reconstructed waveform in the second time-frequency domain and the corresponding magnitude of the reference target direct path signal in the second time-frequency domain.

[0132]

number

[0133]

number

[0134] In some exemplary embodiments, the mixture of acoustic signals 102 may correspond to a multi-channel signal that may be received from an array of microphones in an audio processing system.

[0135] 5 is a block diagram of an audio signal processing system 500 according to an embodiment of the present disclosure. The audio signal processing system 500 uses the signal enhancement system 104. In some exemplary embodiments, the signal enhancement system 104 having DNNs for enhancing speech signals, such as DNN1 232 and DNN2 236, may be implemented on a remote server or in a cloud network. In some embodiments, the audio signal processing system 500 (hereinafter referred to as system 500) uses a dual window size based processing of the audio signal using a first window function 200 on the input side and a second window function 214 on the output side.

[0136] In some demonstrative embodiments, system 500 includes one or more sensors 502 , such as an acoustic sensor, that collects data including an acoustic signal 504 from an environment 506 .

[0137] The acoustic signal 504 may include a multi-channel audio signal, such as the input mixture of audio signals 102 .

[0138] The audio signal processing system 500 includes a hardware processor 508 in communication with a computer storage memory, such as memory 510. The memory 510 includes stored data, including algorithms, instructions, and other data that may be executed by the hardware processor 508. Depending on the requirements of a particular application, it is contemplated that the hardware processor 508 may include two or more hardware processors. These two or more hardware processors may be either internal or external. The audio signal processing system 500 may be incorporated into other components, including output interfaces and transceivers, among others.

[0139] In some alternative embodiments, the hardware processor 508 may be connected to a network 512 that communicates with one or more data sources 514, a computer device 516, a mobile phone device 518, and a storage device 520. The network 512 may include, by way of non-limiting example, one or more local area networks (LANs) and / or wide area networks (WANs). The network 512 may also include an enterprise-wide computer network, an intranet, and the Internet. The audio signal processing system 500 may include one or more client devices, storage components, and data sources. Each of the one or more client devices, storage components, and data sources may comprise a single device or may comprise multiple devices that cooperate in a distributed environment of the network 512.

[0140] In some other alternative embodiments, the hardware processor 508 may be connected to a network-enabled server 522 that is connected to the client device 524. The hardware processor 508 may be connected to an external memory device 526 and a transmitter 528. Additionally, an output may be output for each target speaker according to an intended use 530 by a particular user. For example, the intended use 530 by a particular user may correspond to displaying the speech as text (e.g., speech commands) on one or more display devices such as a monitor or screen, or inputting the text for each target speaker into a computer-related device for further analysis, etc.

[0141] The data source 514 may comprise data resources for training DNNs, such as DNN1 232 and DNN2 236, for speech enhancement tasks. For example, in one embodiment, the training data may include acoustic signals of multiple speakers speaking simultaneously. The training data may also include acoustic signals of a single speaker speaking alone, acoustic signals of single or multiple speakers speaking in a noisy environment, and acoustic signals of a noisy environment.

[0142] The data source 514 may also comprise data resources for training the DNN1 232 and the DNN2 236 for a speech recognition task. The data provided by the data source 514 may include labeled and unlabeled data, such as transcribed and non-transcribed data. For example, in one embodiment, the data may include one or more sounds and may also include corresponding transcription information or labels that may be used to initialize a speech recognition task.

[0143] Additionally, the unlabeled data in the data sources 514 may be provided by one or more feedback loops. For example, usage data from spoken search queries executed on a search engine may be provided as non-transcribed data. Other examples of data sources may include, by way of example and not limitation, various oral language audio or image sources including streaming sound or video, web queries, mobile device camera or audio information, webcam feeds, smart glasses and smartwatch feeds, customer care systems, security camera feeds, web documents, catalogs, user feeds, SMS logs, instant messaging logs, spoken word transcripts, gaming system user interactions such as voice commands or captured images (e.g., depth camera images), tweets, chat or video call recordings, or social networking media. The particular data source 514 used may be determined based on the application, including whether the data is a particular class of data (e.g., data related only to a particular type of sound, including machine systems, entertainment systems) or general (non-class specific) in nature.

[0144] The audio signal processing system 500 may also include a third party device, which may be comprised of any type of computing device, such as an automatic speech recognition (ASR) system on a computing device. For example, the third party device may include a computer device or a mobile device 518. The mobile device 518 may include a personal data assistant (PDA), a smartphone, a smart watch, smart glasses (or other wearable smart devices), an augmented reality headset, a virtual reality headset, a laptop, a tablet, a remote control device, an entertainment system, a vehicle computer system, an embedded system controller, an appliance, a home computer system, a security system, a consumer electronic device, or other similar electronic device. The mobile device 518 may also include a microphone or line input for receiving audio information, a camera for receiving video or image information, or a communication component (e.g., Wi-Fi capability) for receiving such information from another source, such as the Internet or a data source 514. In one exemplary embodiment, the mobile device 518 may be capable of receiving input data, such as audio and image information. For example, the input data may include a speaker's query into a microphone of the mobile device 518 while multiple speakers in a room are speaking. To determine the query content, the input data may be processed by the ASR in the mobile device 518 using the system 500. The system 500 may enhance the input data by reducing noise in the speaker's environment, separating the speaker from other speakers, or enhancing the audio signal of the query to enable the ASR to output an accurate response to the query.

[0145] In some demonstrative embodiments, storage 520 may store data, computer instructions (e.g., software program instructions, routines or services), and / or information including data related to DNNs, such as DNN1 232 and DNN2 236 of system 500. For example, storage 520 may store data from one or more data sources 514, one or more deep neural network models, information for generating and training the deep neural network models, and computer usable information output by one or more deep neural network models.

[0146] 6 illustrates a use case 600 for enhancement of speech signals, according to some exemplary embodiments of the present disclosure. The use case 600 corresponds to a teleconference room including a group of speakers (group of speakers 602A-602F), such as speaker 602A, speaker 602B, speaker 602C, speaker 602D, speaker 602E, and speaker 602F. Speech signals of one or more speakers of the group of speakers 602A-602F are received by an audio receiver 606 of a device 604. The audio receiver 606 includes a signal enhancement system 104 and receives an acoustic speech signal of a speaker or one or more speakers from the group of speakers 602A-602F that needs to be processed with low latency.

[0147] The audio receiver 606 may include a single microphone and / or an array of microphones for receiving a mixture of acoustic signals and noise signals from the group of speakers 602A-602F in the teleconference room. These mixtures of acoustic signals from the group of speakers 602A-602F may be processed using the system 104. For example, the system 104 may analyze an RIR model of the teleconference room. The RIR model may be used to generate a room geometry structure of the teleconference room. The room geometry structure may be used to locate reflection boundaries in the teleconference room. For example, the corresponding room geometry structure may be used to determine the placement of speakers, seating arrangements of the group of speakers 602A-602F, etc., to cancel noise and other disturbances in the teleconference room. Furthermore, the RIR model may be used to remove reflections and reverberations of speech signals of one or more speakers of the group of speakers 602A-602F to perform speech signal enhancement.

[0148] In the illustrated exemplary scenario, multiple speakers in the group of speakers 602A-602F may output speech signals simultaneously. In such a scenario, the system 104 reduces reverberation in the teleconference room and separates the speech signals of each of the speakers 602A-602F. The system 104 may also perform beamforming of a mixture of acoustic signals from an array of microphones to enhance the speech signals of corresponding speakers in the group of speakers 602A-602F. The enhanced speech signals may be used for transcription of the speakers' utterances. For example, the device 604 may include an ASR module. The ASR module may receive the enhanced speech signals and output a transcription. The transcription may be displayed by a display screen of the device 604.

[0149] FIG. 7 is a diagram illustrating a use case 700 for audio signal enhancement in a hearing aid device 704 according to one embodiment of the present disclosure.

[0150] The hearing aid device 704 may be configured to receive an input audio signal 702, such as a multi-channel audio input equivalent to the input 102 described above, using one microphone or an array of microphones 704a. The input audio signal 702 is then passed to an analog-to-digital converter 704b for conversion from the analog domain to the digital domain. The digital domain signal is then passed to a signal enhancement system, such as the signal enhancement system 104, for low-latency signal enhancement. The signal enhancement system 104 then uses dual window size based processing of the digital domain signal using DNN-based signal enhancement. This DNN-based signal enhancement may be used to perform enhancements such as low-latency beamforming, noise and reverberation suppression, superior speaker separation, etc., performed by the DNN architecture described in Figures 3A, 3B, 3C and 3D, resulting in an overall improvement in the quality of the digital domain signal without additional latency. This high-quality audio signal is then amplified and passed to a digital-to-analog converter 704c to output an enhanced amplified audio signal 706.

[0151] Thus, as noted above, the signal enhancement system 104 may be advantageous for use in many low latency voice applications.

[0152] The various embodiments described above may be described as a process that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although a flowchart may describe operations as a sequential process, many of these operations may be performed in parallel or simultaneously. Also, the order of these operations may be rearranged. A process may be terminated when its operations are completed, but may have additional steps not discussed or included in the diagram. Moreover, not all operations in any process specifically described are performed in all embodiments. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, the end of the function may correspond to a return of the function to a calling function or to a main function.

[0153] Furthermore, embodiments of the disclosed subject matter may be implemented at least partially manually or automatically. The manual or automatic implementation may be performed or at least assisted by the use of machines, hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a machine-readable medium. A processor may perform the necessary tasks.

[0154] The above-described embodiments of the present disclosure can be implemented in any of a number of ways. For example, these embodiments may be implemented using hardware, software, or a combination thereof. When implemented in software, the software code may be executed on any suitable processor or collection of processors, whether provided in a single computer or distributed among multiple computers. Such a processor may be implemented as an integrated circuit with one or more processors in an integrated circuit component. However, the processor may be implemented using circuits in any suitable format.

[0155] Also, the various methods or processes outlined herein may be coded as software executable on one or more processors utilizing any one of a variety of operating systems or platforms. Moreover, such software may be written using any of a number of suitable programming languages ​​and / or programming or scripting tools, and compiled as executable machine language code or intermediate code that runs on a framework or virtual machine. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0156] Also, embodiments of the present disclosure may be embodied as a method, an example of which is provided. The acts performed as part of this method may be ordered in any suitable manner. Thus, embodiments may be constructed in which acts are performed in an order different from that shown, which may include performing some acts simultaneously even though they are shown as sequential acts in the example embodiment. It is therefore the object of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the present disclosure.

[0157] Although the disclosure has been described with reference to certain preferred embodiments, it is to be understood that various other adaptations and modifications may be made within the spirit and scope of the disclosure. It is therefore the aspect of the appended claims to cover all such variations and modifications that come within the true spirit and scope of the disclosure.

Claims

1. 1. A computer implemented method for signal enhancement comprising: receiving an input mixture of audio signals via an input interface including a target audio signal, the input mixture of audio signals being at least one of a multi-channel audio signal or a single-channel audio signal, the signal enhancement method further comprising: The signal enhancement method includes splitting the input mixture of received speech signals into a sequence of input overlapping frames using a first sliding window method, the first sliding window method comprising a first window function having a first width associated with a window of a corresponding frame and a shift length associated with a shifting of the window of the first sliding window method, the signal enhancement method further comprising: processing the sequence of input overlapped frames using a first Deep Neural Network (DNN) to generate enhanced overlapped frames, the enhanced overlapped frames comprising a corresponding enhanced frame for each of the processed frames among the input overlapped frames, the signal enhancement method further comprising: generating a frequency domain filtered output for each frame of the enhanced overlapped frames; processing the frequency domain filtering output for each frame of the enhanced duplicated frame using a second DNN to generate a corresponding final enhanced frame for each frame of the enhanced duplicated frame; and combining the final enhanced overlapped frames using a second sliding window technique, the second sliding window technique being associated with a second window function having a second width less than the first width and a shift length equal to that of the first sliding window technique.

2. The signal enhancement method of claim 1 , wherein the frequency domain filtering output for each frame of the enhanced overlapping frames is a causal linear filtering output generated by a causal linear filter.

3. The signal enhancement method of claim 1 , wherein the frequency domain filtered output for each frame of the enhanced overlapped frames is a beamforming output generated by a beamformer.

4. The method of claim 3 , wherein the beamformer is a Multi-Channel Wiener Filter (MCWF).

5. The signal enhancement method of claim 1 , wherein the frequency domain filtering output for each frame of the enhanced overlapping frames is a dereverberation output.

6. The method of claim 5 , wherein the dereverberated output is obtained based on a convolutional prediction.

7. The signal enhancement method of claim 5 , wherein the dereverberated output is obtained based on a weighted prediction error method.

8. The signal enhancement method of claim 1 , wherein the second DNN further processes one or a combination of the input duplicated frame and the enhanced duplicated frame.

9. The method of claim 1 , wherein one or more of the first window function and the second window function are asymmetric window functions.

10. 2. The method of claim 1, wherein the input mixture of received audio signals is a multi-channel audio signal including speech signals from multiple speakers, and the first DNN generates a plurality of outputs, each output of the plurality of outputs corresponding to one speaker from the plurality of speakers.

11. The method of signal enhancement of claim 1 , wherein receiving the input mixture of audio signals comprises receiving the input mixture of audio signals from an array of microphones connected to the input interface.

12. The signal enhancement method of claim 1 , wherein the first DNN is pre-trained to generate the enhanced overlapped frames from a mixture of observed acoustic signals.

13. The pre-training of the first DNN is performed using a training dataset of a mixture of acoustic signals and corresponding reference target direct path signals in the training dataset by minimizing a loss function, the loss function being: a distance function defined based on real and imaginary (RI) components of a first estimate of an intermediate representation for each frame in a first time-frequency domain and RI components of the corresponding reference target direct path signal in the first time-frequency domain; a distance function defined based on a magnitude obtained from the RI component of the first estimate of the intermediate representation for each frame in the first time-frequency domain and a corresponding magnitude of the reference target direct path signal in the first time-frequency domain; a distance function defined based on a reconstructed waveform obtained from the RI component of the first estimate of the intermediate representation for each frame in the first time-frequency domain by reconstruction in the time domain and a waveform of the reference target direct path signal; a distance function defined based on the RI component of the first estimate in the time-frequency domain obtained by further transforming the reconstructed waveform in a second time-frequency domain and the RI component of the reference target direct path signal in the second time-frequency domain; and a distance function defined based on the magnitude obtained from the RI component of the first estimate of the intermediate representation for each frame in the time-frequency domain obtained by further transforming the reconstructed waveform in the second time-frequency domain and the corresponding magnitude of the reference target direct path signal in the second time-frequency domain; The method of claim 12, further comprising one or a combination of:

14. 1. A signal enhancement system comprising: an input interface configured to receive an input mixture of audio signals including a target audio signal, the input mixture of audio signals being at least one of a multi-channel audio signal or a single-channel audio signal, the signal enhancement system further comprising: a memory for storing computer executable instructions; a processor configured to execute the computer-executable instructions, the computer-executable instructions comprising: The computer-executable instructions are further configured to: divide the input mixture of received audio signals into a sequence of input overlapping frames using a first sliding window method, the first sliding window method comprising a first window function having a first width associated with a window of a corresponding frame and a shift length associated with a shifting of the window of the first sliding window method; The computer-executable instructions are further configured to: process the segmented sequence of input overlap frames using a first deep neural network (DNN) to generate enhanced overlap frames, the enhanced overlap frames comprising a corresponding enhanced frame for each of the processed frames among the overlap frames; and configured to generate a frequency domain filtered output for each frame of the enhanced overlapped frames; and processing the frequency domain filtering output for each frame of the enhanced duplicated frame using a second DNN to generate a corresponding final enhanced frame for each frame of the enhanced duplicated frame; 11. The signal enhancement system of claim 10, further comprising: a first sliding window function associated with said first window function and a second sliding window function associated with said first window function;

15. The signal enhancement system of claim 14 , wherein the frequency domain filtered output for each frame of the enhanced overlapping frames is a causal linear filtered output generated by a causal linear filter.

16. The signal enhancement system of claim 14 , wherein the frequency domain filtered output for each frame of the enhanced overlapping frames is a beamforming output generated by a beamformer.

17. The signal enhancement system of claim 16 , wherein the beamformer is a multi-channel Wiener filter (MCWF).

18. The signal enhancement system of claim 14 , wherein each of the first window function and the second window function is an asymmetric window function.

19. The signal enhancement system of claim 14 , wherein the second DNN further processes one or a combination of the input duplicated frames and the enhanced duplicated frames.

20. 15. The signal enhancement system of claim 14, wherein the received input mixture of audio signals is a multi-channel audio signal including speech signals from multiple speakers, and the first DNN generates a plurality of outputs, each output of the plurality of outputs corresponding to one speaker from the plurality of speakers.

Images