Neural network-based time-frequency mask estimation and beam shaping for speech preprocessing

Abstract

Non-volatile computer-readable storage medium(s) containing instructions encoded thereon which, when executed by one or more processors (1120), cause a process for improving a speech signal to be executed, wherein the process comprises: performing (1010) a denaturation (DRV) processing on a first and second signal (115) derived from a first and second signal, respectively.second microphone (110) are received, wherein the first and second signals (115) each comprise a combination of speech and noise; generating (1020) a first and second time-frequency mask (TFM), wherein both the first and second TFM are associated with a corresponding of the DRV-processed first and second signals, wherein both the first and second TFM comprise elements that indicate a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) (510) to the DRV-processed first and second signals; generating (1030) steering vectors based on speech covariance matrices and noise covariance matrices, wherein the first and second TFM are used to select speech components of the DRV-processed first and second signals, respectively.to filter the second signal for the calculation of the speech covariance matrices and to filter noise components of the DRV-processed first and second signals for the calculation of the noise covariance matrices; and performing (1040) beamforming on the DRV-processed first and second signals based on the steering vectors to generate a single-channel speech signal (345).

Description

CROSS-REFERENCE TO RELATED REGISTRATIONS

[0001] The present application claims the benefit of the preliminary US application No. 62 / 673,238, filed on May 18, 2018, which is incorporated herein in its entirety by reference. BACKGROUND

[0002] The availability and use of speech-based devices are becoming increasingly widespread. Accurate speech recognition and understanding are essential for a satisfactory user experience. However, speech signals picked up in the far field of a microphone often lack sufficient quality due to noise and reverberation to meet the requirements of automatic speech recognition systems and other speech processing applications, which must provide a relatively low word error rate for acceptable performance. Existing far-field speech preprocessing techniques attempt to improve the quality of the received signals but suffer from a number of non-trivial problems, including latency, complexity, and the need for a microphone array containing a relatively large number of microphones.In addition, many existing techniques rely on speech activity detection, which generally does not work well at low signal-to-noise ratios. List of characters

[0003] The patent or patent application document contains at least one color drawing. Copies of this patent or patent application publication with color drawing(s) are provided by the patent office upon request and payment of the required fee.

[0004] Features and advantages of embodiments of the claimed subject matter will become apparent during the following detailed description and with reference to the drawings, in which identical numbers represent identical parts. Fig. Figure 1 is a top-level block diagram of a multi-channel speech processing system configured according to certain embodiments of the present disclosure. Fig. Figure 2 is a more detailed block diagram of the multi-channel speech preprocessing pipeline configured according to certain embodiments of the present disclosure. Fig. Figure 3 is a top-level block diagram of a single-channel speech processing system configured according to certain other embodiments of the present disclosure. Fig. Figure 4 is a more detailed block diagram of the single-channel speech preprocessing pipeline configured according to certain other embodiments of the present disclosure. Fig. Figure 5 is a block diagram of a time-frequency (TF: Time-Frequency) mask estimation circuit configured according to certain embodiments of the present disclosure. Fig. Figure 6 illustrates speech spectrograms and associated TF masks according to certain embodiments of the present disclosure. Fig. Figure 7 is a block diagram of an RNN training circuit configured according to certain embodiments of the present disclosure. Fig. Figure 8 illustrates a histogram of amplification factors according to certain embodiments of the present disclosure. Fig. Figure 9 illustrates the results of speech preprocessing according to certain embodiments of the present disclosure. Fig. Figure 10 is a flowchart illustrating a methodology for language preprocessing according to certain embodiments of the present disclosure. Fig. Figure 11 is a block diagram that schematically illustrates a speech-based setup platform configured to perform speech preprocessing according to certain embodiments of the present disclosure.

[0005] Although the following detailed description continues with reference to illustrative embodiments, many alternatives, modifications and variations thereof will become apparent with respect to this disclosure. DETAILED DESCRIPTION

[0006] Techniques for preprocessing speech signals captured in the far field of a microphone are provided, using deep learning (e.g., a neural network) based on time-frequency mask estimation and beamforming, to improve or enhance the quality of the speech component of the signal with respect to potentially present interfering noise and reverberation effects. In one embodiment, the disclosed preprocessing pipeline can serve as a front end for a speech recognition system or other speech processing applications to improve the recognition word error rate. Other applications will become apparent.

[0007] Audio beamforming is used to spatially filter out noise from a signal, but many, if not most, devices and platforms, including personal computers, tablets, and smartphones, are generally equipped with only one or two microphones, limiting the effectiveness of traditional beamforming techniques. However, one embodiment of the present disclosure provides a preprocessing pipeline that uses a trained recurrent neural network (RNN) to estimate time-frequency (TF) masks that distinguish noise from the signal in both time and frequency. This, in turn, enables the computation of beamforming steering vectors with improved performance using relatively few microphones (e.g., one or two), as explained in more detail below, although the preprocessing pipeline will also provide improved performance for arrays with a larger number of microphones.In some such embodiments, the preprocessing pipeline also includes de-reverberation and post-filtering operations. Additionally, methods for training the RNN are disclosed. As can be seen from this disclosure, the techniques can be used in conjunction with an array of microphones, and according to some embodiments, the array can include as few as two microphones. Other embodiments can operate with a single microphone and use the TF masks for noise reduction without beamforming, as explained below.

[0008] This disclosure thus provides techniques for improving the preprocessing of a speech signal, for example, to improve the word error rate of downstream processing modules, including automatic speech recognition (ASR) engines. The disclosed techniques can be implemented, for example, in a computing system or a software product executable or otherwise controllable by such systems, although other embodiments are apparent. The system or product is designed to perform far-field speech preprocessing using deep-learning-based time-frequency mask estimation and beamforming. According to one embodiment, a methodology for implementing these techniques involves performing de-reverberation processing on signals received by an array of microphones, wherein the signals include speech and noise.The procedure also includes generating time-frequency (TF) masks for each of the signals. The TF masks indicate the probability that a time-frequency component of the signal associated with that TF mask element contains speech. TF mask generation is based on applying a trained recurrent neural network to the signals. The procedure further includes generating steering vectors based on speech covariance matrices and noise covariance matrices. The TF masks are used to filter speech components of the signals to calculate speech covariance and to filter noise components of the signals to calculate noise covariance. The procedure also includes performing beamforming on the signals based on the steering vectors to generate the enhanced speech signal by spatially focusing on the speech source and reducing the influence of noise and reverberation arriving from other directions.

[0009] As acknowledged, the techniques described herein can enable an improved user experience with a voice-enabled device by providing an enhanced speech signal for subsequent applications such as automatic speech recognition (ASR), keyword recognition, wake-on-voice (WoV), and / or VoIP (Voice over Internet Protocol) communications. The enhanced speech signal results in reduced word error rates in these applications and an overall improvement in operation.

[0010] The disclosed techniques can be implemented on a wide range of platforms, including laptops, tablets, smartphones, workstations, video conferencing systems, gaming systems, smart home control systems, robotic systems, and embedded low-power DSP / CPU systems or devices. In addition, in some embodiments, the data can be processed entirely on a local platform, or portions of the processing can be offloaded to a remote platform (which may employ, for example, cloud-based processing, a cloud-based speech-enabled service, or a cloud-based speech-enabled application that can be accessed by various local computing systems of a user). Furthermore, the techniques can be implemented in hardware, software, or a combination thereof.

[0011] Fig. Figure 1 is a top-level block diagram of a multi-channel speech processing system. 100, which is configured according to certain embodiments of the present disclosure. The system 100 is a microphone array 110 The device shown comprises two or more microphones and is designed to record audio signals. The audio signals may include speech, noise and reverberation, or other environmental effects, especially when in the far field. 105 The microphone array is used to record signals from a distance of more than half a meter or more than one meter from the microphone. The microphone array is part of the multi-channel speech preprocessor pipeline. 120 a multi-channel input signal 115 Ready. The pipeline 120 is designed to provide an improved speech signal 145 to generate, as discussed in more detail below, any number of speech processing applications 150 to be provided. These applications can, for example, include an ASR engine. 160 , a WoV or keyword recognition processor170 and / or a VoIP communication system 180 include, to name just a few.

[0012] Fig. Figure 2 is a more detailed block diagram of the multi-channel speech preprocessing pipeline. 120 , which is configured according to certain embodiments of the present disclosure. The multi-channel speech preprocessing pipeline 120 is a signal conditioning circuit 210 , an AEC circuit 220 (AEC: Acoustic Echo Cancellation), a DRV circuit 230 (DRV: De-Reverberation), a time-frequency mask estimation circuit 240 , a beam shaping circuit 250 , an IMCRA post-filtering circuit 260 (IMCRA: Improved Minima Controlled Recursive Averaging), an SPM circuit 270 (SPM: Spectral Profile Matching) and an AGC circuit 280(AGC: Automatic Gain Control) is shown. The TF mask estimation circuit is also shown. 240 is implemented as a recursive neural network (RNN).

[0013] The signal conditioning circuit 210 is designed to make any necessary gain adjustments and remove DC bias present in multi-channel signals. 115 These adjustments prepare the signals for subsequent operations in the preprocessing pipeline. The optional AEC circuit 220 is designed to perform a first pass to remove or reduce echoes in the signal using known echo suppression techniques as disclosed herein. The DRV circuit 230is designed to perform a de-reverberation operation on the signals, also using known techniques with regard to the present disclosure, in order to improve the accuracy of the subsequent TF mask estimation.

[0014] The TF mask estimation circuit (RNN) 240 is designed to generate an estimated time-frequency mask for each of the signals processed by the DRV. The TF mask is a two-dimensional data structure (where the dimensions are time and frequency) that comprises elements corresponding to time windows and frequency bins in the manner of a spectrogram. Fig. Figure 6 illustrates speech spectrograms and associated TF masks according to certain embodiments of the present disclosure. A spectrogram of a clear speech sample 610 is together with the resulting TF mask 630The power levels of each element in the spectrogram are color-coded from yellow, representing the highest level at 0 dB, down to the lowest level, coded as blue at -60 dB. The values ​​in the TF mask, representing the estimated probability that speech is present at that corresponding time and frequency, are also color-coded from yellow, representing a normalized probability of 1, down to blue, representing a normalized probability of 0. Thus, a value of 1 indicates that the component is recognized as speech, while a value of 0 indicates that the component is recognized as noise. As can be seen, there is a strong correlation between the spectrogram of clear speech and the resulting estimated TF mask. Another spectrogram of a noisy speech sample is shown. 620 is together with the resulting TF mask 640depicted. In this case, the spectrogram shows 620 Due to the added noise, significant energy is lost over a wider range of time and frequency bins. The TF mask 640 The intoxicated language remains the TF mask 630 However, it is relatively similar to clear speech and serves as a guide to where the speech signal is likely to be present in terms of time and frequency.

[0015] An estimated TF mask is generated for each channel of the signal by a trained RNN that processes the received signals. The training process is described in more detail below. Fig. Figure 5 illustrates a block diagram of the time-frequency mask estimation circuit. 240 , which is configured according to certain embodiments of the present disclosure. The RNN circuit 510 is represented as comprising 5 layers. The first layer 520 and the second layer 530are affine (fully linked) layers with tangent hyperbolic activation functions. The third layer 540 and the fourth layer 550 These are LSTM layers (LSTM: Long Short-Term Memory) with sigmoid activation functions. The fifth layer 560 is an affine layer with a sigmoid activation function. In some embodiments, the layers can 1 , 2 and 5 each have 256 nodes, while the layers 3 and 4Each RNN can have 128 nodes. It is understood that other configurations with different numbers and types of layers, activation functions, and node counts are possible. Generally, however, the RNN will include at least one recurrent layer to provide a history or memory of previous data, enabling the modeling of temporal dependencies without the need for actual storage of past data.

[0016] Again with reference to Fig. 2 is the beam shaping circuit 250 Designed to generate steering vectors based on speech covariance matrices and noise covariance matrices, the previously estimated TF masks are used to filter speech components of the DRV-processed signals for calculating the speech covariance matrices and noise components of the DRV-processed signals for calculating the noise covariance matrices. The noise covariance matrix F NN can be calculated / updated based on an application of the TF mask as follows: Φ N N ( k , t + 1 ) = α ⋅ Φ N N ( k , t ) + ( 1 − α ) ⋅ ( ( 1 − T F M ( k , t ) ) ⋅ X * ( k , t ) ⋅ X ( k , t ) ) where X is the complex spectrum of the recorded signal for the frequency bin k and α is a selected memory decay factor. The t and k parameters specify the time window and frequency bin of each element. Similarly, the speech covariance matrix F SS Based on an application of the TF mask, the following will be calculated / updated: Φ X X ( k , t + 1 ) = α ⋅ Φ X X ( k , t ) + ( 1 − α ) ⋅ ( T F ( k , t ) ⋅ X * ( k , t ) ⋅ X ( k , t ) ) Φ S S ( k , t + 1 ) = Φ X X ( k , t ) − Φ N N ( k , t + 1 )

[0017] In some embodiments, the size of the spectrum can be used instead of the full complex spectrum to simplify and reduce the computational effort of the calculations.

[0018] The beamforming circuit is further designed to perform beamforming on the DRV-processed signals based on the steering vectors to generate a single-channel speech signal that is directed to the speech source. In some embodiments, the beamforming employs generalized eigenvalue beamforming with blind-analytic normalization. The steering vectors or beamformer weights W(k) can be calculated as the maximum eigenvector corresponding to the largest eigenvalue of Φ N N − 1 ( k ) Φ S S ( k ) This corresponds to the beam shaper output, which can then be calculated as follows: Y ( k ) = ∑ i = 1 M W i * ( k ) ⋅ X i ( k ) = ∑ i = 1 M W i * ( k ) ⋅ ( S i ( k ) + N i ( k ) ) where M is the number of channels of the input signal (e.g., the number of microphones in the array). 110 ).

[0019] The IMCRA post-filtering circuit 260is designed to perform an improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal using techniques known with respect to the present disclosure.

[0020] The SPM circuit 270 It is designed to perform spectral shaping on the IMCRA-filtered single-channel speech signal based on heuristics associated with a selected ASR engine. Different ASR engines, provided by different vendors, are often tuned to work best with signals that are spectrally shaped in a specific way. For example, a given ASR engine might expect the input speech signal to be filtered with a high cutoff frequency of 6 kHz.

[0021] The AGC circuit 280 is designed to perform any final amplification control to enhance the speech signal145 to scale to a suitable amplitude range based on the requirements of the subsequent speech processing applications.

[0022] Fig. Figure 3 is a top-level block diagram of a single-channel speech processing system. 300 , which is configured according to certain other embodiments of the present disclosure. The system 300 is a single microphone 310 shown, which is designed to record audio signals. The audio signals may contain speech, noise, reverberation, and other environmental effects, especially when located in the far field. 105 The microphone is picked up from a distance of more than half a meter or one meter (e.g., more than half a meter or one meter from the microphone). The microphone is part of the single-channel speech preprocessor pipeline. 320 a single-channel input signal 315 ready, which is designed to provide an improved speech signal 345to generate, as described in more detail below, which is used in language processing applications 150 to be provided.

[0023] Fig. Figure 4 is a more detailed block diagram of the single-channel speech preprocessing pipeline. 320 , which is configured according to certain other embodiments of the present disclosure. The single-channel speech preprocessing pipeline 320 is a signal conditioning circuit 210 , an AEC circuit 220 (AEC: acoustic echo cancellation), a DRV circuit 230 (DRV: Enthallung), a time-frequency mask estimation circuit 240 , a noise reduction circuit 450 , an SPM circuit 270 (SPM: Spectral Profile Matching) and an AGC circuit 280 (AGC: automatic gain control) shown. The TF mask estimation circuit. 240 is implemented as a recursive neural network (RNN).

[0024] The signal conditioning circuit 210 , the AEC circuit 220 , the DRV circuit 230 and the TF mask estimation circuit (RNN) 240 are designed to perform the functions as previously described for the multi-channel pipeline 120 As described, these will be carried out on the only available channel.

[0025] Instead of the beam shaping circuit 250 the multi-channel preprocessing pipeline 120 is the noise reduction circuit 450 The single-channel preprocessing pipeline is designed to apply the TF mask directly to the spectrum of the captured signal. This can be expressed by the following formula: Y ( t , k ) = T F M ( t , k ) ⋅ X ( t , k ) where X is the complex spectrum of the acquired signal (e.g., the signal represented in the frequency domain), TFM is the time-frequency mask, and Y is the complex spectrum of the improved signal after noise reduction. The t and k parameters specify the time window and frequency bin of each element. In some embodiments, the complex spectrum can be converted into magnitude and phase components, and the modification (e.g., applying the mask) can be performed on only the magnitude spectrum to simplify and reduce the computational effort.

[0026] In some embodiments, the noise reduction circuit 450It may also be designed to perform additional signal processing operations on the TF mask, such as applying a moving average to smooth the mask and thresholding to restrict mask values ​​to a desired range. For example, the mask can be set to zero if the input signal is below a selected threshold. In some embodiments, the mask can be trimmed to a selected interval (e.g., [0,1]). In a recurrent neural network, this can be achieved implicitly by using a sigmoid activation function. In some further embodiments, a weighted sum of the original and processed spectrum can be used to control the degree of noise reduction. This can be expressed, for example, by the following formula: Y ( t , k ) = ( w x + w T F ⋅ T F M ( t , k ) ) ⋅ X ( t , k ) where w X and w TF Selected weighting factors are.

[0027] The SPM circuit 270 and the AGC circuit 280 are designed to perform the functions as previously described for the multi-channel pipeline 120 described.

[0028] Fig. Figure 7 is a block diagram of an RNN training circuit 700 , which is configured according to certain embodiments of the present disclosure. The RNN training circuit 700 is a convolution circuit 750 , a feature extraction circuit 740 , a summing circuit 760 , an IRM calculation circuit 770 (IRM: Ideal Ratio Mask), an RNN circuit 510 , an MSE calculation circuit 780 (MSE: Mean Squared Error) and an RNN parameter update circuit 790 presented below. Furthermore, databases for clear speech samples are included. 710 , reverberation filter 720and intoxication tests 730 depicted.

[0029] For each training iteration, a clear speech signal sample s(t) is taken from the database. 710 A reverberation filter h(t) is selected from the database. 720 selected and a noise signal sample n(t) is taken from the database 730 selected. The reverberation filter models the acoustic echoes or reflections associated with an environment (e.g., rooms of varying sizes and geometries) that can distort a signal picked up from the microphones' far field. The noise samples can include any type of background noise, including white noise, colored noise, and babble (e.g., snippets of background speech or music). The convolution circuit 750 is designed to apply the reverberation filter to the clear speech signal sample to generate s(t)*h(t), where * represents the convolution operator. The summing circuit760 is designed to add the noise signal sample after applying a gain factor g, which is selected to achieve a desired signal-to-noise ratio (SNR), resulting in x(t) = s(t)*h(t) + g · n(t). The feature extraction circuit 740 is designed to generate a frequency-domain feature representation X(k) of the noisy signal x(t). In some embodiments, X(k) can represent extracted feature vectors of the noisy signal, a raw waveform of the noisy signal, or spectral coefficients of the noisy waveform. The frequency-domain feature representation X(k) of the noisy signal is then fed to the RNN circuit to be trained. 510 provided, and an estimated TF mask is generated for this iteration.

[0030] The feature extraction circuit 740It is also designed to generate a frequency-domain feature representation S(k) of the clear speech signal s(t). The IRM calculation circuit 770 The process is designed to generate a mask with an ideal ratio as a training target for the RNN, for example, IRM(k) = S(k) / X(k). This process allows the noisy speech X to be precisely matched to the clear speech S, thus enabling improved training of the neural network. In some embodiments, an ideal binary mask can be used as an alternative to the IRM. In some embodiments, additional processing can be performed on the IRM, including, for example, smoothing and thresholding operations, as previously described in connection with the computation of the TF mask.

[0031] The MSE calculation circuit 780is designed to calculate the mean squared error (or any other desired error measurement) between the IRM and the estimated TF mask. The RNN parameter update circuit 790 is designed to update the parameters for training the neural network for this iteration based on the error using known techniques with respect to the present disclosure.

[0032] The availability of a large number of clear speech samples and a database of impulse responses and noise recordings enables the generation of a training dataset with a large number of utterances and a high degree of variation in the type of disturbances. This results in a more robust neural network capable of correctly inferring the TF mask under challenging noise and far-field conditions.

[0033] Fig. Figure 8 illustrates a histogram 800of amplification factors according to certain embodiments of the present disclosure. Due to the fact that clear speech signals in the database 710 Since the signals are typically recorded at higher levels than those picked up by the microphones in the target device, the training of the TF mask estimator can develop a dependency on the input gain. To counteract this effect, some implementations can apply an additional random gain factor derived from the histogram. 800 selected and applied to the training signal X(k) for the RNN circuit 510 to be applied. The histogram 800 is represented as asymmetric, with a mean of -7 dB, although other distributions are possible.

[0034] Fig. Figure 9 illustrates the results of speech preprocessing according to certain embodiments of the present disclosure. A spectrogram of the input signal 910is together with a spectrogram of the signal improved by the preprocessing pipeline 920 This is demonstrated using a 2-microphone array. As can be seen, preprocessing leads to an improvement in noise reduction, for which some examples are shown by reference number. 930 are specified. methodology

[0035] Fig. 10 is a flowchart that illustrates an exemplary procedure. 1000for the preprocessing of far-field speech using deep-learning-based time-frequency mask estimation and beamforming according to certain embodiments of the present disclosure. As can be seen, the exemplary method includes a number of phases and subprocesses, the sequence of which may vary from one embodiment to another. However, when viewed as a whole, these phases and subprocesses constitute a speech preprocessing process according to certain embodiments disclosed herein. These embodiments can, for example, be implemented using the methods described in the Fig. 1-5 and Fig. The system architecture illustrated in section 7 can be implemented as described above. However, other system architectures may be used in other embodiments, as will be evident from this disclosure. For this purpose, it is not intended that the correlation of the various in Fig. The functions shown in Figure 10 for the specific components illustrated in the other figures are not intended to indicate any structural and / or usage limitations. Instead, other embodiments may, for example, include varying degrees of integration, where multiple functionalities are effectively performed by a single system. In an alternative embodiment, for example, a single module with decoupled submodules may be used to perform all the functions of the method. 1000to carry out. Thus, other embodiments may have fewer or more modules and / or submodules, depending on the granularity of the implementation. In still other embodiments, the described methodology can be implemented as a computer program product that includes one or more non-volatile, machine-readable media which, when executed by one or more processors, cause the methodology to be carried out. Numerous variations and alternative configurations are evident with respect to this disclosure.

[0036] As in Fig. As illustrated in section 10, the process begins 1000 for speech signal preprocessing in an embodiment involving the execution of a process 1010A Disenchantment Reverberation (DRV) process is applied to each of several signals received by an array of microphones. The signals, which may contain a combination of speech and noise, can be obtained from the far field of the microphones (e.g., more than one meter from the microphones). The DRV process removes or reduces far-field environmental effects, such as reverberation, before subsequent preprocessing operations, thus improving their performance.

[0037] Next, in the process 1020Several time-frequency masks were generated. Each TF mask is associated with one of the DRV-processed signals. The TF masks are two-dimensional and comprise elements corresponding to time and frequency bins. Each TF mask element specifies a probability that the time-frequency component of the DRV-processed signal associated with that element contains speech. The TF mask generation is based on the application of a trained recurrent neural network (RNN) to the DRV-processed signals.

[0038] In some embodiments, the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0039] During the process 1030 Beamforming vectors are generated based on speech covariance matrices and noise covariance matrices of the DRV-processed signals. TF masks are used to filter speech components of the DRV-processed signals for calculating the speech covariance matrices and to filter noise components of the DRV-processed signals for calculating the noise covariance matrices.

[0040] During the process 1040Beamforming is performed on the multiple DRV-processed signals based on the steering vectors to generate an improved single-channel speech signal. In some embodiments, the beamforming employs generalized eigenvalue beamforming with blind-analytical normalization.

[0041] In some embodiments, additional processes can be performed, as previously described in connection with the system. For example, improved minimum controlled recursive averaging (IMCRA) can be performed to reduce stationary noise components of the single-channel speech signal generated by beamforming. In some embodiments, the improved single-channel speech signal is provided to a speech processing application, such as an ASR (Automatic Speech Recognition) engine, a keyword recognition application, a Wake-on-Voice (WoV) application, and / or a VoIP (Voice over Internet Protocol) application. Exemplary system

[0042] Fig. Figure 11 illustrates an exemplary language-supported furnishing platform 1100, which, according to certain embodiments of the present disclosure, is configured to perform speech preprocessing to improve the speech signal before use by a speech processing application. In some embodiments, the platform can 1100It may be hosted on or otherwise integrated into a personal computer, workstation, server system, smart home management system, laptop computer, ultra-laptop computer, tablet, touchpad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cordless phone, combined cordless phone and PDA, smart device (e.g., smartphone or smart tablet), mobile internet device (MID), messaging device, data communication device, wearable device, embedded system, and so on. In certain embodiments, any combination of different devices may be used.

[0043] In some embodiments, the platform can 1100 any combination of a processor 1120 , a storage 1130, speech preprocessor circuits 120 and / or 320 , speech processing applications 160 , 170 , 180 , a network interface 1140 , an input / output (I / O) system 1150, a user interface 1160 , a microphone array 110 and a storage system 1170 include. As can also be seen, a bus and / or interconnect is also required. 1192 Provided to enable communication between the various components listed above and / or other components not shown. The platform 1100 can be accessed via the network interface 1140 with a network 1194 must be coupled to enable communication with other computing facilities, platforms, controllable systems, or other resources. Other components and functionalities not shown in the block diagram of Fig. 11 are reflected in this disclosure, and it is understood that other embodiments are not limited to any particular hardware configuration.

[0044] The processor 1020 can be any suitable processor and can include one or more coprocessors or one or more controllers, such as an audio processor, a graphics processing unit, or a hardware accelerator, to assist in controlling and processing operations associated with the platform. 1100 are associated with helping. In some embodiments, the processor can 1120as any number of processor cores. The processor (or processor cores) can be any type of processor, such as a microprocessor, an embedded processor, a digital signal processor (DSP), a graphics processing unit (GPU), a network processor, a field-programmable gate array, or any other device configured to execute code. The processors can be multithreaded cores in the sense that they can contain more than one hardware thread context (or "logic processor") per core. The processor 1120 It can be implemented as a CISC (Complex Instruction Set Computer) or RISC (Reduced Instruction Set Computer) processor. In some embodiments, the processor can 1120 be configured as an x86 instruction set-compatible processor.

[0045] The storage 1130It can be implemented using any suitable type of digital storage, including, for example, flash memory and / or random access memory (RAM). In some embodiments, the memory can 1130 This includes various layers of storage hierarchy and / or storage caches, as is known to experts. The storage 1130 It can be implemented as a non-persistent storage device, such as, among others, a RAM, dynamic RAM (DRAM), or static RAM (SRAM) device. The storage system 1170can be implemented as a persistent storage device, such as, but not limited to, a hard disk drive (HDD), a solid-state drive (SSD), a USB (Universal Serial Bus) drive, an optical disk drive, a tape drive, an internal storage device, an attached storage device, flash memory, a battery-backed synchronous DRAM (SDRAM), and / or a network-accessible storage device. In some embodiments, the storage can be 1170 Technology to increase storage performance with enhanced protection for valuable digital media when multiple hard drives are included.

[0046] The processor 1120can be designed to run an operating system (OS) 1180, which may include any suitable operating system, such as Google Android (Google Inc., Mountain View, California), Microsoft Windows (Microsoft Corp., Redmond, Washington), Apple OS X (Apple Inc., Cupertino, California), Linux, or a real-time operating system (RTOS). As appreciated with respect to this disclosure, the techniques provided herein may be used regardless of the specific operating system associated with the platform. 1100 is provided, can be implemented, and can therefore also be implemented using any suitable existing or subsequently developed platform.

[0047] The network interface circuit 1140can be any suitable network chip or chipset that provides a wired and / or wireless connection between other components of the setup platform 1100 and / or the network 1194 enabled, thereby enabling the platform 1100This enables communication with other local and / or remote computing systems, servers, cloud-based servers, and / or other resources. Wired communication can conform to existing (or yet-to-be-developed) standards, such as Ethernet. Wireless communication can conform to existing (or yet-to-be-developed) standards, such as cellular communications including LTE (Long Term Evolution), Wireless Fidelity (WiFi), Bluetooth, and / or Near Field Communication (NFC). Examples of wireless networks include wireless local area networks (LANs), wireless personal networks (PPNs), wireless urban networks (NESs), radio networks, and satellite networks.

[0048] The I / O system 1150 can be designed to communicate between different I / O devices and other components of the facility platform. 1100to create an interface. I / O devices can, among other things, control the user interface. 1160 and the microphone array 110 include the user interface 1160The I / O system 1150 may include devices (not shown) such as a speaker, a display, a touchpad, a keyboard, and a mouse, etc. It may include a graphics subsystem designed to process images for display on the display. The graphics subsystem may be, for example, a graphics processing unit (GPU) or a visual processing unit (VPU). An analog or digital interface may be used to provide communication between the graphics subsystem and the display. The interface may be, for example, any HDMI (High Definition Multimedia Interface), DisplayPort, wireless HDMI, and / or any other suitable interface using wireless high-definition compliant techniques. In some embodiments, the graphics subsystem may be integrated into the processor. 1120 or any chipset of the platform 1100be integrated.

[0049] It goes without saying that the various components of the platform 1100 In some embodiments, they can be combined or integrated within a system-on-a-chip (SoC) architecture. In some embodiments, the components can be hardware components, firmware components, software components, or any suitable combination of hardware, firmware, or software.

[0050] The speech preprocessor circuits 120 , 320 are designed to improve the quality of a received speech signal via TF mask-based beamforming (in the case of a microphone array) or TF mask-based noise reduction (in the case of a single microphone), as described previously. The speech preprocessor circuits 120 , 320 can any or all of the items in the Fig. 1-5 and Fig. The seven illustrated circuits / components are included, as described above. These components can be used with a variety of suitable software and / or hardware compatible with the platform. 1100 These components are coupled with, or otherwise form part of, implemented, or otherwise used in conjunction with. These components can additionally or alternatively be implemented with, or otherwise used in conjunction with, user I / O facilities capable of providing information to and receiving information and commands from a user.

[0051] In some embodiments, these circuits can be located locally on the platform. 1100 be installed as in the exemplary embodiment of Fig. 11 is shown. Alternatively, the platform can 1100in a client-server arrangement where at least some of the functionality associated with these circuits is provided by the platform 1100 This is provided using an applet, such as a JavaScript applet, or another downloadable module or set of submodules. Such remotely accessible modules or submodules can be made available in real time in response to a request from a client computer system to access a given server with resources of interest to the client computer system user. In such implementations, the server can connect locally to the network. 1194 are located or are remote from the network via one or more other networks and / or one or more other communication channels 1194be coupled. In some cases, access to resources on a given network or computing system may require login credentials, such as usernames, passwords, and / or compliance with any other appropriate security mechanism.

[0052] In various embodiments, the platform can 1100 It can be implemented as a wireless system, a wired system, or a combination of both. If implemented as a wireless system, the platform can 1100Components and interfaces suitable for communication over a shared wireless medium are included, such as one or more antennas, one or more transmitters, receivers, transceivers, amplifiers, one or more filters, control logic, and so on. An example of a shared wireless medium could be parts of a wireless spectrum, such as the radio frequency spectrum, and so on. If implemented as a wired system, the platform can 1100Components and interfaces include those suitable for communication via wired communication media, such as input / output adapters, physical connectors to connect the input / output adapter to a corresponding wired communication medium, a network interface card (NIC), a disk controller, a video controller, an audio controller, and so on. Examples of wired communication media can include wire, cable metal conductors, a printed circuit board (PCB), a backplane, a switch fabric, a semiconductor material, a twisted-pair wire, a coaxial cable, an optical fiber, and so on.

[0053] Various embodiments can be implemented using hardware elements, software elements, or a combination of both. Examples of hardware elements include processors, microprocessors, circuits, circuit components (for example, transistors, resistors, capacitors, inductors, and so on), integrated circuits, ASICs, programmable logic devices, digital signal processors, FPGAs, logic gates, registers, semiconductor devices, chips, microchips, chipsets, and so on.Examples of software may include software components, programs, applications, computer programs, application programs, system programs, machine programs, operating system software, middleware, firmware, software modules, routines, subroutines, functions, methods, procedures, software interfaces, application program interfaces, instruction sets, computational code, computer code, code segments, computer code segments, words, values, symbols, or any combination thereof. The determination of whether an embodiment is implemented using hardware elements and / or software elements may vary according to any number of factors, such as desired computation rate, performance levels, thermal tolerances, processing cycle budget, input data rates, output data rates, memory resources, data bus speeds, and other design or performance constraints.

[0054] Some embodiments can be described using the terms "coupled" and "connected," along with their derivatives. These terms are not intended to be synonymous. For example, some embodiments may be described using the terms "connected" and / or "coupled" to indicate that two or more elements are in direct physical or electrical contact with each other. However, the term "coupled" can also mean that two or more elements are not in direct contact with each other but nevertheless continue to cooperate or interact.

[0055] The various embodiments disclosed herein can be implemented in various forms of hardware, software, firmware, and / or specialized processors. In one embodiment, for example, at least one non-volatile, machine-readable storage medium contains instructions encoded therein which, when executed by one or more processors, cause one or more of the language preprocessing methods disclosed herein to be implemented. The instructions can be encoded using a suitable programming language, such as C, C++, object-oriented C, Java, JavaScript, Visual Basic .NET, BASIC (Beginner's All-Purpose Symbolic Instruction Code), or alternatively, using user-defined or proprietary instruction sets.The instructions can be provided in the form of one or more computer software applications and / or applets, which are tangiblely implemented on a storage device and can be executed by a computer with any suitable architecture. In one embodiment, the system can be hosted on a given website and implemented, for example, using JavaScript or another suitable browser-based technology. In certain embodiments, the system can, for example, utilize processing resources provided by a network connection. 1194The functionalities disclosed herein can be effectively deployed in an accessible remote computer system. In other embodiments, the functionalities disclosed herein can be integrated into other speech-supported devices and speech-based software applications, such as, for example, automotive control / navigation, smart home management, entertainment, and robotics applications. The computer software applications disclosed herein can include any number of different modules, submodules, or other components with varying functionalities and can provide information to or receive information from other components. These modules can be used, for example, to communicate with input and / or output devices, such as a display screen, a touch-sensitive surface, a printer, and / or any other suitable device.Other components and functionalities not reflected in the illustrations will become apparent with respect to this disclosure, and it will be appreciated that other embodiments are not limited to any particular hardware or software configuration. Thus, the platform can be… 1100 In other embodiments, additional, fewer, or alternative sub-components are used compared to those in the exemplary embodiment of Fig. 11 are included.

[0056] The aforementioned non-volatile, computer-readable medium can be any suitable medium for storing digital information, such as a hard disk, a server, flash memory and / or random access memory (RAM), or a combination of such storage devices. In alternative embodiments, the components and / or modules disclosed herein can be implemented with hardware, including gate-level logic, such as a field-programmable gate array (FPGA), or alternatively, a dedicated semiconductor, such as an application-specific integrated circuit (ASIC). Still other embodiments can be implemented with a microcontroller having a number of input / output ports for receiving and outputting data and a number of embedded routines for performing the various functionalities disclosed herein.It is understood that any suitable combination of hardware, software and firmware can be used, and that other embodiments are not limited to any particular system architecture.

[0057] Some embodiments may, for example, be implemented using a machine-readable medium or article capable of storing an instruction or set of instructions which, when executed by a machine, can cause the machine to perform a method, process, and / or operations according to the embodiments. Such a machine may, for example, include any suitable processing platform, computing platform, computing device, processing device, any suitable computing system, processing system, any suitable computer, process, or the like, and may be implemented using any suitable combination of hardware and / or software.The machine-readable medium or machine-readable article may, for example, include any suitable type of storage unit, storage device, storage article, storage medium, storage device, storage article, storage medium and / or storage unit, such as memory, removable or non-removable media, erasable or non-erasable media, writable or rewritable media, digital or analog media, a hard disk, a floppy disk, a CD-ROM (Compact Disk Read Only Memory), a CD-R (Compact Disk Recordable) memory, a CD-RW (Compact Disk Rewritable) memory, an optical disk, magnetic media, magneto-optical media, removable memory cards or disks, various types of DVD (Digital Versatile Disk), a tape, a cassette, or the like.The instructions may include any suitable type of code, such as source code, compiled code, interpreted code, executable code, static code, dynamic code, encrypted code, and the like, implemented using any suitable high-level or low-level, object-oriented, visual, compiled, and / or interpreted programming language.

[0058] Unless specifically stated otherwise, terms such as "processing," "calculating," "calculating," "determining," or the like may be understood to refer to the action and / or process of a computer or computing system or similar electronic computing device that manipulates and / or transforms data represented as physical entities (for example, electronics) in the registers and / or storage units of the computer system into other data that are likewise represented as physical entities in the registers, storage units, or other such information storage, transmission, or display of the computer system. The embodiments are not limited in this context.

[0059] The terms "circuit" or "circuit," as used herein in any embodiment, are functional and may, for example, individually or in any combination, include a hard-wired circuit, a programmable circuit such as computer processors comprising one or more individual instruction-processing cores, a state machine circuit, and / or firmware storing instructions executed by a programmable circuit. The circuit may include a processor and / or controller designed to execute one or more instructions to perform one or more of the operations described herein. The instructions may be implemented, for example, as an application, software, firmware, etc., designed to cause the circuit to perform any of the foregoing operations.Software can be implemented as a software package, code, instructions, records, and / or data recorded on a computer-readable storage device. Software can be implemented to contain any number of processes, and processes, in turn, can be implemented to contain any number of threads, etc., in a hierarchical manner. Firmware can be implemented as code, instructions or instruction sets, and / or data that are hard-coded (e.g., persistent) in storage devices. A circuit can be implemented, collectively or individually, as a circuit that forms part of a larger system, for example, an integrated circuit (IC), an application-specific integrated circuit (ASIC), a system-on-a-chip (SoC), a desktop computer, a laptop computer, a tablet computer, a server, a smartphone, etc.Other embodiments may be implemented as software executed by a programmable logic controller (PLC). In such cases, the terms "circuit" or "circuit" are intended to encompass a combination of software and hardware, such as a programmable logic controller or a processor capable of executing the software. As described herein, various embodiments may be implemented using hardware elements, software elements, or any combination thereof. Examples of hardware elements include processors, microprocessors, circuits, and circuit components (e.g., transistors, transistors, etc.).transistors, resistors, capacitors, inductors and so on), integrated circuits, application-specific integrated circuits (ASIC), programmable logic devices (PLDs), digital signal processors (DSPs), a field-programmable gate array (FPGA), logic gates, registers, a semiconductor device, chips, microchips, chipsets and so on.

[0060] Numerous specific details have been set forth herein to provide a comprehensive understanding of the embodiments. However, a person skilled in the art will understand that the embodiments can be implemented without these specific details. In other cases, well-known processes, components, and circuits have not been described in detail, so that the embodiments are not rendered unclear. It may be appreciated that the specific structural and functional details disclosed herein may be representative and do not necessarily limit the scope of protection of the embodiments. In addition, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described herein, even though the subject matter has been described in language specific to structural features and / or methodological actions.Instead, the specific features and actions described herein are revealed as exemplary forms for implementing the claims. Further examples of implementation

[0061] The following examples relate to further embodiments, from which numerous permutations and configurations become apparent.

[0062] Example 1 is at least one non-volatile, computer-readable storage medium containing instructions encoded on it which, when executed by one or more processors, cause a process for improving a speech signal to be executed, wherein the process comprises: performing a denaturation reduction (DRV) processing on a first and second signal derived from a first and second signal, respectively.a second microphone is received, wherein the first and second signals each comprise a combination of speech and noise; generating a first and second time-frequency mask (TFM), wherein both the first and second TFM are associated with a corresponding one of the DRV-processed first and second signals, wherein both the first and second TFM comprise elements that indicate a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed first and second signals; generating steering vectors based on speech covariance matrices and noise covariance matrices, wherein the first and second TFM are used to select speech components of the DRV-processed first and second signals, respectively.to filter the second signal to calculate the speech covariance matrices and to filter noise components of the DRV-processed first and second signals to calculate the noise covariance matrices; and to perform beamforming on the DRV-processed first and second signals based on the steering vectors to generate a single-channel speech signal.

[0063] Example 2 includes the subject of Example 1, wherein the process further includes performing an improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal.

[0064] Example 3 includes the subject of Examples 1 or 2, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0065] Example 4 includes the subject of one of Examples 1-3, where the beam shaping uses a beam shaping with a generalized eigenvalue with blind-analytic normalization.

[0066] Example 5 includes the subject of one of Examples 1-4, wherein the first and second signals are received from a far field of the first and second microphones.

[0067] Example 6 includes the subject of one of Examples 1-5, wherein the process further includes providing the single-channel speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application.

[0068] Example 7 is a system for preprocessing a speech signal, wherein the system comprises: a noise reduction (DRV) circuit for performing DRV processing on multiple signals received by an array of microphones, the signals comprising a combination of speech and noise; a time-frequency mask (TFM) estimation circuit for generating multiple TFMs, each of which is associated with one of the DRV-processed signals, the TFMs comprising elements that specify a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, the TFM generation being based on an application of a recurrent neural network (RNN) to the DRV-processed signals;A beamforming circuit for generating steering vectors based on speech covariance matrices and noise covariance matrices, wherein the TFMs are used to filter speech components of the DRV-processed signals for calculating the speech covariance matrices and to filter noise components of the DRV-processed signals for calculating the noise covariance matrices; and wherein the beamforming circuit is further intended to perform beamforming on the multiple DRV-processed signals based on the steering vectors in order to generate a single-channel speech signal.

[0069] Example 8 includes the subject of Example 7, further comprising a post-filtering circuit for performing an improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal.

[0070] Example 9 includes the subject of Examples 7 or 8, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0071] Example 10 includes the subject of one of Examples 7-9, where the beam shaping uses a beam shaping with a generalized eigenvalue with blind-analytic normalization.

[0072] Example 11 includes the subject of one of Examples 7-10, wherein the multiple received signals are received from a far field of microphones.

[0073] Example 12 includes the subject of one of Examples 7-12, wherein the single-channel speech signal is provided to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application.

[0074] Example 13 is at least one non-volatile, computer-readable storage medium containing instructions encoded on it which, when executed by one or more processors, cause a process to enhance a speech signal to be executed, wherein the process comprises: performing a denaturation reduction (DRV) processing on a signal received from a microphone, wherein the signal comprises a combination of speech and noise; generating a time-frequency mask (TFM) associated with the DRV-processed signal, wherein the TFM comprises elements that specify a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed signal;and performing noise reduction on the DRV-processed signal to produce an improved speech signal, the noise reduction using the TFM to identify noise to be filtered.

[0075] Example 14 includes the subject of Example 13, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0076] Example 15 includes the subject of Examples 13 or 14, wherein the received signal is received from a far field of the microphone.

[0077] Example 16 includes the subject of one of Examples 13-15, wherein the process further includes providing the enhanced speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application.

[0078] Example 17 is a processor-implemented procedure for training a recursive neural network (RNN) to generate a time-frequency mask (TFM), wherein the procedure comprises: selecting, by a processor-based system, a clear speech signal sample from a speech database; selecting, by the processor-based system, a reverberation filter from a filter database; selecting, by the processor-based system, a noise signal sample from a noise database; scaling, by the processor-based system, the noise signal sample based on a selected signal-to-noise ratio (SNR); applying, by the processor-based system, the reverberation filter to the clear speech signal sample and adding the scaled noise signal sample to the result of the reverberation filter application to generate a noisy speech signal;Generating, by the processor-based system, an estimated TFM based on an application of the RNN to features extracted from the noisy speech signal; generating, by the processor-based system, a target TFM based on a ratio of features extracted from the clear speech signal sample to the features extracted from the noisy speech signal; and training, by the processor-based system, the RNN based on a calculated error between the estimated TFM and the target TFM.

[0079] Example 18 includes the subject matter of Example 17, further comprising iterating the procedure for training the RNN to generate the TFM based on additional selections of the clear speech signal sample, the reverberation filter, the noise signal sample and the SNR.

[0080] Example 19 includes the subject of Examples 17 or 18, wherein the reverberation filter is intended to simulate far-field environmental effects on a signal picked up by a microphone in the far field of the microphone.

[0081] Example 20 includes the subject of one of Examples 17-19, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0082] Example 21 is a system for improving a speech signal, wherein the system comprises: means for performing a reduction in reverberation (DRV) processing on a first and second signal derived from a first and second signal, respectively.a second microphone, wherein the first and second signals each comprise a combination of speech and noise; means for generating a first and second time-frequency mask (TFM), wherein both the first and second TFMs are associated with a corresponding one of the DRV-processed first and second signals, wherein both the first and second TFMs comprise elements that specify a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed first and second signals; means for generating steering vectors based on speech covariance matrices and noise covariance matrices, wherein the first and second TFMs are used to select speech components of the DRV-processed first and second signals, respectively.to filter the second signal for the calculation of the speech covariance matrices and to filter noise components of the DRV-processed first and second signals for the calculation of the noise covariance matrices; and means for performing beamforming on the DRV-processed first and second signals based on the steering vectors to generate a single-channel speech signal.

[0083] Example 22 includes the subject matter of Example 21, wherein the system further comprises means for performing improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal.

[0084] Example 23 includes the subject of Example 21 or 22, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0085] Example 24 includes the subject of one of Examples 21-23, where the beam shaping uses a beam shaping with a generalized eigenvalue with blind-analytic normalization.

[0086] Example 25 includes the subject of one of Examples 21-24, wherein the first and second signals are received from a far field of the first and second microphones.

[0087] Example 26 includes the subject matter of one of Examples 21-25, wherein the system further includes means for providing the single-channel speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application.

[0088] Example 27 is a system for improving a speech signal, comprising: means for performing reverberation reduction (DRV) processing on a signal received from a microphone, wherein the signal comprises a combination of speech and noise; means for generating a time-frequency mask (TFM) associated with the DRV-processed signal, wherein the TFM comprises elements indicating a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed signal; and means for performing noise reduction on the DRV-processed signal to produce an improved speech signal, wherein the noise reduction uses the TFM to identify noise to be filtered out.

[0089] Example 28 includes the subject of Example 27, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0090] Example 29 includes the subject of Examples 27 or 28, wherein the received signal is received from a far field of the microphone.

[0091] Example 30 includes the subject matter of one of Examples 27-29, wherein the process further includes means for providing the enhanced speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application.

[0092] Example 31 is a system for training a recursive neural network (RNN) to generate a time-frequency mask (TFM), the system comprising: means of selecting a clear speech signal sample from a speech database; means of selecting a reverberation filter from a filter database; means of selecting a noise signal sample from a noise database; means of scaling the noise signal sample based on a selected signal-to-noise ratio (SNR); means of applying the reverberation filter to the clear speech signal sample and adding the scaled noise signal sample to the result of the reverberation filter application to generate a noisy speech signal; means of generating an estimated TFM based on an application of the RNN to features extracted from the noisy speech signal;Means for generating a target TFM based on a ratio of features extracted from the clear speech signal sample to the features extracted from the noisy speech signal; and means for training the RNN based on a calculated error between the estimated TFM and the target TFM.

[0093] Example 32 includes the subject matter of Example 31, further comprising means for iterating the procedure for training the RNN to generate the TFM based on additional selections of the clear speech signal sample, the reverberation filter, the noise signal sample and the SNR.

[0094] Example 33 includes the subject of Examples 31 or 32, wherein the reverberation filter is intended to simulate far-field environmental effects on a signal picked up by a microphone in the far field of the microphone.

[0095] Example 34 includes the subject of one of Examples 31-33, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function, and a fifth affine layer with a sigmoid activation function.

[0096] The terms and expressions used herein are employed for descriptive purposes only and not for limitation. The use of such terms and expressions is not intended to exclude any equivalents of the features (or parts thereof) presented and described, and it is understood that various modifications to the scope of protection of the claims are possible. Accordingly, it is intended that the claims cover all such equivalents. Various features, aspects, and embodiments have been described herein. These features, aspects, and embodiments can be combined, varied, and modified, as those skilled in the art will understand. The present disclosure should therefore be regarded as including such combinations, variations, and modifications.It is intended that the scope of protection of the present disclosure shall not be limited by this detailed description, but instead by the claims attached herein. Future applications claiming priority over this application may claim the disclosed subject matter in a different manner and may generally include any set of one or more elements as variously disclosed herein or otherwise set forth. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 62 / 673238

[0001]

Claims

[1] Non-volatile computer-readable storage medium or non-volatile computer-readable storage media containing instructions encoded on them which, when executed by one or more processors, cause a process to enhance a speech signal to be executed, wherein the process comprises: Performing a reduction in reverberation (DRV) processing on a first and second signal received from a first and second microphone respectively, wherein the first and second signals each comprise a combination of speech and noise; Generating a first and second time-frequency mask (TFM), wherein both the first and second TFM are associated with a corresponding first and second signal of the DRV-processed signal, wherein both the first and second TFM include elements that indicate a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed first and second signal; Generating steering vectors based on speech covariance matrices and noise covariance matrices, wherein the first and second TFMs are used to filter speech components of the DRV-processed first and second signals, respectively, to calculate the speech covariance matrices, and to filter noise components of the DRV-processed first and second signals to calculate the noise covariance matrices; and Performing beamforming on the DRV-processed first and second signals based on the steering vectors to generate a single-channel speech signal. [2] Computer-readable storage medium according to claim 1, wherein the process further comprises performing an improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal. [3] Computer-readable storage medium according to one of claims 1-2, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function and a fifth affine layer with a sigmoid activation function. [4] Computer-readable storage medium according to one of claims 1-3, wherein the beam shaping uses a beam shaping with a generalized eigenvalue with blind-analytic normalization. [5] Computer-readable storage medium according to one of claims 1-4, wherein the first and second signals are received by a far field of the first and second microphone. [6] Computer-readable storage medium according to any one of claims 1-5, wherein the process further comprises providing the single-channel speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application. [7] System for preprocessing a speech signal, the system comprising: a noise reduction (DRV) circuit for performing DRV processing on multiple signals received by an array of microphones, wherein the signals comprise a combination of speech and noise; a time-frequency mask (TFM) estimation circuit for generating multiple TFMs, wherein each of the TFMs is associated with one of the DRV-processed signals, the TFMs comprising elements that indicate a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed signals; a beamforming circuit for generating steering vectors based on speech covariance matrices and noise covariance matrices, wherein the TFMs are used to filter speech components of the DRV-processed signals to calculate the speech covariance matrices and to filter noise components of the DRV-processed signals to calculate the noise covariance matrices; and the beamforming circuit is further intended to perform beamforming on the multiple DRV-processed signals based on the steering vectors in order to generate a single-channel speech signal. [8] System according to claim 7, further comprising a post-filtering circuit for performing an improved minimum-controlled recursive averaging to reduce stationary noise components of the single-channel speech signal. [9] System according to one of claims 7-8, wherein the RNN comprises a first affine layer with a tangent hyperbolic activation function, a second affine layer with a tangent hyperbolic activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function and a fifth affine layer with a sigmoid activation function. [10] System according to one of claims 7-9, wherein the beam shaping uses a beam shaping with generalized eigenvalue with blind-analytic normalization. [11] System according to one of claims 7-10, wherein the multiple received signals are received from a far field of the microphones. [12] System according to one of claims 7-11, wherein the single-channel speech signal is provided to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application. [13] Non-volatile computer-readable storage medium or non-volatile computer-readable storage media containing instructions encoded on them which, when executed by one or more processors, cause a process to enhance a speech signal to be executed, wherein the process comprises: Performing a reduction in reverberation (DRV) processing on a signal received from a microphone, where the signal comprises a combination of speech and noise; Generating a time-frequency mask (TFM) associated with the DRV-processed signal, wherein the TFM comprises elements that indicate a probability that a time-frequency component of the DRV-processed signal associated with that TFM element contains speech, wherein the TFM generation is based on an application of a recurrent neural network (RNN) to the DRV-processed signal; and Performing noise reduction on the DRV-processed signal to produce an improved speech signal, with the noise reduction using the TFM to identify noise to be filtered. [14] Computer-readable storage medium according to claim 13, wherein the RNN comprises a first affine layer with a hyperbolic tangent activation function, a second affine layer with a hyperbolic tangent activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function and a fifth affine layer with a sigmoid activation function. [15] Computer-readable storage medium according to one of claims 13-14, wherein the received signal is received from a far field of the microphone. [16] Computer-readable storage medium according to one of claims 13-15, wherein the process further comprises providing the enhanced speech signal to an ASR (Automatic Speech Recognition) engine and / or a keyword recognition application and / or a Wake-on-Voice (WoV) application and / or a VOIP (Voice over Internet Protocol) application. [17] Processor-implemented method for training a recursive neural network (RNN) to generate a time-frequency mask (TFM), wherein the method includes: Selecting, by a processor-based system, a clear speech signal sample from a speech database; Selecting, by the processor-based system, a reverberation filter from a filter database; Selecting, by the processor-based system, a noise signal sample from a noise database; Scaling, by the processor-based system, of the noise signal sample based on a selected signal-to-noise ratio (SNR); Applying the reverberation filter to the clear speech signal sample using the processor-based system and adding the scaled noise signal sample to the result of the reverberation filter application to generate a noisy speech signal; Generating, through the processor-based system, an estimated TFM based on an application of the RNN to features extracted from the noisy speech signal; Generate, through the processor-based system, a target TFM based on a ratio of features extracted from the clear speech signal sample to the features extracted from the noisy speech signal; and Training, through the processor-based system, of the RNN based on a calculated error between the estimated TFM and the target TFM. [18] Method according to claim 17, further comprising iterating the method for training the RNN to generate the TFM based on additional selections of the clear speech signal sample, the reverberation filter, the noise signal sample and the SNR. [19] Method according to one of claims 17-18, wherein the reverberation filter is intended to simulate far-field environmental effects on a signal recorded by a microphone in the far field of the microphone. [20] Method according to one of claims 17-19, wherein the RNN comprises a first affine layer with a tangent hyperbolic activation function, a second affine layer with a tangent hyperbolic activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function and a fifth affine layer with a sigmoid activation function. [21] Device for training a recursive neural network (RNN) to generate a time-frequency mask (TFM), the device comprising: Means of selecting a clear speech signal sample from a speech database; Means of selecting a reverberation filter from a filter database; Means of selecting a noise signal sample from a noise database; Means for scaling the noise signal sample based on a selected signal-to-noise ratio (SNR); Means of applying the reverberation filter to the clear speech signal sample and adding the scaled noise signal sample to the result of the reverberation filter application in order to generate a noisy speech signal; Means of generating an estimated TFM based on an application of the RNN to features extracted from the noisy speech signal; Means of generating a target TFM based on a ratio of features extracted from the clear speech signal sample to the features extracted from the noisy speech signal; and Means of training the RNN based on a calculated error between the estimated TFM and the target TFM. [22] Device according to claim 21, further comprising means for iterating the training of the RNN to generate the TFM based on additional selections of the clear speech signal sample, the reverberation filter, the noise signal sample and the SNR. [23] Device according to one of claims 21-22, wherein the reverberation filter is intended to simulate far-field environmental effects on a signal that is picked up by a microphone in the far field of the microphone. [24] Device according to one of claims 21-23, wherein the RNN comprises a first affine layer with a tangent hyperbolic activation function, a second affine layer with a tangent hyperbolic activation function, a third LSTM (long short-term memory) layer with a sigmoid activation function, a fourth LSTM (long short-term memory) layer with a sigmoid activation function and a fifth affine layer with a sigmoid activation function.