Method, computer device, apparatus and storage medium for generating target speech

Abstract

A method and apparatus including computer code for generating an enhanced target speech from audio data, the method including receiving audio data corresponding to one or more speakers; simultaneously generating, based on the audio data, an estimated target speech, an estimated noise, and an estimated echo using a jointly trained complex ratio mask; predicting, based on the estimated target speech, the estimated noise, and the estimated echo, frame-level multi-tap time-frequency (T-F) spatio-temporal echo filter weights using a trained neural network model; and predicting, based on the frame-level multi-tap T-F spatio-temporal echo filter weights, the enhanced target speech.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Application No. 17 / 455,497, filed November 18, 2021, with the United States Patent and Trademark Office, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] Embodiments of this disclosure relate to data and / or signal processing, more specifically to speech processing, and even more particularly to a method, computer apparatus, device, and non-transitory computer-readable storage medium for generating enhanced target speech from audio data. Background Technology

[0004] Speech enhancement and speech separation have attracted considerable research attention because they are frequently encountered in real-world scenarios such as virtual meetings, intelligent speakers, and communication systems. Classical approaches to speech enhancement and speech separation use a cascade of techniques to build systems. However, this cascade of techniques and methods is often inconsistent and cannot achieve an optimal solution.

[0005] Errors generated at the start of the system can propagate to subsequent stages and eventually permeate the entire system. Due to some unrealistic assumptions and numerical problems, they may have high levels of residual noise and echo. Some methods may exhibit nonlinear distortion, which is detrimental to backend automatic speech recognition (ASR) systems. Summary of the Invention

[0006] According to an embodiment, a method for generating enhanced target speech from audio data, executed by a computing device, is provided, characterized in that the method includes: receiving audio data corresponding to one or more speakers; simultaneously generating estimated target speech, estimated noise, and estimated echo based on the audio data using a jointly trained complex ratio mask; predicting frame-level multi-tap time-frequency (TF) spatiotemporal echo filter weights based on the estimated target speech, the estimated noise, and the estimated echo using a trained neural network model; and predicting enhanced target speech based on the frame-level multi-tap TF spatiotemporal echo filter weights.

[0007] According to an embodiment, a computer device is provided, characterized in that the computer device includes: one or more computer-readable non-transitory storage media configured to store computer program code; and one or more computer processors configured to access the computer program code and execute the above-described method for generating enhanced target speech from audio data as instructed by the computer program code.

[0008] According to an embodiment, an apparatus for generating enhanced target speech from audio data is provided, characterized in that the apparatus comprises: a first receiving unit configured to receive audio data corresponding to one or more speakers; a first generating unit configured to simultaneously generate estimated target speech, estimated noise, and estimated echo based on the audio data using a jointly trained complex ratio mask; a first prediction unit configured to predict frame-level multi-tap time-frequency (TF) spatiotemporal echo filter weights based on the estimated target speech, the estimated noise, and the estimated echo using a trained neural network model; and a second prediction unit configured to predict enhanced target speech based on the frame-level multi-tap TF spatiotemporal echo filter weights.

[0009] According to an embodiment, a non-transitory computer-readable storage medium storing storage instructions is provided, characterized in that the instructions include: one or more instructions that, when executed by one or more processors of a device for generating enhanced target speech from audio data, cause the one or more processors to perform the aforementioned method for generating enhanced target speech from audio data.

[0010] The method, computer device, apparatus, and non-transitory computer-readable storage medium of the present invention for generating enhanced target speech from audio data utilize a time-frequency (TF) mask estimator based on an end-to-end neural network (NN) to significantly reduce the word error rate (WER) of the ASR system with less distortion and improve the residual noise problem. Furthermore, the use of a multi-head attention RNN model effectively addresses the nonlinearity and time shift present in the echo. Attached Figure Description

[0011] These and other objects, features, and advantages will become apparent from the following detailed description of illustrative embodiments, which should be read in conjunction with the accompanying drawings. The various features in the drawings are not to scale, as the illustrations are provided for clarity in order to facilitate understanding by those skilled in the art in conjunction with the detailed description.

[0012] In the picture:

[0013] Figure 1 A networked computer environment according to an embodiment of this disclosure is shown.

[0014] Figure 2 This is an exemplary voice processing system according to an embodiment of the present disclosure.

[0015] Figure 3 This is an operational flowchart illustrating speech separation and / or enhancement according to embodiments of the present disclosure.

[0016] Figure 4 This is an operational flowchart illustrating another embodiment of speech separation and / or enhancement according to this disclosure.

[0017] Figure 5 This is an exemplary implementation of speech separation and / or speech enhancement in a real-world setting.

[0018] Figure 6 It is based on at least one embodiment. Figure 1 A block diagram depicting the internal and external components of a computer and server.

[0019] Figure 7 It includes, according to at least one embodiment. Figure 1 A block diagram illustrating a cloud computing environment for a computer system.

[0020] Figure 8 It is based on at least one embodiment. Figure 7 A block diagram illustrating the functional layers of an illustrative cloud computing environment. Detailed Implementation

[0021] This document discloses detailed embodiments of the claimed structures and methods; however, it is to be understood that the disclosed embodiments are merely illustrative of the claimed structures and methods, which can be implemented in various forms. These structures and methods can be implemented in many different forms and should not be construed as limited to the exemplary embodiments set forth herein. Rather, these exemplary embodiments are provided so that this disclosure will be thorough and complete and will fully convey the scope to those skilled in the art. Details of well-known features and techniques may be omitted in the description to avoid unnecessarily obscuring the presented embodiments.

[0022] This disclosure relates to the field of data processing, and more particularly to speech recognition, speech separation, and / or speech enhancement. This disclosure provides systems, methods, and computer programs for separating the speech of a target speaker from a noisy speech mixture using a fully neural network approach. Therefore, some embodiments have the ability to improve the computational field by allowing improved speech enhancement, speech separation, and / or dereverberation tasks performed by a computer. Furthermore, the disclosed methods, systems, and computer-readable media can be used to improve the performance of automatic speech recognition in fields such as hearing aids and communications.

[0023] Speech enhancement and speech separation methods have attracted considerable research attention. Techniques such as speech denoising, dereverberation, beamforming, and acoustic echo cancellation (AEC) are frequently used to improve the target speech quality and intelligibility in real-world front-end systems. However, they are often used in pipelines, leading to suboptimal solutions by combining different techniques and / or modules. Errors generated at the beginning of the pipeline propagate downstream.

[0024] Several signal processing-based methods have been developed to integrate beamformers (e.g., MVDR) and acoustic echo cancellation (AEC) algorithms from related technologies. For example, weighted power minimization distortionless response convolutional beamformers (WPD) are used for joint separation, denoising, and dereverberation. However, residual noise levels remain high in these methods, particularly in cases of low signal-to-noise ratios or speech overlap. Furthermore, matrix inversion of the noise covariance matrix and principal component analysis (PCA) of the target speech covariance matrix involved in MVDR and neural networks are unstable and can lead to suboptimal results. According to embodiments of this disclosure, matrix inversion and principal component analysis (PCA) involved in beamformers (e.g., MVDR) from related technologies can be implicitly replaced by a robust multi-tap multi-head attention RNN model that utilizes weighted information from all previous frames locally and globally, and does not require any heuristic update factors between consecutive frames as needed in methods based on recursive MVDR beamformers.

[0025] Recently, some deep learning-assisted joint optimization methods have been proposed to address the problems mentioned above. However, due to flawed assumptions and mathematical instabilities, these neural network-based methods exhibit high levels of residual noise and echo. Even end-to-end neural network-based methods for jointly addressing AEC and denoising-related problems suffer from some nonlinear distortion, which is detrimental to backend automatic speech recognition (ASR) systems.

[0026] This disclosure aims to address the speech processing problems mentioned above, including speech separation and / or speech enhancement. With the resurgence of neural networks, better objective performance can be achieved using deep learning methods. This disclosure aims to reduce noise while maintaining the target speech without distortion. Since block-level or speech-level beamforming weights are not optimal for noise reduction, this disclosure utilizes its time-frequency (TF) mask estimator based on an end-to-end neural network (NN) to significantly reduce the word error rate (WER) of ASR systems with less distortion and improve the residual noise problem. Furthermore, using a multi-head attention RNN model, this disclosure effectively addresses the nonlinearity and time shift present in echoes.

[0027] This disclosure presents a jointly trained front-end system based on a multi-head attention recurrent neural network (RNN) model for joint AEC, denoising, dereverberation, and separation. The echo reference and estimated signal covariance matrix are jointly modeled to effectively remove echo. A multi-head attention RNN is also proposed to learn multi-tap and multi-channel cross-correlation. A spatiotemporal echo filter is then predicted to remove echo, interfering speech, background noise, and reverberation, and to generate enhanced target speech. This disclosure achieves less residual noise and residual echo. It also results in less nonlinear distortion compared to some purely "black box" methods.

[0028] This disclosure discloses a novel unified front-end framework (ADL-UFE) based on full deep learning. According to embodiments of this disclosure, a speech processing system can simultaneously perform joint optimization of acoustic echo cancellation (AEC), speech denoising, speech separation, and speech dereverberation to estimate speech, noise, and echo. According to embodiments, joint optimization can be performed simultaneously using a jointly trained mask estimator, which may include a complex ratio filter.

[0029] According to the implementation, the estimated target speech, estimated noise, and estimated echo can be multi-tap expanded before being input into the multi-head self-attention RNN to better utilize cross-frame correlations in the echo and target speech. Furthermore, the echo reference and estimated target speech can be modeled together and / or simultaneously using the multi-head attention RNN to learn frame-level TF spatiotemporal echo filter weights. The frame-level TF spatiotemporal echo filter weights, combined with the audio data and echo reference, can be used to generate enhanced target speech from noisy audio data.

[0030] According to embodiments of this disclosure, a mask estimator based on a Convolutional 1D Gated Recurrent Unit (RNN) can be used to estimate echo, speech, and noise from noisy audio data. According to embodiments of this disclosure, a multi-tap multi-head attention RNN beamformer (RNNBF) can be used to predict frame-level multi-tap TF spatiotemporal echo filter weights. The RNNBF can use the estimated target speech, estimated noise, and estimated echo to predict the frame-level multi-tap TF spatiotemporal echo filter weights. The frame-level multi-tap TF spatiotemporal echo filter weights can be used to generate enhanced target speech from noisy audio data.

[0031] RNNs (e.g., RNNBF) are well-suited for real-world applications such as virtual meetings, smart cars, and hearing aids in estimating speech, echoes, and noise from noisy data and generating enhanced target speech, because they significantly improve the performance of ASR, wake word detection, hearing aids, and communications.

[0032] Various aspects are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer-readable media according to various embodiments. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0033] Figure 1 A functional block diagram of a networked computer environment is shown, illustrating a speech processing system 100 (hereinafter referred to as the "System") for separating the speech of a target speaker using a fully neural network approach. It should be understood that... Figure 1 This illustration provides only one possible implementation and does not imply any limitation on the environments in which different implementations can be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0034] System 100 may include computer 102 and server computer 114. Computer 102 may communicate with server computer 114 via communication network 110 (hereinafter referred to as "network"). Computer 102 may include processor 104 and software program 108 stored on data storage device 106, and computer 102 is capable of interfacing with a user and communicating with server computer 114. (Referring to the following...) Figure 6 The computer 102 discussed may include internal components 800A and external components 900A, and the server computer 114 may include internal components 800B and external components 900B. The computer 102 may be, for example, a mobile device, telephone, personal digital assistant, netbook, laptop computer, tablet computer, desktop computer, or any type of computing device capable of running programs, accessing a network, and accessing a database.

[0035] As shown below Figure 5 and Figure 6 The server computer 114 discussed can also operate in cloud computing service models such as Software as a Service (SaaS), Platform as a Service (PaaS), or Infrastructure as a Service (IaaS). The server computer 114 can also reside in cloud computing deployment models such as private cloud, community cloud, public cloud, or hybrid cloud.

[0036] The server computer 114, which can be used for speech processing, is capable of running a speech processing program 116 (hereinafter referred to as the "program") that can interact with the database 112. The following is about... Figure 3 The speech processing method will be explained in more detail below. In one embodiment, computer 102 may operate as an input device including a user interface, while program 116 may run primarily on server computer 114. In an alternative embodiment, program 116 may run primarily on one or more computers 102, while server computer 114 may be used to process and store the data used by program 116. It should be noted that program 116 may be a standalone program or may be integrated into a larger speech processing program.

[0037] However, it should be noted that in some instances, processing of program 116 can be shared in any proportion between computer 102 and server computer 114. In another embodiment, program 116 can run on more than one computer, server computer, or a combination of computers and server computers, such as multiple computers 102 communicating with a single server computer 114 across network 110. In another embodiment, for example, program 116 can run on multiple server computers 114 communicating with multiple client computers across network 110. Alternatively, the program can run on a network server communicating with both the server and multiple client computers across a network.

[0038] Network 110 may include wired connections, wireless connections, fiber optic connections, or combinations thereof. Typically, network 110 may be any combination of connections and protocols that support communication between computer 102 and server computer 114. Network 110 may include various types of networks, such as local area networks (LANs), wide area networks (WANs) such as the Internet, telecommunications networks such as the Public Switched Telephone Network (PSTN), wireless networks, public switched networks, satellite networks, cellular networks (e.g., fifth-generation (5G), Long Term Evolution (LTE), third-generation (3G), Code Division Multiple Access (CDMA), etc.), public land mobile networks (PLMNs), metropolitan area networks (MANs), private networks, ad hoc networks, intranets, fiber optic-based networks, and / or combinations of these or other types of networks.

[0039] Figure 1 The number and arrangement of devices and networks shown are provided as examples. In practice, there may be... Figure 1 The equipment and / or network shown is compared to additional equipment and / or networks, fewer equipment and / or networks, different equipment and / or networks, or equipment and / or networks with different arrangements. Furthermore, Figure 1 The two or more devices shown can be implemented within a single device, orFigure 1 The single device shown can be implemented as multiple distributed devices. Alternatively or alternatively, a group of devices in system 100 (e.g., one or more devices) can perform one or more functions described as being performed by another group of devices in system 100.

[0040] While this disclosure has described several exemplary embodiments, variations, substitutions, and various alternative equivalents fall within the scope of this disclosure. Therefore, it will be appreciated that those skilled in the art will be able to conceive of many systems and methods that, although not expressly shown or described herein, embody the principles of this disclosure and are therefore within its spirit and scope.

[0041] Figure 2 An exemplary speech processing system 200 for speech separation and / or speech enhancement is shown, illustrating one or more embodiments of the present disclosure.

[0042] like Figure 2 As shown, the exemplary speech processing system includes audio data 210, a jointly trained complex ratio mask 220, a multi-head self-attention RNN 230, and enhanced target speech 270.

[0043] Audio data 210 may include noisy speech mixtures 254 corresponding to one or more speakers, echo references 252 corresponding to one or more speakers, and speaker-independent features (e.g., logarithmic power spectrum (LPS) and interaural phase difference (IPD)) and speaker-related features (e.g., directional feature d(θ)).

[0044] According to embodiments of this disclosure, the audio data includes a noisy mixture of speech corresponding to one or more speakers, without information about the locations of the different speakers. However, with sufficient context about the situation, a coarse estimate of the global direction of arrival (DOA) can be made. As an example, the exact locations of the different speakers in the car may not be known. However, a coarse estimate of the global DOA for four acoustic regions in the car can be made, such as... Figure 5 As shown. Furthermore, if the position of the microphone and / or microphone array can be estimated, the position-guided directional feature (DF)d(θ) can be used to extract target speech from a specific DOA. DF can calculate the cosine similarity between the target steering vector v(θ) and the IPD. The estimated mask or filter will help in calculating the speech, noise, and echo mentioned above using the audio data.

[0045] According to embodiments of this disclosure, the noisy speech mixture 254 may include target speech, echo signals, and interference noise components on multiple channels.

[0046] Considering the noisy speech mixture y = [y1, y2, ..., Y] M ] T Recorded using an M-channel near-end microphone array. Let s represent the target speech, d represent the echo signal, and n represent the interference noise (including interference speech and background noise) of the M channels, then has

[0047] y(t)=s(t)*h target Equation (1) is (t)+d(t)+n(t).

[0048] Among them, h target (t) could be the target speaker-to-microphone array room impulse response (RIR). Further consideration is needed.

[0049] d(t)=e(t)*h echo (t)+d nonlinear (t) Equation (2).

[0050] Where e(t) can be the echo reference signal of the distant speaker, h echo (t) can be the room impulse response (RIR) or acoustic echo path from a loud speaker to the microphone array, and d nonlinear (t) can be a nonlinear component introduced by a loud speaker.

[0051] A signal can be transformed into the time-frequency domain using the Short-Time Fourier Transform (STFT):

[0052] Y(t,f)=S(t,f)·H target Equation (3) is (t, f) + D(t, f) + N(t, f).

[0053] Here, (t, f) can indicate the time and frequency exponent of the acoustic signal in the time-frequency (TF) domain.

[0054] According to embodiments of this disclosure, enhanced, separated, and / or dereverberated target speech S(t,f) can be predicted from a near-end M-channel mixture Y(t,f), wherein the echo reference signal E(t,f) is at least a portion of the input.

[0055] Generating and / or predicting enhanced target speech can be challenging. In some cases, the echo reference signal E(t,f) may differ from the echo signal D(t,f) received by the near-end M-channel microphone array because D(t,f) can be generated by linear distortion (h). echo (t) and nonlinear distortion (d) nonlinear (t)) consists of. In other cases, it may be necessary to remove the room impulse response H. target(t, f) and the noise N(t, f) may contain interfering speech from multiple random locations and diffused background noise.

[0056] According to implementation methods, some approaches to address the aforementioned problems may include using a mask-based minimum variance distortionless response (MVDR) beamformer, wherein the separated speech can be obtained as

[0057]

[0058] in, Let f represent the MVDR weights at the frequency exponent f, and H represent the Hermitian operator. The goal of an MVDR beamformer can be to minimize noise power while maintaining undistorted target speech, which can be formulated as:

[0059]

[0060] Where, Φ NN The covariance matrix that can represent the noise power density spectrum (PSD), and The steering vector of the target speech can be represented. Different solutions can be used to derive the MVDR beamforming weights. One solution is based on the steering vector and can be derived by applying principal component analysis (PCA) to the speech covariance matrix. Another solution is based on reference channel selection.

[0061]

[0062] Where, Φ SS The covariance matrix of the speech PSD can be represented, and This can be a one-hot vector of the selected reference microphone channel. In some cases, matrix inversion and PCA are unstable, especially when using jointly trained neural networks.

[0063] Complex ratio masks (referred to as cRMs) can be used to accurately estimate target speech with less phase distortion, which is beneficial to human listeners. According to embodiments of this disclosure, the estimated speech... and speech covariance matrix Φ SS It can be calculated as follows:

[0064]

[0065] Here, * can represent complex number multiplication, and cRM S This can represent the estimated covariance matrix (cRM) for a speech target. The noise covariance matrix Φ NNThis can be obtained in a similar manner. However, the covariance matrix Φ derived here may be at the utterance level, which is not optimal for each frame, resulting in high levels of residual noise. Matrix inversion involved in beamformers of related techniques (e.g., MVDR) always suffers from numerical instability problems.

[0066] According to embodiments of this disclosure, multi-tap multi-head self-attention RNNs can be used to unify front-end signal processing and overcome the problems mentioned above. Matrix inversion and principal component analysis (PCA) involved in beamformers of related art (e.g., MVDR) can be implicitly replaced by a powerful multi-tap multi-head self-attention RNN model, which is superior to the GRU-RNN model. The advantage of using multi-tap multi-head self-attention RNNs is that they can utilize weighted information from all previous frames both locally and globally. Therefore, multi-tap multi-head self-attention RNN models do not require any heuristic update factors between consecutive frames as needed in methods based on recursive MVDR beamformers.

[0067] According to embodiments of this disclosure, in order to better utilize nearby TF information and stabilize the first-order estimated target speech, estimated noise, and estimated echo, a complex ratio filtering method can be used to estimate the speech, echo, and noise components. According to embodiments of this disclosure, a jointly trained deep learning-based complex ratio mask 220 can be used to estimate complex ratio masks and / or complex ratio filters, which are used to generate the estimated target speech, estimated noise, and estimated echo. In some embodiments, the estimated target speech, estimated noise, and estimated echo can be generated simultaneously using a subset of the same complex ratio mask and / or filters. In some embodiments, the estimated target speech, estimated noise, and estimated echo can be generated simultaneously using complex ratio masks and / or filters derived specifically for the estimated target speech, estimated noise, and estimated echo, respectively. In some embodiments, a multi-tap multi-head self-attention RNN can be used to derive one or more complex ratio masks and / or filters. In some implementations, a mask estimator based on a Convolutional 1D Gated Recurrent Unit (RNN) can be used to estimate the target speech, the noise, and the echo.

[0068] For each TF bin, cRF can be applied to its K×L neighboring bins, where K and L represent the number of neighboring time and frequency bins.

[0069]

[0070] in, It can indicate the use of a speech complex ratio filter (cRF). SEstimated multichannel speech. cRF can be equivalent to K×K cRMs. S Each CRM S The corresponding shifted version is applied to the noisy spectrogram (i.e., along the time and frequency axes).

[0071] It can indicate the use of a noise complex ratio filter cRF N Estimated multi-channel noise. Similarly, cRF E It can be used to estimate the actual near-end echo signal.

[0072]

[0073]

[0074] According to embodiments of this disclosure, the estimated target speech, estimated noise, and estimated echo can be multi-tap extended. According to some embodiments, the extended estimated target speech and estimated echo can be linked into a first intermediate link. The extended estimated noise and estimated echo can be linked into a second intermediate link.

[0075] For example, the extended estimated target speech and the estimated echo can be concatenated into Z. S (t, f). The extended estimated noise and estimated echo can be concatenated into Z. N (t, f).

[0076]

[0077]

[0078] According to embodiments of this disclosure, intermediate links can be flattened and / or normalized. A first level of normalization can be performed using a first intermediate link based on the extended estimated target speech and the estimated echo. A second level of normalization can be performed using a second intermediate link based on the extended estimated noise and the estimated echo.

[0079] As an example, the linked Z S (t, f) and Z N (t, f) can be flattened or layer normalized. In some implementations, layer normalization can be performed before using the multi-head self-attention RNNBF 230 to determine and / or generate frame-level multi-tap TF-level spatiotemporal echo filter weights.

[0080] Z′ S (t, f) = LayerNorm(Z) S Equation (15) (t, f)

[0081] Z′ N (t, f) = LayerNorm(Z) N Equation (16) (t, f)

[0082] Z′ of the layer-standardized connection S (t, f) and Z′ N (t, f) can be input into a multi-head self-attention RNNBF 230. The multi-head self-attention RNNBF 230 can be used for Z′-based connections with layer normalization. S (t, f) and Z′ N (t, f) is used to determine and / or generate frame-level multi-tap TF spatiotemporal echo filter weights. A multi-head self-attention RNN beamformer (RNN-BF) is used to learn higher-order cross-correlation across multiple frames and channels more effectively.

[0083] [w(t-τ1,f)...w(t,f,w e (t-τ1,f)...w e [(t, f)] =

[0084] MA-RNN-BF([Z′ S (t, f), Z′ N Equation (17) (t, f)

[0085] Based on the frame-level multi-tap TF spatiotemporal echo filter weights generated using a multi-head self-attention RNNBF 230, the noisy speech mixture 254, and the echo reference 252, the enhanced target speech 270 can be predicted and / or generated as

[0086] S (i) (t, f) = [w (t-τ1, f)...w (t, f), w e (t-τ1,f)...w e (t, f)] H *[Y(t-τ1, f)...Y(t, f), E(t-τ1, f)...E(t, f) equation (18).

[0087] Here, (i) indicates a specific target source. Multiple target sources can be estimated simultaneously using a unified model.

[0088] Figure 3 An example process 300 for speech processing, including speech separation and / or speech enhancement, according to an embodiment of this disclosure is shown.

[0089] At operation 310, audio data corresponding to one or more speakers can be received. As an example, audio data 210 can be received by speech processing system 200. The received audio data may include noisy speech mixture 254, echo reference 252, and other speaker-independent components.

[0090] At operation 315, a jointly trained complex ratio mask can be used to simultaneously generate the estimated target speech, estimated noise, and estimated echo based on the audio data. As an example, a jointly trained complex ratio mask 220 can be used to simultaneously generate the estimated target speech, estimated noise, and estimated echo. According to some implementations, the jointly trained complex ratio mask is based on a multi-head attention neural network model, and wherein the same jointly trained complex ratio mask is used to generate the estimated target speech, estimated noise, and estimated echo.

[0091] According to embodiments of this disclosure, generating estimated target speech, estimated noise, and estimated echo using jointly trained complex ratio masks may include applying complex ratio filters to multiple proximity time and frequency bins associated with the audio data. According to some embodiments, the complex ratio filter applied to each of the estimated target speech, estimated echo, and estimated noise may be a subset of jointly trained and derived complex ratio masks, such as those applied to subsets of the multiple proximity time and frequency bins associated with the audio data.

[0092] According to embodiments of this disclosure, generating estimated target speech may include applying a complex ratio filter, wherein the complex ratio filter may be based on a speech complex ratio filter, such as a corresponding shifted version applied to a plurality of nearby time and frequency bins associated with the audio data. According to embodiments of this disclosure, generating estimated noise may include applying a complex ratio filter, wherein the complex ratio filter may be based on a noise complex ratio filter, such as a corresponding shifted version applied to a plurality of nearby time and frequency bins associated with the audio data.

[0093] At operation 320, the estimated target speech, estimated noise, and estimated echo can be used to predict frame-level multi-tap TF spatiotemporal echo filter weights based on the estimated target speech, estimated noise, and estimated echo using a trained neural network model. As an example, the estimated target speech, estimated noise, and estimated echo generated using a jointly trained complex ratio mask 220 can be used to predict frame-level multi-tap TF spatiotemporal echo filter weights using a multi-head self-attention RNN 230.

[0094] At operation 325, the predicted frame-level multi-tap TF spatiotemporal echo filter weights can be used to generate and / or predict enhanced target speech corresponding to one or more speakers. As an example, the predicted frame-level multi-tap TF spatiotemporal echo filter weights, noisy speech mixing 254, and echo reference 252 can be used to generate and / or predict enhanced target speech 270.

[0095] Figure 4 An example process 400 for generating enhanced target speech according to an embodiment of the present disclosure is shown.

[0096] At operation 410, multi-tap expansion can be performed on the estimated target speech, estimated noise, and estimated echo. As an example, multi-tap expansion can be performed on the estimated target speech, estimated noise, and estimated echo generated using the jointly trained complex ratio mask 220.

[0097] At operation 415, a first intermediate link can be generated based on the estimated target speech and the estimated echo. As an example, the extended estimated target speech and the estimated echo can be concatenated into a first intermediate link.

[0098] At operation 420, a second intermediate link can be generated based on the estimated noise and the estimated echo. As an example, the extended estimated noise and the estimated echo can be linked together as a second intermediate link.

[0099] At operation 425, a first normalization can be performed, wherein the first normalization may include layer normalization of the first intermediate link. As an example, the first normalization can be performed using the first intermediate link based on the extended estimated target speech and the estimated echo.

[0100] At operation 430, a second normalization can be performed, which may include layer normalization of the second intermediate link. As an example, the second normalization can be performed using the second intermediate link based on the extended estimated noise and the estimated echo.

[0101] At operation 435, a trained neural network model can be used to generate frame-level multi-tap TF spatiotemporal echo filter weights. According to some implementations, the trained neural network model is a recurrent neural network based on multi-tap multi-head attention. As an example, a multi-head self-attention RNN 230 can be used to generate frame-level multi-tap TF spatiotemporal echo filter weights. The frame-level multi-tap TF spatiotemporal echo filter weights, audio data, and estimated echoes can then be used to predict and / or generate enhanced target speech 270. In one implementation, using the TF spatiotemporal echo weights to predict the enhanced target speech corresponding to the one or more speakers includes generating the enhanced target speech based on the frame-level multi-tap TF spatiotemporal echo filter weights, the audio data, and Hermitian operators.

[0102] Figure 5 An example application in a car 500 according to the current implementation is shown.

[0103] like Figure 5 As shown, the car 500 includes four acoustic zones 510, 515, 520 and 525 and two microphone arrays 550 and 555.

[0104] In real-world scenarios, the exact locations of different speakers within a vehicle may be unknown, leading to problems with speech separation and / or speech enhancement corresponding to one or more speakers. However, as seen in vehicle 500, vehicle 500 can be divided into four acoustic regions 510, 515, 520, and 525. Using the four acoustic regions 510, 515, 520, and 525, the global direction of arrival (DOA) for the four acoustic regions in the vehicle can be roughly estimated. The roughly estimated global DOA of the four acoustic regions 510, 515, 520, and 525 can be used to inform the speech processing system 200 to extract the enhanced target speech 270 from the noisy multi-speaker speech mixture 254.

[0105] Furthermore, the audio data 210 may also include speaker-independent features (e.g., logarithmic power spectrum (LPS) and interaural phase difference (IPD)) as well as speaker-related features (e.g., directional feature d(θ)). Two microphone arrays 550 and 555 may be located in front of the vehicle 500. The position-guided directional feature (DF) d(θ) can be used to extract target speech from a specific DOA. The DF can be used to calculate the cosine similarity between the target steering vector v(θ) and the IPD.

[0106] Figure 6 According to the illustrative implementation method Figure 1 Block diagram 600 depicts the internal and external components of a computer. It should be understood that... Figure 6This illustration provides only one possible implementation and does not imply any limitation on the environments in which different implementations can be implemented. Many modifications can be made to the depicted environment based on design and implementation requirements.

[0107] Computer 102 ( Figure 1 ) and server computer 114 ( Figure 1 ) can include Figure 6 The corresponding sets of internal components 800A, 800B and external components 900A, 900B are shown. Each of the sets of internal components includes one or more processors 820 on one or more buses 826, one or more computer-readable RAMs 822 and one or more computer-readable ROMs 824, one or more operating systems 828, and one or more computer-readable tangible storage devices 830.

[0108] Processor 820 is implemented in hardware, firmware, or a combination of hardware and software. Processor 820 is a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), microprocessor, microcontroller, digital signal processor (DSP), field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), or another type of processing unit. In some implementations, processor 820 includes one or more processors that can be programmed to perform functions. Bus 826 includes components that allow communication among internal components 800A, 800B.

[0109] One or more operating systems 828, software programs 108 ( Figure 1 ) and server computer 114 ( Figure 1 The speech processing program 116 on ) Figure 1 The data is stored on one or more of the respective computer-readable tangible storage devices 830 for execution by one or more of the respective processors 820 via one or more of the respective RAMs 822 (which typically include cache memory). Figure 6 In the illustrated embodiments, each of the computer-readable tangible storage devices 830 is a disk storage device of an internal hard disk drive. Alternatively, each of the computer-readable tangible storage devices 830 is a semiconductor storage device such as ROM 824, EPROM, flash memory, optical disk, magneto-optical disk, solid-state disk, compact disc (CD), digital versatile disc (DVD), floppy disk, cassette tape, magnetic tape, and / or other types of non-transitory computer-readable tangible storage devices capable of storing computer programs and digital information.

[0110] Each set of internal components 800A, 800B also includes an R / W drive or interface 832 for reading from and writing to one or more portable computer-readable tangible storage devices 936, such as CD-ROMs, DVDs, memory sticks, magnetic tapes, disks, optical discs, or semiconductor storage devices. Such as software programs 108 ( Figure 1 ) and speech processing program 116 ( Figure 1 The software program can be stored on one or more of the corresponding portable computer-readable tangible storage devices 936, read via the corresponding R / W drive or interface 832, and loaded into the corresponding hard disk drive 830.

[0111] Each set of internal components 800A and 800B also includes a network adapter or interface 836, such as a TCP / IP adapter card; a wireless Wi-Fi interface card; or a 3G, 4G, or 5G wireless interface card or other wired or wireless communication links. Software program 108 ( Figure 1 ) and server computer 114 ( Figure 1 The speech processing program 116 on ) Figure 1 It can be downloaded from an external computer to computer 102 via a network (such as the Internet, LAN or other, WAN) and a corresponding network adapter or interface 836. Figure 1 The network includes a network adapter or interface 836 and a server computer 114. Software program 108 and a voice processing program 116 on the server computer 114 are loaded from the network adapter or interface 836 into the corresponding hard disk drive 830. The network may include copper wire, fiber optic, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers.

[0112] Each of the sets of external components 900A and 900B may include a computer display detector 920, a keyboard 930, and a computer mouse 934. External components 900A and 900B may also include a touchscreen, a virtual keyboard, a touchpad, a pointing device, and other human-machine interface devices. Each of the sets of internal components 800A and 800B also includes a device driver 840 that interfaces with the computer display detector 920, keyboard 930, and computer mouse 934. Device driver 840, R / W driver or interface 832, and network adapter or interface 836 include hardware and software (stored in storage device 830 and / or ROM 824).

[0113] It should be understood in advance that although this disclosure includes a detailed description of cloud computing, the implementation of the teachings described herein is not limited to cloud computing environments. Rather, some implementations can be combined with any other type of computing environment now known or developed in the future.

[0114] Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (such as networks, network bandwidth, servers, processing power, memory, storage devices, applications, virtual machines, and services) that can be rapidly provided and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five features, at least three service models, and at least four deployment models.

[0115] The features are as follows:

[0116] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring manual interaction with service providers.

[0117] Extensive network access: Capabilities are available through the network and are accessed via standard mechanisms facilitated by heterogeneous thin-client or thick-client platforms (such as mobile phones, laptops, and PDAs).

[0118] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically allocated and reallocated based on demand. There is a sense of location agnosticness because consumers typically do not control or know the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0119] Rapid elasticity: Capabilities can be provided quickly and elastically (in some cases automatically) to scale outwards rapidly and to scale inwards rapidly. For consumers, the available capacity to be provided often appears unlimited and can be purchased in any quantity at any time.

[0120] Measurement services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at some level of abstraction appropriate to service types (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be detected, controlled, and reported, providing transparency for both service providers and consumers.

[0121] The service model is as follows:

[0122] Software as a Service (SaaS): This provides consumers with the ability to use a provider's applications running on cloud infrastructure. These applications can be accessed from various client devices via a thin client interface such as a web browser (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage devices, or even individual application capabilities, with the possible exception of limited user-specific application configuration settings.

[0123] Platform as a Service (PaaS): This provides consumers with the ability to deploy consumer-created or acquired applications, built using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage devices, but they have control over the deployed applications and the configuration of any possible hosting environments.

[0124] Infrastructure as a Service (IaaS): This provides consumers with the capability to supply processing, storage, networking, and other basic computing resources, whereby consumers can deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but have control over the operating system, storage, deployed applications, and possibly limited control over the selection of networking components (e.g., the main firewall).

[0125] The deployment model is as follows:

[0126] Private cloud: The cloud infrastructure operates solely for the organization. It can be managed by the organization or a third party and can exist either on-premises or off-premises.

[0127] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.

[0128] Public cloud: Cloud infrastructure that is available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0129] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursts for load balancing between clouds).

[0130] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. The core of cloud computing is its infrastructure, which includes a network of interconnected nodes.

[0131] Reference Figure 7The illustration depicts a cloud computing environment 700. As shown, the cloud computing environment 700 includes one or more cloud computing nodes 10, and local computing devices used by cloud consumers, such as personal digital assistants (PDAs) or cellular phones 54A, desktop computers 54B, laptop computers 54C, and / or automotive computer systems 54N, can communicate with the cloud computing nodes 10. The cloud computing nodes 10 can communicate with each other. The cloud computing nodes 10 can be physically or virtually grouped (not shown) into one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds or combinations thereof as described above. This allows the cloud computing environment 700 to provide Infrastructure as a Service, Platform as a Service, and / or Software as a Service without requiring cloud consumers to maintain resources on their local computing devices for Infrastructure as a Service, Platform as a Service, and / or Software as a Service. It should be understood that... Figure 7 The types of computing devices 54A to 54N shown are intended to be illustrative only, and the cloud computing node 10 and cloud computing environment 700 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0132] Reference Figure 8 This demonstrates the 700 (cloud computing environment) Figure 7 The 800 provides a set of functional abstraction layers. It should be understood beforehand that... Figure 8 The components, layers, and functions shown are intended to be illustrative only, and the implementation is not limited thereto. As depicted, the following layers and corresponding functions are provided:

[0133] The hardware and software layer 60 includes hardware components and software components. Examples of hardware components include: a mainframe 61; a RISC (Reduced Instruction Set Computer) based server 62; a server 63; a blade server 64; a storage device 65; and networking and interconnection components 66. In some implementations, software components include network application server software 67 and database software 68.

[0134] The virtualization layer 70 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 71; virtual storage device 72; virtual network including virtual private network 73; virtual application and operating system 74; and virtual client 75.

[0135] In one example, management layer 80 can provide the following functions: Resource Provisioning 81 provides dynamic acquisition of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 82 provides cost monitoring when utilizing resources in the cloud computing environment, as well as billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, and protection for data and other resources. User Access Point 83 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 84 provides cloud resource allocation and management to meet required service levels. Service Level Agreement (SLA) Planning and Implementation 85 provides pre-scheduling and acquisition of cloud resources for anticipated future needs according to the SLA.

[0136] Workload layer 90 provides examples of functionalities that can be leveraged within a cloud computing environment. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 91; software development and lifecycle management 92; virtual classroom delivery 93; data analytics and processing 94; transaction processing 95; and speech processing 96. Speech processing 96 can use a fully neural network approach to separate the speech of the target speaker.

[0137] Some implementations may relate to systems, methods, and / or computer-readable media at any possible level of integration technical detail. A computer-readable medium may include a computer-readable non-transitory storage medium (or media) having computer-readable program instructions thereon for causing a processor to perform operations.

[0138] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. For example, computer-readable storage media can be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or raised structures in recesses on which instructions are recorded, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as being a transient signal such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0139] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0140] Computer-readable program code / instructions for performing operations can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, configuration data of an integrated circuit system, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of execution entirely on a remote computer or server, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the use of the Internet provided by an Internet service provider). In some implementations, electronic circuit systems, including, for example, programmable logic circuit systems, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), can perform various aspects or operations by using state information of computer-readable program instructions to personalize the electronic circuit system.

[0141] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0142] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer-implemented process, such that the instructions, which execute on the computer, other programmable apparatus or other device, perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer-readable media according to various embodiments. In this regard, each block in a flowchart or block diagram may represent a portion of a module, segment, or instruction, including one or more executable instructions for implementing a specific logical function. The method, computer system, and computer-readable medium may include additional blocks, fewer blocks, different blocks, or blocks arranged differently compared to those depicted in the drawings. In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the drawings. For example, two blocks shown consecutively may actually be executed simultaneously or substantially simultaneously, or the blocks may sometimes be executed in reverse order, depending on the functions involved. It will also be noted that each block in the block diagrams and / or flowcharts, as well as combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0144] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation method. Therefore, this document describes the operation and behavior of the systems and / or methods without referring to any specific software code—it should be understood that software and hardware can be designed to implement the systems and / or methods based on the description herein.

[0145] Unless explicitly stated otherwise, no element, action, or instruction used herein should be construed as critical or necessary. Furthermore, as used herein, the articles “a” and “an” are intended to include one or more items and are interchangeable with “one or more.” Additionally, as used herein, the term “group” is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with “one or more.” The term “an” or similar language is used where only one item is intended. Moreover, as used herein, the terms “has,” “have,” “having,” etc., are intended to be open-ended terms. Furthermore, unless explicitly stated otherwise, the phrase “based on” is intended to mean “at least partially based on.”

[0146] Descriptions of various aspects and implementations have been presented for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed implementations. Even combinations of features recited in the claims and / or disclosed in the specification are not intended to limit the disclosure of possible implementations. In fact, many of these features can be combined in ways not specifically recited in the claims and / or not disclosed in the specification. Although each dependent claim listed below may directly refer to only one claim, the disclosure of possible implementations includes every dependent claim combined with each other claim in the claim set. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the described implementations. The terminology used herein has been chosen to best explain the principles of the implementations, their practical application, or technical improvements to techniques found in the market, or to enable others skilled in the art to understand the implementations disclosed herein.

Claims

1. A method executed by a computing device for generating enhanced target speech from audio data, characterized in that, The method includes: Receive audio data corresponding to one or more speakers; Based on the audio data, a jointly trained complex ratio mask is used to simultaneously generate estimated target speech, estimated noise, and estimated echo; generating the estimated target speech includes applying a complex ratio filter, wherein the complex ratio filter is a speech complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data; generating the estimated noise includes applying a complex ratio filter, wherein the complex ratio filter is a noise complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data; generating the estimated echo includes applying a complex ratio filter, wherein the complex ratio filter is an echo complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data; Based on the estimated target speech, the estimated noise, and the estimated echo, a trained neural network model is used to predict frame-level multi-tap time-frequency (TF) spatiotemporal echo filter weights; and Enhanced target speech is predicted based on the frame-level multi-tap TF spatiotemporal echo filter weights.

2. The method according to claim 1, characterized in that, The jointly trained complex ratio mask is based on a multi-head attention neural network model, and the same jointly trained complex ratio mask is used to generate the estimated target speech, the estimated noise, and the estimated echo.

3. The method according to claim 1, characterized in that, Generating the estimated target speech, the estimated noise, and the estimated echo using the jointly trained complex ratio mask includes: A complex ratio filter is applied to multiple nearby time and frequency bins associated with the audio data.

4. The method according to any one of claims 1 to 3, characterized in that, Predicting the frame-level multi-tap TF spatiotemporal echo filter weights based on the estimated target speech, the estimated noise, and the estimated echo includes: The estimated target speech, the estimated noise, and the estimated echo are expanded; A first intermediate link is generated based on the connection between the estimated target speech and the estimated echo. A second intermediate link is generated based on the connection between the estimated noise and the estimated echo; Perform a first standardization, wherein the first standardization includes the layer standardization of the first intermediate link; Perform a second standardization, wherein the second standardization includes the layer standardization of the second intermediate link; and The frame-level multi-tap TF spatiotemporal echo filter weights are generated using the trained neural network model based on the first and second standardizations.

5. The method according to claim 4, characterized in that, The trained neural network model is a recurrent neural network based on multi-tap multi-head attention.

6. The method according to any one of claims 1 to 3, characterized in that, The enhanced target speech is predicted based on the frame-level multi-tap TF spatiotemporal echo filter weights, the audio data, and the estimated echo.

7. The method according to any one of claims 1 to 3, characterized in that, Using the TF spatiotemporal echo weights to predict the enhanced target speech corresponding to the one or more speakers includes: The enhanced target speech is generated based on the frame-level multi-tap TF spatiotemporal echo filter weights, the audio data, and the Hermitian operator.

8. A computer device, characterized in that, The computer device includes: One or more computer-readable non-transitory storage media configured to store computer program code; and One or more computer processors configured to access the computer program code and execute the method according to any one of claims 1 to 7 as instructed by the computer program code.

9. An apparatus for generating enhanced target speech from audio data, characterized in that, The device includes: A first receiving unit is configured to receive audio data corresponding to one or more speakers; A first generation unit is configured to simultaneously generate estimated target speech, estimated noise, and estimated echo based on the audio data using a jointly trained complex ratio mask. Generating the estimated target speech includes applying a complex ratio filter, wherein the complex ratio filter is a speech complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data. Generating the estimated noise includes applying a complex ratio filter, wherein the complex ratio filter is a noise complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data. Generating the estimated echo includes applying a complex ratio filter, wherein the complex ratio filter is an echo complex ratio filter applied to corresponding shifted versions of multiple nearby time and frequency bins associated with the audio data. A first prediction unit, configured to predict frame-level multi-tap time-frequency (TF) spatiotemporal echo filter weights based on the estimated target speech, the estimated noise, and the estimated echo using a trained neural network model; and The second prediction unit is configured to predict enhanced target speech based on the frame-level multi-tap TF spatiotemporal echo filter weights.

10. The apparatus according to claim 9, characterized in that, The jointly trained complex ratio mask is based on a multi-head attention neural network model, and the same jointly trained complex ratio mask is used to generate the estimated target speech, the estimated noise, and the estimated echo.

11. The apparatus according to claim 9, characterized in that, The device further includes: A first application unit is configured to apply a complex ratio filter to a plurality of nearby time and frequency bins associated with the audio data.

12. The apparatus according to any one of claims 9 to 11, characterized in that, The first prediction unit includes: A first expansion subunit is configured to expand the estimated target speech, the estimated noise, and the estimated echo. A second generation subunit is configured to generate a first intermediate link based on the connection between the estimated target speech and the estimated echo. A third generation subunit is configured to generate a second intermediate link based on the estimated noise and the estimated echo link. A first execution subunit is configured to execute a first standardization, wherein the first standardization includes the layer standardization of the first intermediate link; A second execution subunit, configured to execute a second normalization, wherein the second normalization includes the layer normalization of the second intermediate link; and A fourth generation subunit is configured to generate the frame-level multi-tap TF spatiotemporal echo filter weights using the trained neural network model based on the first normalization and the second normalization.

13. The apparatus according to claim 12, characterized in that, The trained neural network model is a recurrent neural network based on multi-tap multi-head attention.

14. The apparatus according to any one of claims 9 to 11, characterized in that, The enhanced target speech is predicted based on the frame-level multi-tap TF spatiotemporal echo filter weights, the audio data, and the estimated echo.

15. The apparatus according to any one of claims 9 to 11, characterized in that, Using the TF spatiotemporal echo weights to predict the enhanced target speech corresponding to the one or more speakers includes: The enhanced target speech is generated based on the frame-level multi-tap TF spatiotemporal echo filter weights, the audio data, and the Hermitian operator.

16. A non-transitory computer-readable storage medium storing instructions, characterized in that, The instructions include: one or more instructions that, when executed by one or more processors of a device for generating enhanced target speech from audio data, cause the one or more processors to perform the method according to any one of claims 1 to 7.

Images