Real-time low-complexity stereo speech enhancement preserving spatial cues

Abstract

Real-time low-complexity stereo speech enhancement that preserves spatial cues may be performed. A stereo speech enhancement system receives stereo input signals (e.g., a left input signal and a right input signal). The stereophonic speech enhancement system estimates spatial cues of a target speaker and downmixes the stereophonic input signal into a monoaural signal. A low complexity model may then process the monaural signal to generate an enhanced monaural signal. The stereo speech enhancement system upmixes the enhanced monaural signal based on the estimated spatial cues of the target speaker to generate an enhanced stereo output signal.

Description

Background Art

[0001] In the past few years, deep learning-based audio enhancement methods (e.g., for recorded human speech) have significantly outperformed traditional methods (e.g., due to techniques such as spectral subtraction and spectral estimation). Audio enhancement methods can be used in a variety of applications. For example, a teleconferencing system may be used in a noisy and reverberant environment, so speech enhancement techniques may be needed to ensure clear communication.

[0002] By using multiple microphones in a conference call, spatial aspects of sound, such as the location of speakers or other sound sources, can be captured. Pursuing high speech enhancement performance while preserving spatial cues can be useful, as spatial cue information can help listeners determine who is speaking during a conference call. However, the huge maintenance and training costs of implementing stereo-specific speech enhancement models (e.g., deep neural network (DNN) models) can be prohibitive. In addition, applying a mono noise suppressor to each microphone signal independently may not preserve the spatial location of the speech and may also lead to auditory artifacts, especially when the spatial properties of the target speech and interfering noise are different. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] Figure 1 A logical block diagram is shown for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments.

[0004] Figure 2 An example provider network is shown that can implement an audio delivery service that implements real-time low-complexity stereo speech enhancement that preserves spatial cues in accordance with some embodiments.

[0005] Figures 3A-3C A logical block diagram illustrating different interactions of an audio sensor with a provider network service according to some embodiments.

[0006] Figure 4 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0007] Figure 5 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0008] Figure 6 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0009] Figure 7A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0010] Figure 8 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0011] Fig. 9 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments.

[0012] Fig.10 A logical block diagram illustrating an example algorithm for real-time low-complexity speech enhancement preserving spatial cues for processing multiple input signals in accordance with some embodiments.

[0013] Fig.11 A high-level flow chart is presented for implementing various methods and techniques for real-time low-complexity speech enhancement preserving spatial cues in accordance with some embodiments.

[0014] Fig.12 Example systems implementing the various methods, techniques, and systems described herein are presented in accordance with some embodiments.

[0015] Although embodiments are described herein by way of example for several embodiments and illustrative figures, those skilled in the art will recognize that embodiments are not limited to the described embodiments or drawings. It should be understood that the drawings and detailed descriptions thereof are not intended to limit the embodiments to the specific forms disclosed, but on the contrary, are intended to cover all modifications, equivalents and alternatives falling within the spirit and scope described by the appended claims. The titles used herein are only for organizational purposes and are not intended to be used to limit the scope of this specification or claims. As used throughout this application, the word "may" is used in a permissive sense (i.e., meaning possible) rather than a mandatory sense (i.e., meaning must). Similarly, the word "include / including / includes" means including but not limited to.

[0016] It will also be understood that although the terms first, second, etc. may be used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, without departing from the scope of the present invention, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact. Both the first contact and the second contact are contacts, but they are not the same contact. DETAILED DESCRIPTION

[0017] Various techniques for real-time low-complexity stereo speech enhancement that preserves spatial cues are described herein. With the ubiquity of real-time audio communication systems, there has been great interest in speech enhancement algorithms that operate in real-time with low complexity. Users of these communication systems (e.g., target speakers) may find themselves in competing background sounds (e.g., noise from various sources). In various embodiments, any number of microphones may be used during a conference call to generate multiple input signals. For example, two microphones may be used to provide stereo input.

[0018] Implementing stereo-specific speech enhancement models (e.g., highly complex deep neural networks (DNNs) or other types of models) may be daunting. Instead, various embodiments may use low-complexity models and / or algorithms (such as the example "PercepNet" model discussed herein) to deliver high-quality speech enhancement in real time. As discussed in more detail below, to utilize such models, stereo inputs are downmixed to monaural signals (while also estimating spatial cues, e.g., steering vectors and / or binaural phase / level differences between different inputs), processed by the model, and then upmixed to produce an enhanced stereo signal (taking into account spatial cues).

[0019] In various embodiments, audio enhancement techniques based on machine learning (ML) models can utilize real-time target speaker audio enhancement techniques for monaural signals (e.g., after estimating spatial cues and downmixing from a stereo signal), which provide a perceptually motivated approach to real-time, low-complexity target speaker enhancement, such as PercepNet. Real-time target speaker audio enhancement techniques can perform conditioning and / or other audio enhancement techniques on the target speaker's voice. This enables the machine learning model for audio enhancement to clearly identify and enhance the target speaker's speech while suppressing all other interference, even in the presence of multiple speakers or other speech-like sounds. As discussed below, the enhanced monaural signal is then upmixed based on the estimated spatial cues of the target speaker to generate an enhanced stereo output signal. In some embodiments, the above process can be performed for any number of different target speakers.

[0020] In some embodiments, it may be assumed that there are two microphone input signals (note that in various embodiments, any number of microphone inputs may be used). m,t (t is the time index) can be modeled in the time domain as follows: m,t =s t *h m +n t , where s t is the speech source signal, h mis the impulse response between the speech source position and the mth microphone, and n t is the background noise signal. In an embodiment, downmixing, model-based (e.g., PercepNet or other models) monaural speech enhancement, and upmixing are all performed in the time-frequency domain. In an embodiment, x m,t The time-frequency representation of can be written as follows: l,k =s l,k a k +n l,k , where l is the frame index, k is the frequency index, and x l,k =[x T 1,l,k x T 2,l,k ] T , x m,l,k is the time domain signal x m,t Time-frequency representation. s l,k and n l,k The definition of is similar. k =[a 1,k a 2,k ] T is the steering vector, and a m,k Yes m The time-frequency representation of .

[0021] Typically, the goal of speech enhancement is to obtain a l,k To estimate sl,k. In our application, the steering vector a k is also important because it captures spatial information, so our goal is to l,k To determine l,k and a k In addition, since it is difficult and expensive to retrain DNN models for different numbers and locations of sound sources and microphones, we focus on using a monophonic DNN model pre-trained for monaural speech enhancement.

[0022] Figure 1 A logical block diagram is shown for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments.

[0023] In an embodiment, the example stereo speech enhancement system 100 may include three steps / stages. In the first step, the spatial clues of one or more target sources (e.g., speakers) represented in the stereo microphone input signal are estimated and stored as metadata (e.g., binaural phase / level differences between different inputs of one or more target speakers), and the stereo microphone input signal is converted into a monaural signal (downmix 102). The second step includes using an ML model 104 and / or algorithm to enhance the monaural signal (e.g., PercepNet or other ML models). In an embodiment, the algorithm may operate on a 10 millisecond frame with a 30 millisecond look-ahead, but any other amount of time may be used in various embodiments. As mentioned above, in order to utilize a model of a single microphone as a pre-trained model (e.g., a DNN model or other pre-trained ML model), the input signal of the model is generated by downmixing the stereo microphone input signal to a monaural input signal (retaining spatial information / clues). By applying the retained spatial information, a stereo signal can be generated from the enhanced monaural output signal of the model (e.g., applying binaural phase and level differences to the monaural signal to generate two different output signals). Therefore, the third step consists in converting the output signal of the model into a stereo signal (upmix 106).

[0024] In various embodiments, the stereo speech enhancement system 100 can be implemented as part of various network-based systems or services or stand-alone systems that receive audio data (e.g., stereo speech audio, which may include target speaker audio and various background audios of a first input signal from a left microphone and a second input signal from a right microphone) and provide enhanced audio data as output (e.g., enhanced stereo speech audio, which may include enhanced target speaker audio and various background audios of the first output signal and the second output signal). For example, the stereo speech enhancement system 100 can be implemented on a "server side", such as a Figure 3A , where the audio sensor that captures the audio data may be separate from the service or system that implements the stereo voice enhancement system 100. In such embodiments, the audio data may be sent from the audio sensor / microphone (e.g., via a network connection) to the system or service for stereo voice enhancement. In other embodiments, the stereo voice enhancement system 100 may be implemented as part of the same device as the audio sensor (e.g., as part of an audio processing component or system implemented within a device that includes the audio sensor, such as a mobile phone or device, including various types of "smart" phones, "smart" speakers, "smart" televisions, content delivery or audio / video streaming devices that capture audio data, etc.).

[0025] In an embodiment, a stereo speech enhancement system may receive a stereo input signal through an interface of the stereo speech enhancement system, the stereo input signal including a first input signal and a second input signal (e.g., a left input signal and a right input signal corresponding to a left microphone and a right microphone, respectively, or a right input signal and a left input signal corresponding to a right microphone and a left microphone, respectively). The first input signal may include speech data corresponding to a target speaker (e.g., generated by an audio sensor / microphone that senses the voice of the target speaker), and the second input signal may include other speech data corresponding to the target speaker (e.g., generated by another audio sensor / microphone located at a different position that senses the voice of the target speaker).

[0026] The stereo speech enhancement system can downmix the stereo input signal to generate a monaural signal. Before downmixing the signal, the spatial cues of the target speaker are first estimated and retained (e.g., stored as metadata) using any suitable technique (e.g., storing the spatial cues to a temporary storage location in a memory or other type of data storage device / database). The stereo speech enhancement system then applies the ML model to the monaural signal to generate an enhanced monaural signal. In an embodiment, noise in the monaural signal is reduced or removed from the monaural signal by the ML model to generate an enhanced monaural signal.

[0027] The stereo speech enhancement system can then upmix the enhanced monaural signal based at least on the spatial cues of the target speaker to generate an enhanced stereo output signal. The enhanced stereo output signal includes a first output signal and a second output signal. The first output signal includes enhanced speech data corresponding to the target speaker, and the second output signal includes other enhanced speech data corresponding to the target speaker. The stereo speech enhancement system can then send the enhanced stereo output signal to the destination through the interface of the stereo speech enhancement system. Note that in some embodiments, only partially processed stereo signals can be provided depending on the capacity or needs of the target destination. For example, if the target destination only supports monaural playback (e.g., has only one speaker), the enhanced monaural signal can be sent to the destination (in some cases, the estimated spatial cues can also be sent). In some embodiments, the monaural signal and / or the estimated spatial cues can be sent to the destination (e.g., before any processing can be modeled).

[0028] As described herein, the stereo speech enhancement system can use various techniques to perform the upmixing and / or downmixing stages. For example, downmixing the stereo input signal can include applying beamforming (e.g., delay-sum beamforming) to the stereo input signal. In some embodiments, in order to estimate spatial cues (e.g., spatial information), the stereo speech enhancement system can estimate steering information (e.g., an estimated steering vector) for the target speaker. The system can apply delay-sum beamforming to the stereo input signal using the estimated steering information (e.g., an estimated steering vector) to generate a monaural signal.

[0029] In some embodiments, upmixing the enhanced monaural signal may include applying the steering information of the target speaker to the enhanced monaural signal (e.g., multiplying the estimated steering vector of the target speaker with the enhanced monaural signal). In an embodiment, the stereo speech enhancement system may estimate the steering vector of the target speaker using principal component analysis of the estimated spatial covariance matrix. In some embodiments, the stereo input signal may be captured with corresponding video data, and the video data may be provided to the same destination as the enhanced stereo output signal.

[0030] This specification includes a general description of a provider network that implements a variety of different services ( Figure 2 ), wherein the plurality of different services includes an audio transmission service that can implement real-time low-complexity stereo speech enhancement for the transmitted audio. Then, Figures 3A-3C Various examples (including different components / modules) or arrangements of components / modules that can be used as part of implementing the service are discussed. A variety of different methods and techniques (e.g., different systems for downmixing and / or upmixing) for implementing real-time low-complexity stereo speech enhancement are then discussed, some of which are shown in the accompanying flow charts. Finally, a description of an example computing system on which various components, modules, systems, devices, and / or nodes can be implemented is provided. Various examples are provided throughout the specification.

[0031] Downmix

[0032] Using conventional techniques, downmixing can be performed by averaging the stereo signals. However, averaging may cause the speech signals to cancel out and may result in performance degradation. In an embodiment, rather than averaging the stereo signals, the stereo input signal is converted into a monaural signal using beamforming techniques. To avoid signal cancellation associated with adaptive beamforming techniques, downmixing can be performed using delayed sum beamforming (DSBF). Assume b lk is the estimated steering vector. Downmixed monaural signal d lk Obtained as follows: d lk =b Hl,k x l,k , where H is the Hermitian transpose operator of the matrix / vector. To perform DSBF, estimate the speech source b lk The steering vector b is estimated by using principal component analysis (PCA) with estimated spatial covariance (SCM) lk .

[0033] Spatial covariance matrix estimation

[0034] To perform real-time speech enhancement, SCM R is updated in an online manner as follows l,k : R lk =γ lk R l-1,k+(1- γ lk )x lk x lk H ,and Where α is the forgetting factor. is a time-frequency mask that controls the updating of the SCM according to the presence of a speech source at each time-frequency bin. When a speech source is present, is close to 1. On the other hand, when there is no speech source, is close to 0. The time-frequency mask can be estimated in two ways Time average of microphone input signal (TAM) or time-frequency mask based speech enhancement output (TFMSE).

[0035] Time average of microphone input signal (TAM)

[0036] When there is no information about the presence of a speech source at each time-frequency bin, the best approach is to assume that a speech source is always present. In this case, the time-frequency mask should be determined as follows: This corresponds to the time averaging of the microphone input signal used for SCM estimation. Time averaging works well when the noise signals between channels are uncorrelated and the noise level is relatively lower than the speech signal level. However, when the noise level is relatively greater than the speech signal level or the noise signals between channels are correlated, the accuracy of SCM estimation decreases.

[0037] Time-Frequency Masking Based Speech Enhancement Output (TFMSE)

[0038] In order to avoid performance degradation of the SCM estimation, it may be important to extract the time-frequency bins where the speech source is dominant. Applying the monaural speech enhancement results to estimate the time-frequency mask can be performed as follows: where c lk is the enhanced speech signal. lkIt can be generated by multiplying the steering vector with the monaural speech enhancement result. is the output monaural signal after speech enhancement. Comparing with the microphone input signal, time upmixing is performed and we can obtain a stereo noise-free signal as follows: in is the estimated steering vector of the speech signal. The block diagram of this method is shown in Figure 2 In order to obtain the monaural speech enhancement result, a feedback loop between the downmix and the upmix is ​​utilized. Due to the feedback loop, the number of DNN inferences is one pass per frame. In various embodiments, there are other variants of TFMSE. For example, instead of using a feedback loop, a stereo enhanced speech signal can be obtained by executing a model (e.g., PercepNet) separately for each channel. In such embodiments, the model can be executed twice per frame.

[0039] Upmix

[0040] Upmixing converts a monaural signal into a stereo signal. Upmixing can be performed in two ways: steering vector multiplication or a common gain based approach.

[0041] Guided vector multiplication

[0042] The steering vector multiplication converts the monaural signal into a stereo signal by multiplying the estimated steering vector with the monaural signal. In an embodiment, the steering vector may be estimated for the upmix stage in the same or similar manner as the steering vector is estimated for the downmix stage. Let e lk is the estimated steering vector of the upmix. The upmixed signal can be obtained as follows: lk =d lk e lk , where f is the output stereo signal. Similar to downmixing, the steering vector is estimated by PCA of the estimated SCM. We can use TAM and TFMSE for SCM estimation.

[0043] Common Gain Based Approach

[0044] Sharing time-frequency gains between channels is considered to be an effective way to preserve spatial cues

[25] . Our common gain-based approach shares 32 frequency band gains estimated in PercepNet between channels. Common gains are usually applied to noisy microphone input signals. We refer to this approach as common1. However, common1 lacks the ability to exploit beamforming. Another approach is to apply a common frequency band gain to the enhanced speech signal, i.e., c in Equation 8. lk。We call this method common2. In common2, upmixing can be interpreted as a hybrid combination of steering vector multiplication and common band gain multiplication.

[0045] In various embodiments, the hybrid framework combines model-based monaural speech enhancement (e.g., a DNN model) and array signal processing as a real-time stereo speech enhancement technique that retains spatial cues. By using beamforming-based downmixing and steering vector multiplication-based upmixing, both spatial cue retention performance and speech enhancement performance can be improved. In an embodiment, steering vector estimation based on time-frequency masking and / or a common gain method for beamformer output may be very effective. In some embodiments, using the techniques discussed herein, DNN inference once per frame may be sufficient. Using the embodiments discussed herein, there may be significant advantages over traditional techniques in terms of computational cost, especially when the number of microphones is large. In addition, the embodiments can be easily extended to more general multi-channel speech enhancement algorithms without the need for DNN retraining.

[0046] Figure 2 An example provider network is shown that can implement an audio delivery service that implements real-time low-complexity stereo speech enhancement that preserves spatial cues in accordance with some embodiments.

[0047] In one embodiment, provider network 200 may be a private or closed system, or may be set up by an entity such as a company or public sector organization to provide one or more services (such as various types of cloud-based storage devices) that are accessible via the Internet and / or other networks to clients 250. In one embodiment, provider network 200 may be implemented in a single location, or may include multiple data centers that host various resource pools required to implement and distribute the infrastructure and services provided by provider network 200, such as a collection of physical and / or virtualized computer servers, storage devices, networking devices, etc. (e.g., as described below with respect to Fig.12 1200). In some embodiments, the provider network 200 may implement various computing resources or services, such as an audio delivery service 210, a storage service 230, and / or any other type of network-based service 240 (which may include virtual computing services and various other types of storage, database or data processing, analytics, communications, event processing, visualization, data cataloging, data ingestion (e.g., ETL), and security services).

[0048] In various embodiments, Figure 2The components shown in can be implemented directly in computer hardware, as instructions that can be directly or indirectly executed by computer hardware (e.g., a microprocessor or a computer system), or using a combination of these techniques. For example, in one embodiment, Figure 2 The components of can be implemented by a system including multiple computing nodes (or simply nodes), each of which can be similar to Fig.12 2 and described below. In various embodiments, the functionality of a given system or service component (e.g., a component of the audio delivery service 210) may be implemented by a specific node, or may be distributed across several nodes. In some embodiments, a given node may implement the functionality of more than one service system component (e.g., more than one data storage component).

[0049] Audio transport service 210 may implement interface 211 to allow a client (e.g., client 250 or a client implemented internally in provider network 200, such as a client application hosted on another provider network service such as an event-driven code execution service or a virtual computing service) to send audio data (e.g., a stereo voice input signal) for enhancement, storage, and / or transmission. In at least some embodiments, audio transport service 210 may also support transmission of video data as well as corresponding audio data, and thus may be an audio / video transport service that, in some embodiments, may perform the above-described enhancements on audio data captured along with video data. Figure 1 and the following about Figure 3A-10 For example, the audio transport service 210 can implement an interface 211 (e.g., a graphical user interface, a programming interface implementing an application program interface (API), and / or a command line interface) that can be implemented to allow a client application to submit an audio stream captured by the sensor 252 for storage as enhanced audio data 232, which is stored in the storage service 230 or other storage location or resource within the provider network 200 or external to the provider network 200 (e.g., on a premise data storage device in a private network). The interface 211 can allow a client to use the above-mentioned Figure 1 and below about Figure 3A-10 The discussed techniques enable audio enhancement (eg, as part of an audio transmission, such as a voice transmission, such as Voice over IP (VoIP)).

[0050] The audio transmission service 210 can implement a control plane 212 to perform various control operations to implement the features of the audio transmission service 210. For example, the control plane 212 can monitor the health and performance of requests at the audio transmission 213 and audio enhancement 215 of different components (e.g., the health or performance of various nodes that implement these features of the audio transmission service 210). If a node fails, a request fails, or other interruption occurs, the control plane 212 can restart the job to complete the request (e.g., instead of sending a failure response to the client). In some embodiments, the control plane 212 can arbitrate, balance, select, or dispatch requests from different nodes in various embodiments. For example, the control plane 212 can receive a request interface 211 (which can be a programming interface) and identify an available node to begin processing the request.

[0051] The audio transport service 210 may implement an audio transport 213 that may facilitate audio communications (e.g., for audio-only, video, or other voice communications), voice commands or voice recordings, or various other audio transports, as described below with respect to Figure 3A and 3B The audio delivery service 210 may implement the audio enhancement 215 to provide an audio enhancement system (e.g., Figure 1 The stereo speech enhancement system 100 or similar system in FIG. 100 may be implemented as follows with respect to Figure 4-10 The audio enhancement systems discussed herein and for example Fig.11 The technologies discussed.

[0052] The data storage service 230 may implement different types of data storage devices for storing, accessing, and managing data on behalf of the client 250 as a network-based service that enables the client 250 to operate a data storage system in a cloud or network computing environment. In some embodiments, the data storage service 230 may also include a variety of relational or non-relational databases. In some embodiments, the data storage service 230 may include an object or file data storage device for placing, updating, and obtaining data objects or files. The data storage service 230 may be accessed through a programming interface (e.g., an API) or a graphical user interface. In some embodiments, the enhanced audio 232 may be placed and / or retrieved from the data storage service 230 through an interface of the data storage service 230, as described below with respect to Figure 3C discussed.

[0053] In general, client 250 can encompass any type of client that can submit a network-based request to provider network 200 via network 260, the request including a request for audio transmission service 210 (e.g., a request to enhance, transmit and / or store audio data). For example, a given client 250 may include a suitable version of a web browser, or may include a plug-in module or other type of code module that can be executed as an extension of an execution environment provided by a web browser or within the execution environment. Alternatively, client 250 can encompass applications (or their user interfaces), media applications, office applications, or any other applications that can utilize audio transmission service 210 (or other services of provider network 200) to implement various applications. In some embodiments, such applications may include sufficient protocol support (e.g., for a suitable version of hypertext transfer protocol (HTTP)) for generating and processing network-based service requests without having to implement full browser support for all types of network-based data. That is, client 250 can be an application that can interact directly with provider network 200. In some embodiments, client 250 may generate a network-based service request according to a representational state transfer (REST) ​​style network-based service architecture, a document or message based network-based service architecture, or another suitable network-based service architecture.

[0054] In some embodiments, client 250 can provide access to provider network 200 to other applications in a transparent manner. In one embodiment, client 250 can transmit network-based service requests (e.g., requests for interaction with services such as audio transmission service 210) through network 260. In various embodiments, network 260 can cover any suitable combination of networking hardware and protocols necessary for network-based communication between client 250 and provider network 200. For example, network 260 can generally cover various telecommunication networks and service providers that implement the Internet together. In one embodiment, network 260 can also include private networks (such as local area networks (LANs) or wide area networks (WANs)) and public or private wireless networks. For example, given client 250 and provider network 200 can be provisioned in an enterprise with its own internal network, respectively. In such embodiments, network 260 can include hardware (e.g., modems, routers, switches, load balancers, proxy servers, etc.) and software (e.g., protocol stacks, accounting software, firewall / security software, etc.) necessary for establishing networking links between given client 250 and the Internet and between the Internet and provider network 200. Note that in some embodiments, client 250 may communicate with provider network 200 using a private network rather than the public Internet.

[0055] In various embodiments, a sensor 252 (e.g., a microphone) can collect, capture, and / or report a variety of audio data (or audio data as part of other captured data (e.g., video data). The sensor 252 can be implemented as part of a device such as various mobile or other communication and / or playback devices, such as a microphone embedded in a "smart speaker" or other voice command-enabled device. In some embodiments, some or all of the audio enhancement techniques can be implemented as part of a device including the sensor 252 before the enhanced audio is transmitted to the audio transmission service 210, as described below with respect to Figure 3B and 3C For example, the downmix stage may be performed by a local / client device at a client network, and the monaural signal (along with metadata with preserved spatial cues) may then be transmitted to a provider network for the remaining processing (e.g., model processing and upmix processing) to be performed at the provider network to generate an enhanced stereo output.

[0056] As discussed above, in some embodiments, different interactions between sensors capturing audio data and services of provider network 200 may invoke audio enhancement. Figures 3A-3C A logical block diagram illustrating different interactions of an audio sensor with a provider network service according to some embodiments.

[0057] exist Figure 3A In the embodiment, the audio sensor 310 can capture audio data from various environments, including as described above with respect to Figure 1 Speech audio in a noisy environment as discussed. In some embodiments, a device having an audio sensor 310 can send captured audio data 312 directly to an audio transmission service 210 via an interface for the audio transmission service 210 (e.g., interface 211), such as sending the captured audio data 312 to the audio transmission service 210 via a wired or wireless network connection. In some embodiments, a device having an audio sensor 310 can provide the captured audio data to another device, which sends the captured audio data 312 to an audio transmission service (not shown). In some embodiments, the captured audio data can be transmitted as an audio file or object or as an audio stream. For example, for real-time communications such as VoIP calls, the captured audio data 312 can be an audio data stream.

[0058] In various embodiments, the audio transmission service 210 may process the captured audio data 312 through audio enhancement 215 (e.g., stereo speech enhancement). For example, the following description may be implemented. Figure 1 and 4-10 to provide enhanced audio data 314, including as described above with respect to Figure 1 and below about Figure 4-10 In some embodiments, the audio transport 213 may receive the enhanced audio data 314, identify a destination for the enhanced audio, such as an audio playback device 320, and send the enhanced audio data 316 to the audio playback device 320. The audio playback device 320 may play the enhanced audio data 316 to one or more listeners, taking into account the improvements to the audio quality resulting from the audio enhancement, including the reduction of noisy frequency bands using rate mask post-filtering (e.g., which may benefit from improvements to the captured audio data in the form of more clear and perceptible speech).

[0059] In some embodiments, the audio enhancement system may also be implemented separately from the audio delivery service 210. For example, Figure 3B As shown in FIG. 1 , the device having the audio sensor 330 may also implement audio enhancement 332, which may be used, for example, for the following description of Figure 4-10 As discussed above with respect to stereo speech enhancement systems, the system may be implemented to provide enhanced audio data 334 (eg, enhanced stereo output), as described above with respect to Figure 1 and below about Figure 4-10 Audio enhancement 332 may be implemented as part of other pre-transmission processing performed by a device having audio sensor 330 , such as various encryption, compression, or other operations performed on captured audio data prior to transmission to audio transmission service 210 .

[0060] In some embodiments, the device with the audio sensor 330 may then send the captured / enhanced audio data 334 to the audio transport service 210 for transmission (e.g., via the interface 211). In some embodiments, the audio transport 213 may receive the enhanced audio data 334, identify the destination of the enhanced audio, such as the audio playback device 340, and send the enhanced audio data 336 to the audio playback device 340.

[0061] As mentioned above, in various embodiments, any portion of the audio enhancement process may be performed at the local client network (e.g., by a device having an audio sensor 330), and the remainder of the audio enhancement process may be performed by the provider network. For example, the downmix stage may be performed by a local / client device at the client network, and the monaural signal (and estimated spatial cues) may then be transmitted to the provider network for the remaining processing (e.g., model processing and upmix processing) to be performed at the provider network to generate an enhanced stereo output. As another example, model processing of the monaural signal may be performed by a local / client device at the client network, and the enhanced monaural signal (and estimated spatial cues) may then be transmitted to the provider network for the remaining processing (e.g., upmix processing) to be performed at the provider network to generate an enhanced stereo output.

[0062] In some embodiments, the audio may be stored for later retrieval and / or processing. Figure 3C As shown in FIG. 1 , the device having the audio sensor 350 may also implement audio enhancement 352, which may be used, for example, for the following description of Figure 4-10 The systems discussed above for stereo speech enhancement provide enhanced audio data 354, including Figure 1 The audio enhancement 352 may be implemented as part of other pre-transmission processing performed by the device with the audio sensor 350, such as various encryption, compression, or other operations performed on the captured audio data prior to storage in the storage service 230. In some embodiments, the device with the audio sensor 350 may then store the captured / enhanced audio data 354 to the storage service 230, which may store the enhanced audio data 360 until it is retrieved for future processing and / or playback.

[0063] Figure 4 A logical block diagram of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments is shown. As shown, in the downmix stage 402, delayed-sum beamforming 404 is applied to the left and right input signals (stereo signals) to convert the stereo input signals into monaural signals. The monaural signals are then processed by the ML model 406, where the speech signal is enhanced. In an embodiment, speech-only time-frequency bins are used to optimize the speech enhancement filter. The time-frequency bins can be extracted from the output of the ML model; therefore, the feedback structure may be important.

[0064] The upmix stage 408 performs additional processing on the output of the ML model and the input signal. As shown, a steering vector multiplication 410 is performed on the result of the ML model. Buffer 412 (30 millisecond delay) receives the left and right input signals and provides the input signals to TFM estimation 414 and spatial covariance matrix estimation 416. TFM estimation 414 is performed based on the input signal and the output of steering vector multiplication 410. Spatial covariance matrix estimation 416 is performed based on the input signal and the output of TFM estimation 414. Steering vector estimation 418 is performed based on the output of spatial covariance matrix estimation 416. Steering vector estimation 418 sends its output to an additional steering vector multiplication 420 and delayed sum beamforming 404 of the downmix stage. Steering vector multiplication 420 is performed based on the output from ML model 406 and steering vector estimation 418 to produce a stereo output.

[0065] Figure 5 A logical block diagram is shown of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues in accordance with some embodiments. Figure 5 The algorithm has the same Figure 4 Same middle part (ML model 406) and same upmix (upmix 408); the only difference between the algorithms is downmix 502, which replaces Figure 4 Downmix 402. As shown, to obtain the time-frequency bins where the speech signal is dominant, the same ML models 504a and 504b are executed for each channel (eg, left and right inputs), respectively, and a time-frequency mask is estimated.

[0066] The output of each ML model 504 is provided for TFM estimation 506 (in an embodiment, ML models 504a and 504b can be the same type of model / same model). Buffering 508 (30 millisecond delay) also provides the left and right input signals to TFM estimation 506. TFM estimation is performed on the above inputs to provide an output of spatial covariance estimation 510. Spatial covariance estimation 510 is performed based on the output and the buffered left and right input signals. An output is provided for steering vector estimation 512. The output of steering vector estimation 512 and the left and right input signals are provided for delay-sum beamforming 514 to produce a monaural signal.

[0067] Figure 6A logic block diagram of an example algorithm for real-time low-complexity stereo speech enhancement for retaining spatial cues according to some embodiments is shown. In the depicted example, band gain hat gb is shared between each channel. As shown, downmix 602 stages utilize left input signal, right input signal and steering vector estimation 606 as input to perform delay-sum beamforming 604 (DSBF). ML model 608 and steering vector multiplication 610 receive the monaural signal output from downmix 602 stages. As shown, the example algorithm performs various operations in the upmix stage to produce an enhanced stereo signal, the operations include buffering 612, TFM estimation 614, spatial covariance matrix estimation 616, ML model 618 is applied to the left channel with shared spectral gain, and ML model 620 is applied to the left channel with shared spectral gain.

[0068] Figure 7 A logical block diagram of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments is shown. Figure 7-9 As shown, some stages send information to another stage so that the other stage can take this information into account in order to generate an output signal and / or output further information. The examples shown are Figure 4 The algorithm depicted performs target source enhancement as well as another source or noise enhancement. In an embodiment, the upmix 702 stage is a guided vector multiplication.

[0069] As shown, the example algorithm uses DSBF 704 for target source enhancement and DSBF 706 for another source or noise enhancement. The same ML models 708a and 708b are performed on the target source and the another source or noise (e.g., in an embodiment, as different ML models of the same type of model / same model). As shown, TFM estimation and SCM adaptation 710 are also performed. In an embodiment, the upmix 702 step can perform a steering vector multiplication.

[0070] Figure 8 A logical block diagram of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments is shown. The example shown is Figure 6 The depicted algorithm performs a target source enhancement as well as another source or noise enhancement. In an embodiment, the upmix 802a, 802b stage is a steering vector multiplication.

[0071] As shown, the example algorithm uses DSBF 804a for target source enhancement and DSBF 804b for another source or noise enhancement. The same ML models 806a and 806b (e.g., in an embodiment, different ML models of the same type of model / same model) are executed for the target source and the another source or noise. As shown, SCM adaptation / steering vector estimation 808 is also performed. Mixing 810 is performed on the outputs of the ML models 806a, 806b to generate an enhanced stereo output.

[0072] Fig. 9 A logical block diagram of an example algorithm for real-time low-complexity stereo speech enhancement preserving spatial cues according to some embodiments is shown. The example shown is Figure 8 An extension of the algorithm of ; instead of DSBF, blind source extraction 908 (BSE) is utilized. The depicted algorithm performs target source enhancement as well as another source or noise enhancement. In an embodiment, the upmix 902a, 902b stage is a steering vector multiplication.

[0073] As shown, the example algorithm uses filtering 904a for target source enhancement and filtering 904b for another source or noise enhancement. The same ML models 906a and 906b (e.g., in an embodiment, different ML models of the same type of model / same model) are executed on the target source and the another source or noise. As shown, BSE filter adaptation 908 is also performed. Mixing 910 is performed on the outputs of the ML models 906a, 906b to generate an enhanced stereo output.

[0074] Fig.10 A logical block diagram illustrating an example algorithm for real-time low-complexity speech enhancement preserving spatial cues for processing multiple input signals in accordance with some embodiments.

[0075] In various embodiments, any number of microphones may be used during a conference call, thereby generating multiple input signals / channels. The illustrated stereo speech enhancement system 1000 represents Figure 1 , wherein three or more different microphones provide three or more corresponding input signals. In embodiments, two or more input signals (e.g., based on two or more corresponding microphones) may be referred to as "multi-source" signals. A stereo input signal may be a multi-source input signal including a first input signal and a second input signal (e.g., a left input signal and a right input signal). Thus, downmix 1002, ML model 1004, and upmix 1006 may be performed on any number of input signals. Figure 1The downmix 102, ML model 104, and upmix 106 of the embodiment of the present invention perform the same or similar functions on the first and second input signals. The three or more input signals can be processed according to any of the techniques and algorithms discussed herein to generate three or more corresponding enhanced output signals (e.g., by performing real-time low-complexity speech enhancement that preserves spatial cues on the three or more input signals). This can allow binaural phase / level differences of any number of microphones to be taken into account when generating enhanced output signals for the microphones.

[0076] Although the Figure 2-10 Described and illustrated, but Figure 2-10 The various components shown and described in the can be easily adapted to other systems implementing audio enhancement. Figure 2-10 Other embodiments of audio enhancement are not intended to be limiting.

[0077] Fig.11 A high-level flow chart of various methods and techniques for implementing real-time low-complexity speech enhancement that preserves spatial cues according to some embodiments is presented. Various different systems and devices may implement the various methods and techniques described below, either alone or together. Therefore, any other systems or devices cited in the above examples and / or when performing the presented methods are not intended to limit other different components, modules, systems or devices.

[0078] As indicated at 1110, the stereo speech enhancement system may receive a stereo input signal (a first input signal and a second input signal) through an interface of the stereo speech enhancement system. For example, the stereo input signal may be received from two or more audio sensors, as described above with respect to Figure 2-3A As discussed above, and provided to a provider network service, such as audio transmission service 210, or may be received at an audio enhancement system implemented as part of an edge or other device that performs audio enhancement prior to transmitting the enhanced audio data to the provider network service, as described above with respect to Figure 3B-3C discussed above, or may be recorded, uploaded, or otherwise submitted to another system that implements audio enhancement, such as described above with respect to Figure 1 As discussed. In some embodiments, the audio data may be encrypted and / or compressed when received. Thus, the received audio data may be decrypted and decompressed by the audio enhancement system.

[0079] As indicated at 1120, the stereo speech enhancement system estimates spatial cues of a target speaker represented in a stereo input signal, and downmixes the stereo input signal to generate a monaural signal. The spatial cues are stored for subsequent use during the upmixing stage (e.g., as metadata / auxiliary information). As indicated at 1130, the stereo speech enhancement system may apply a machine learning model to the monaural signal to generate an enhanced monaural signal; noise is reduced or removed to generate an enhanced monaural signal. As indicated at 1140, the stereo speech enhancement system may upmix the enhanced monaural signal based on the estimated spatial cues of the target speaker to generate an enhanced stereo input signal (the stereo signal includes a first output signal and a second output signal). Based on the estimated spatial cues, the output signals of the stereo signal may have different binaural phases and / or levels, thereby allowing the listener to perceive the position of the speaker and the movement of the speaker when the speaker is speaking.

[0080] As indicated at 1150, in some embodiments, the stereo speech enhancement system may provide (e.g., store, transmit, or otherwise communicate) an enhanced stereo output signal (e.g., as described above with respect to Figure 1-3C For example, the enhanced stereo output signal can be sent by an audio (or audio-video) transmission service to another service as part of a two-way audio or video communication between devices that capture, send, and receive audio data. In some embodiments, the enhanced stereo output signal can be stored for subsequent access or playback.

[0081] The methods described herein may be implemented in various embodiments by any combination of hardware and software. For example, in one embodiment, the methods may be implemented on one or more computer systems (e.g., Fig.12 The present invention is implemented on or across one or more computer systems including one or more processors that execute program instructions stored on one or more computer-readable storage media coupled to the processors. The program instructions may implement the functions described herein (e.g., the functions of various servers and other components of the stereo speech enhancement system described herein). The various methods shown in the accompanying drawings and described herein represent example embodiments of the methods. The order of any method may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

[0082] Embodiments of real-time low complexity speech enhancement preserving spatial cues as described herein may be executed on one or more computer systems, which may interact with various other devices. Fig.12One such computer system is shown. In various embodiments, computer system 1200 may be any of various types of devices, including, but not limited to, a personal computer system, a desktop computer, a laptop computer, a notebook or netbook computer, a mainframe computer system, a handheld computer, a workstation, a network computer, a camera, a set-top box, a mobile device, a consumer device, a video game console, a handheld video game device, an application server, a storage device, a peripheral device such as a switch, a modem, a router, or generally any type of computing device, computing node, computing node, or electronic device.

[0083] In the illustrated embodiment, the computer system 1200 includes one or more processors 1210 coupled to a system memory 1220 via an input / output (I / O) interface 1230. The computer system 1200 further includes a network interface 1240 coupled to the I / O interface 1230, and one or more input / output devices 1250, such as a cursor control device 1260, a keyboard 1270, and a display 1280. The display 1280 may include a standard computer monitor and / or other display systems, technologies, or devices. In at least some embodiments, the input / output device 1250 may also include a touch-enabled or multi-touch device, such as a tablet or input board, through which a user enters input via a stylus-type device and / or one or more fingers. In some embodiments, it is contemplated that embodiments may be implemented using a single instance of the computer system 1200, while in other embodiments, multiple such systems or multiple nodes constituting the computer system 1200 may host different portions or instances of the embodiments. For example, in one embodiment, some elements may be implemented by one or more nodes of the computer system 1200 that are different from those nodes that implement other elements.

[0084] In various embodiments, computer system 1200 may be a uniprocessor system including one processor 1210, or a multiprocessor system including several processors 1210 (e.g., two, four, eight, or another suitable number). Processor 1210 may be any suitable processor capable of executing instructions. For example, in various embodiments, processor 1210 may be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as an x86, PowerPC, SPARC, or MIPS ISA, or any other suitable ISA. In a multiprocessor system, each of processors 1210 may typically, but not necessarily, implement the same ISA.

[0085] In some embodiments, at least one processor 1210 may be a graphics processing unit. A graphics processing unit or GPU may be considered as a dedicated graphics rendering device for a personal computer, workstation, game console, or other computing or electronic device. Modern GPUs may be very efficient in manipulating and displaying computer graphics, and their highly parallel structure may make them more efficient than typical CPUs for a series of complex graphics algorithms. For example, a graphics processor may implement the graphics primitive operations in a manner that makes it much faster to directly draw to the screen using a host central processing unit (CPU) than to perform multiple graphics primitive operations. In various embodiments, graphics rendering may be implemented at least in part by program instructions executed on one of such GPUs or executed in parallel on two or more of such GPUs. GPUs may implement one or more application programming interfaces (APIs) that allow programmers to call the functions of GPUs. Suitable GPUs may be commercially available from suppliers such as NVIDIA Corporation, ATI Technologies (AMD), etc.

[0086] The system memory 1220 may store program instructions and / or data accessible to the processor 1210. In various embodiments, the system memory 1220 may be implemented using any suitable memory technology such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory. In the illustrated embodiment, program instructions and data implementing desired functions such as rate mask post-filtering for audio enhancement as described above are shown as stored in the system memory 1220 as program instructions 1225 and data storage 1235, respectively. In other embodiments, program instructions and / or data may be received, sent, or stored on different types of computer accessible media or on similar media separate from the system memory 1220 or the computer system 1200. In general, non-transitory computer readable storage media may include storage media or memory media, such as magnetic media or optical media, for example, a disk or CD / DVD-ROM coupled to the computer system 1200 via the I / O interface 1230. Program instructions and data stored in computer-readable media can be transmitted via transmission media or signals (such as electrical signals, electromagnetic signals or digital signals), which can be transmitted via communication media such as networks and / or wireless links, such as can be implemented through network interface 1240.

[0087] In some embodiments, the I / O interface 1230 may coordinate I / O traffic between the processor 1210, the system memory 1220, and any peripheral devices in the device including the network interface 1240 or other peripheral interfaces such as the input / output device 1250. In some embodiments, the I / O interface 1230 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 1220) into a format suitable for use by another component (e.g., the processor 1210). In some embodiments, for example, the I / O interface 1230 may include support for devices attached through various types of peripheral buses (e.g., a variant of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, for example, the functionality of the I / O interface 1230 may be split into two or more separate components, such as a north bridge and a south bridge. In addition, in some embodiments, some or all of the functionality of the I / O interface 1230 (e.g., the interface to the system memory 1220) may be directly incorporated into the processor 1210.

[0088] The network interface 1240 may allow data to be exchanged between the computer system 1200 and other devices attached to the network (such as other computer systems) or between nodes of the computer system 1200. In various embodiments, the network interface 1240 may support: communications over a wired or wireless general data network (such as any suitable type of Ethernet); communications over a telecommunications / telephone network (such as an analog voice network or a digital fiber optic communications network); communications over a storage area network (such as a fiber optic acoustic channel SAN); or communications over any other suitable type of network and / or protocol.

[0089] In some embodiments, input / output devices 1250 may include one or more display terminals, keyboards, keypads, touch pads, scanning devices, voice or optical recognition devices, or any other device suitable for entering or retrieving data through one or more computer systems 1200. Multiple input / output devices 1250 may be present in computer system 1200 or may be distributed across different nodes of computer system 1200. In some embodiments, similar input / output devices may be separate from computer system 1200 and may interact with one or more nodes of computer system 1200 through wired or wireless connections (e.g., through network interface 1240).

[0090] like Fig.12As shown, memory 1220 may include program instructions 1225 that implement various methods and techniques as described herein, including the application of real-time low-complexity speech enhancement that retains spatial cues, and various data accessible to program instructions 1225. In one embodiment, program instructions 1225 may include software elements of embodiments as described herein and as shown in the accompanying drawings. Data storage 1235 may include data that can be used in embodiments. In other embodiments, other or different software elements and data may be included.

[0091] Those skilled in the art will appreciate that computer system 1200 is illustrative only and is not intended to limit the scope of the technology as described herein. Specifically, computer systems and devices may include any combination of hardware or software that can perform the indicated functions, including computers, personal computer systems, desktop computers, laptop computers, notebook or netbook computers, large computer systems, handheld computers, workstations, network computers, cameras, set-top boxes, mobile devices, network devices, Internet appliances, PDAs, wireless phones, pagers, consumer devices, video game consoles, handheld video game devices, application servers, storage devices, peripheral devices (such as switches, modems, routers) or generally any type of computing or electronic devices. Computer system 1200 may also be connected to other undisplayed devices, or may alternatively be operated as an independent system. In addition, in some embodiments, the functions provided by the displayed components may be combined in fewer components or distributed in other components. Similarly, in some embodiments, the functions of some components in the displayed components may not be provided and / or other additional functions may be used.

[0092] Those skilled in the art will also understand that, although various items are shown as being stored in memory or on a storage device when in use, for the purpose of memory management and data integrity, these items or parts thereof may be transmitted between memory and other storage devices. Alternatively, in other embodiments, some or all of the software components may be executed in a memory on another device and communicated with the computer system shown through inter-computer communication. Some or all of the system components or data structures may also be stored (e.g., as instructions or structured data) on a computer-accessible medium or portable article to be read by an appropriate drive, and various examples of the appropriate drive are described above. In some embodiments, instructions stored on a non-temporary computer-accessible medium separated from the computer system 1200 may be transmitted to the computer system 1200 via a transmission medium or signal (e.g., an electrical signal, an electromagnetic signal, or a digital signal conveyed by a communication medium such as a network and / or a wireless link). Various embodiments may further include receiving, sending, or storing instructions and / or data implemented according to the foregoing description on a computer-accessible medium. Therefore, the present invention may be practiced with other computer system configurations.

[0093] Note that any distributed system embodiment described herein or any of its components can be implemented as one or more network services. In certain embodiments, network-based services can be implemented by software and / or hardware systems that are designed to support the interoperable machine-to-machine interaction carried out by the network. Network-based services can have interfaces described in machine-processable formats such as network service description language (WSDL). Other systems can interact with network services in a manner specified by the description of the interface of network-based services. For example, network-based services can describe the various operations that other systems can call, and can describe specific application programming interfaces (APIs), and when requesting various operations, other systems may be expected to follow the specific application programming interfaces.

[0094] In various embodiments, a network-based service may be requested or invoked using a message that includes parameters and / or data associated with a network-based service request. Such messages may be formatted according to a specific markup language such as Extensible Markup Language (XML), and / or may be encapsulated using a protocol such as Simple Object Access Protocol (SOAP). To execute a network service request, a network-based service client may assemble a message including the request using an Internet-based application layer transport protocol such as Hypertext Transfer Protocol (HTTP), and deliver the message to an addressable endpoint (e.g., a Uniform Resource Locator (URL)) corresponding to the network service.

[0095] In some embodiments, a web service may be implemented using a representational state transfer ("RESTful") technique rather than a message-based technique. For example, a web service implemented according to RESTful techniques may be invoked via parameters included in an HTTP method (such as PUT, GET, or DELETE) rather than being encapsulated in a SOAP message.

[0096] The various methods shown in the accompanying drawings and described herein represent example embodiments of the methods. The methods may be implemented in software, hardware, or a combination thereof. The order of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc.

[0097] It will be apparent to those skilled in the art having the benefit of this disclosure that various modifications and changes may occur. The invention is intended to cover all such modifications and changes, and, therefore, the above description is to be regarded as illustrative rather than restrictive.

[0098] Embodiments of the present disclosure may be described in light of the following terms:

[0099] Clause 1. A system comprising:

[0100] at least one processor; and

[0101] a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a stereo speech enhancement system, the stereo speech enhancement system being configured to:

[0102] receiving, through an interface of the stereo speech enhancement system, a stereo input signal comprising a left input signal and a right input signal, wherein the left input signal comprises speech data corresponding to a target speaker and the right input signal comprises additional speech data corresponding to the target speaker;

[0103] estimating spatial cues of the target speaker;

[0104] downmixing the stereo input signal to generate a monaural signal;

[0105] applying a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal;

[0106] upmixing the enhanced monaural signal based at least on estimated spatial cues of the target speaker to generate an enhanced stereo output signal, wherein the enhanced stereo output signal comprises a left output signal and a right output signal, wherein the left output signal comprises enhanced speech data corresponding to the target speaker and the right output signal comprises additional enhanced speech data corresponding to the target speaker; and

[0107] The enhanced stereo output signal is sent to a destination through the interface of the stereo speech enhancement system.

[0108] Clause 2. The system of clause 1, wherein to downmix the stereo input signal, the stereo speech enhancement system is configured to:

[0109] Beamforming is applied to the stereo input signal.

[0110] Clause 3. A system according to any one of clauses 1 to 2, wherein to upmix the enhanced monaural signal, the stereo speech enhancement system is configured to:

[0111] Steering information of the target speaker is applied to the enhanced monaural signal, wherein the steering information is based on the estimated spatial cues.

[0112] Clause 4. A system according to any one of clauses 1 to 3, wherein the system further comprises an audio sensor that captures the stereo input signal, and wherein the destination is an audio transmission service implemented as part of a provider network, which transmits the enhanced stereo output signal to an audio playback device via a network connection.

[0113] Clause 5. A system according to any one of clauses 1 to 4, wherein the stereo speech enhancement system is implemented as part of an audio transmission service provided by a provider network, wherein the interface of the stereo speech enhancement system supports receiving the stereo input signal via a network connection, and wherein the destination is an audio playback device identified by the audio transmission service for the enhanced stereo input signal.

[0114] Clause 6. A method comprising:

[0115] receiving a multi-source input signal including at least a first input signal and a second input signal through an interface of a stereo speech enhancement system;

[0116] estimating spatial cues of a target speaker represented in the stereo input signal;

[0117] downmixing the multi-source input signal to generate a monaural signal;

[0118] applying a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal;

[0119] upmixing the enhanced monaural signal based at least on the estimated spatial cues of the target speaker to generate an enhanced multi-source output signal comprising a first output signal and a second output signal; and

[0120] The enhanced multi-source output signal is provided by the stereo speech enhancement system.

[0121] Clause 7. The method of clause 6, wherein downmixing the multi-source input signals comprises:

[0122] Beamforming is applied to the multi-source input signals.

[0123] Clause 8. The method of clause 7, wherein applying beamforming to the multi-source input signals comprises:

[0124] Delay-sum beamforming is applied to the stereo input signal.

[0125] Clause 9. The method of clause 8, wherein estimating the spatial cues comprises estimating a steering vector for the target speaker, and further comprising:

[0126] The delay-and-sum beamforming is applied using the estimated steering vector.

[0127] Clause 10. The method of any one of clauses 6 to 9, wherein upmixing the enhanced monaural signal comprises:

[0128] Steering information of the target speaker is applied to the enhanced monaural signal.

[0129] Clause 11. The method of any one of clauses 6 to 10, wherein estimating spatial cues of the target speaker comprises:

[0130] The steering vector for the target speaker is estimated using principal component analysis of the estimated spatial covariance matrix.

[0131] Clause 12. A method according to any one of clauses 6 to 11, wherein the multi-source input signal further comprises one or more additional input signals, and wherein the enhanced multi-source output signal further comprises one or more additional output signals corresponding to the one or more additional input signals, respectively.

[0132] Clause 13. The method of any of clauses 6 to 12, wherein providing the enhanced multi-source output signal comprises storing the enhanced multi-source output signal to a data storage service provided by a provider network.

[0133] Clause 14. A method according to any one of clauses 6 to 13, wherein the stereo speech enhancement system is implemented as part of a device including an audio sensor that captures the multi-source input signal, and wherein providing the enhanced multi-source output signal includes sending the enhanced multi-source output signal to an audio transmission service implemented as part of a provider network, which transmits the enhanced multi-source output signal to an audio playback device via a network connection.

[0134] Clause 15. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to:

[0135] receiving a stereo input signal including a first input signal and a second input signal through an interface of a stereo speech enhancement system;

[0136] estimating spatial cues of a target speaker represented in the stereo input signal;

[0137] downmixing the stereo input signal to generate a monaural signal;

[0138] causing application of a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal;

[0139] upmixing the enhanced monaural signal based at least on the spatial cues of the target speaker to generate an enhanced stereo output signal comprising a first output signal and a second output signal; and

[0140] The enhanced stereo output signal is sent by the stereo speech enhancement system to a destination.

[0141] Clause 16. The one or more non-transitory computer-readable storage media of clause 15, wherein when downmixing the stereo input signal to generate a monaural signal, the program instructions cause the one or more computing devices to perform the following:

[0142] Beamforming is applied to the stereo input signal.

[0143] Clause 17. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 16, wherein when downmixing the stereo input signal to generate a monaural signal, the program instructions cause the one or more computing devices to perform the following:

[0144] Delay-and-sum beamforming is applied using the estimated spatial cues.

[0145] Clause 18. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 17, wherein in upmixing the enhanced monaural signal, the program instructions cause the one or more computing devices to perform the following:

[0146] Steering information of the target speaker is applied to the enhanced monaural signal.

[0147] Clause 19. One or more non-transitory computer-readable storage media according to any one of clauses 15 to 19, wherein the stereo speech enhancement system is implemented as part of a device including an audio sensor that captures the stereo input signal, and wherein sending the enhanced stereo output signal comprises sending the enhanced stereo output signal to an audio transmission service implemented as part of a provider network, the audio transmission service transmitting the enhanced stereo output signal to an audio playback device via a network connection.

[0148] Clause 20. One or more non-transitory computer-readable storage media as described in any of clauses 15 to 19, wherein the stereo input signal is captured together with corresponding video data, and the video data is provided to the same destination as the enhanced stereo output signal.

Claims

1. A system comprising: at least one processor; and a memory storing program instructions that, when executed by the at least one processor, cause the at least one processor to implement a stereo speech enhancement system, the stereo speech enhancement system being configured to: receiving a multi-source input signal including at least a first input signal and a second input signal through an interface of the stereo speech enhancement system; estimating spatial cues of a target speaker represented in the multi-source input signals; downmixing the multi-source input signal to generate a monaural signal; applying a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal; upmixing the enhanced monaural signal based at least on the estimated spatial cues of the target speaker to generate an enhanced multi-source output signal; as well as The enhanced multi-source output signal is provided through the interface of the stereo speech enhancement system.

2. The system of claim 1, wherein to downmix the multi-source input signal, the stereo speech enhancement system is configured to: Beamforming is applied to the multi-source input signals.

3. The system according to any one of claims 1 to 2, wherein for upmixing the enhanced monaural signal, the stereo speech enhancement system is configured to: Steering information of the target speaker is applied to the enhanced monaural signal, wherein the steering information is based on the estimated spatial cues.

4. The system of any one of claims 1 to 3, wherein the system further comprises an audio sensor that captures the multi-source input signal, and wherein the destination is an audio transmission service implemented as part of a provider network, the audio transmission service transmitting the enhanced multi-source output signal to an audio playback device over a network connection.

5. A system according to any one of claims 1 to 4, wherein the stereo speech enhancement system is implemented as part of an audio transmission service provided by a provider network, wherein the interface of the stereo speech enhancement system supports receiving the multi-source input signal via a network connection, and wherein the destination is an audio playback device identified by the audio transmission service for the enhanced multi-source input signal.

6. A method comprising: receiving a multi-source input signal including at least a first input signal and a second input signal through an interface of a stereo speech enhancement system; estimating spatial cues of a target speaker represented in the multi-source input signals; downmixing the multi-source input signal to generate a monaural signal; applying a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal; upmixing the enhanced monaural signal based at least on estimated spatial cues of the target speaker to generate an enhanced multi-source output signal comprising at least a first output signal and a second output signal; as well as The enhanced multi-source output signal is provided by the stereo speech enhancement system.

7. The method of claim 6, wherein downmixing the multi-source input signals comprises: Beamforming is applied to the multi-source input signals.

8. The method of claim 7, wherein applying beamforming to the multi-source input signals comprises: Delay-and-sum beamforming is applied to the multi-source input signals.

9. The method of claim 8, wherein estimating the spatial cues comprises estimating a steering vector for the target speaker, and further comprising: The delay-and-sum beamforming is applied using the estimated steering vector.

10. The method according to any one of claims 6 to 9, wherein upmixing the enhanced monaural signal comprises: Steering information of the target speaker is applied to the enhanced monaural signal.

11. One or more non-transitory computer-readable storage media storing program instructions that, when executed on or across one or more computing devices, cause the one or more computing devices to: receiving a multi-source input signal including at least a first input signal and a second input signal through an interface of a stereo speech enhancement system; estimating spatial cues of a target speaker represented in the multi-source input signals; downmixing the multi-source input signal to generate a monaural signal; causing application of a machine learning model to the monaural signal to generate an enhanced monaural signal, wherein noise in the monaural signal is reduced or removed to generate the enhanced monaural signal; upmixing the enhanced monaural signal based at least on the spatial cues of the target speaker to generate an enhanced multi-source output signal comprising at least a first output signal and a second output signal; as well as The enhanced multi-source output signal is sent by the stereo speech enhancement system to a destination.

12. The one or more non-transitory computer-readable storage media of claim 11, wherein in downmixing the multi-source input signal to generate a monaural signal, the program instructions cause the one or more computing devices to perform the following: Beamforming is applied to the multi-source input signals.

13. The one or more non-transitory computer-readable storage media of any one of claims 11 to 12, wherein when downmixing the multi-source input signal to generate a monaural signal, the program instructions cause the one or more computing devices to perform the following: Delay-and-sum beamforming is applied using the estimated spatial cues.

14. The one or more non-transitory computer-readable storage media of any one of claims 11 to 13, wherein when upmixing the enhanced monaural signal, the program instructions cause the one or more computing devices to: Steering information of the target speaker is applied to the enhanced monaural signal.

15. The one or more non-transitory computer-readable storage media of any one of claims 11 to 14, wherein the multi-source input signal is captured with corresponding video data that is provided to the same destination as the enhanced multi-source output signal.

Images