Text echo cancellation

Abstract

A method (400) includes receiving an overlapping audio signal (202) comprising audio spoken by a speaker (10) overlapping a segment (156) of synthesized playback audio (154). The method further includes encoding a character sequence corresponding to the synthesized playback audio into a text embedding representation (212). For each character in the character sequence, the method further includes using the text embedding representation to generate a corresponding cancellation probability (222). The cancellation probability indicates the likelihood of the corresponding character being associated with a segment of the synthesized playback audio that overlaps with the audio spoken by the speaker in the overlapping audio signal.

Description

Technical Field

[0001] This disclosure relates to text echo cancellation. Background Technology

[0002] When audio systems capture and transmit audio, these peripheral devices can be affected by echoes. Echoes typically occur when an audible signal generated from an audio output device (e.g., a speaker) propagates as a sound wave through the acoustic environment (e.g., air) and a modified version of that wave reflects back to the microphone. Another form of echo is electrical echo, which arises due to undesirable electrical coupling effects between the audio output device (e.g., a speaker) and the microphone. The signal generated by such an echo path is then transmitted as an audio echo. Echoes can be generated due to various factors such as, for example, the proximity of the peripheral device or the quality of the peripheral device. When an echo occurs in a speech-enabled environment where one or more devices are performing speech processing, the echo can adversely affect the accuracy of the speech processing; thus affecting the user's experience with the device and / or the speech system. As people increasingly use echo-affected systems for communication, echo cancellation systems and methods can be implemented to improve the quality of echo-affected speech processing. Summary of the Invention

[0003] One aspect of this disclosure provides a computer-implemented method that, when executed on data processing hardware, causes the hardware to perform operations including: receiving an overlapping audio signal comprising audio spoken by a speaker overlapping a segment of synthesized playback audio; and encoding a character sequence corresponding to the synthesized playback audio into a text embedding representation. The operation further includes generating a corresponding cancellation probability for each character in the character sequence using the text embedding representation, and generating an enhanced audio signal by removing segments of the synthesized playback audio from the overlapping audio signal using a cancellation neural network configured to receive the overlapping audio signal and the corresponding cancellation probabilities generated for each character in the character sequence as input. Each corresponding cancellation probability indicates the likelihood of a corresponding character being associated with a segment of the synthesized playback audio, which is overlapped in the overlapping audio signal by audio spoken by a speaker.

[0004] Implementations of this disclosure may include one or more of the following optional features. In some implementations, encoding the character sequence includes encoding each character in the character sequence into a corresponding character embedding to generate a character embedding sequence. In these implementations, the overlapping audio signal may include a sequence of frames, and generating a corresponding cancellation probability for each character in the character sequence may include applying weights to the corresponding character embedding using an attention mechanism when the corresponding character embedding corresponds to a frame in the frame sequence of the overlapping audio signal. Each frame in the frame sequence corresponds to a portion of the audio spoken by the speaker that overlaps with a segment of the synthesized playback audio.

[0005] Text-to-speech (TTS) systems can convert character sequences into synthesized speech, including synthesized playback audio. Optionally, the text embedding representation may include a single fixed-dimensional text embedding vector. In some examples, the text encoder of a text encoding neural network encodes the character sequence corresponding to the synthesized playback audio into a text embedding representation. In these examples, the text encoder may be shared by a TTS system configured to generate synthesized playback audio from the character sequence.

[0006] In some implementations, the operation further includes training the elimination neural network on multiple training examples, wherein each training example includes: a ground truth audio signal corresponding to the non-synthetic speech, a training overlapping audio signal that overlaps with the synthetic audio signal, and a corresponding text representation of the synthetic audio signal. Here, the text representation includes a sequence of characters.

[0007] The elimination neural network can include a Long Short-Term Memory (LSTM) network with multiple LSTM layers. The operation can further include receiving an indication that a text representation of the synthesized playback audio is available.

[0008] Another aspect of this disclosure provides a system including data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed by the data processing hardware, cause the data processing hardware to perform operations including: receiving an overlapping audio signal comprising audio spoken by a speaker overlapping a segment of synthesized playback audio; and encoding a character sequence corresponding to the synthesized playback audio into a text embedding representation. The operation further includes generating a corresponding cancellation probability for each character in the character sequence using the text embedding representation, and generating an enhanced audio signal by removing segments of the synthesized playback audio from the overlapping audio signal using a cancellation neural network configured to receive the overlapping audio signal and the corresponding cancellation probabilities generated for each character in the character sequence as input. Each corresponding cancellation probability indicates the likelihood of the corresponding character being associated with a segment of the synthesized playback audio, which is overlapped in the overlapping audio signal by audio spoken by a speaker.

[0009] This aspect may include one or more of the following optional features. In some implementations, encoding the character sequence includes encoding each character in the character sequence into a corresponding character embedding to generate a character embedding sequence. In these implementations, the overlapping audio signal may include a sequence of frames, and generating a corresponding cancellation probability for each character in the character sequence may include applying weights to the corresponding character embedding using an attention mechanism when the corresponding character embedding corresponds to a frame in the frame sequence of the overlapping audio signal. Each frame in the frame sequence corresponds to a portion of the audio spoken by the speaker that overlaps with a segment of the synthesized playback audio.

[0010] Text-to-speech (TTS) systems can convert character sequences into synthesized speech, including synthesized playback audio. Optionally, the text embedding representation may include a single fixed-dimensional text embedding vector. In some examples, the text encoder of a text encoding neural network encodes the character sequence corresponding to the synthesized playback audio into a text embedding representation. In these examples, the text encoder may be shared by a TTS system configured to generate synthesized playback audio from the character sequence.

[0011] In some implementations, the operation further includes training the elimination neural network on multiple training examples, wherein each training example includes a ground truth audio signal corresponding to the non-synthetic speech, a training overlapping audio signal that overlaps with the synthetic audio signal, and a corresponding text representation of the synthetic audio signal. Here, the text representation includes a sequence of characters.

[0012] The elimination neural network can include a Long Short-Term Memory (LSTM) network with multiple LSTM layers. The operation can further include receiving an indication that a text representation of the synthesized playback audio is available.

[0013] Details of one or more embodiments of this disclosure are set forth in the accompanying drawings and the following description. Other aspects, features, and advantages will be apparent from the specification, the drawings, and the claims. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of an example speech environment.

[0015] Figures 2A to 2C yes Figure 1 A schematic diagram of an example text echo cancellation system for a speech environment.

[0016] Figure 2D This is a schematic diagram of an example text echo cancellation system that shares a text encoder with a text-to-speech system.

[0017] Figure 2EThis is a schematic diagram of an example text echo cancellation system that does not have an attention mechanism.

[0018] Figure 3 It is used for training Figure 1 A schematic diagram of an example training process for a text echo cancellation system.

[0019] Figure 4 This is a flowchart of an example layout of operations for generating enhanced audio signals to eliminate echoes.

[0020] Figure 5 This is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein.

[0021] In the various figures, the same reference numerals indicate the same elements. Detailed Implementation

[0022] Speech-enabled devices are capable of generating synthetic playback audio and communicating it to one or more users within a speech environment. Here, synthetic playback audio refers to audio generated by a speech-enabled device that originates from the speech-enabled device itself or a machine processing system associated with it, rather than from a person or other audible sound source outside the speech-enabled device. Generally, speech-enabled devices use text-to-speech (TTS) systems to generate synthetic playback audio. TTS systems convert text into an audio representation of that text, where the audio representation of the text is modeled as similar to the audio representation of spoken language.

[0023] When the audio output component (e.g., speaker) of a voice-enabled device outputs synthesized playback audio, the audio capture component (e.g., microphone) of the voice-enabled device can still actively capture audible sounds within the speech environment (i.e., listen). This means that a portion of the synthesized playback audio output from the speaker will be received at the audio capture component as an echo. Unfortunately, using this form of echo from the synthesized playback audio, it may be difficult for a speech recognizer implemented at the voice-enabled device, or at a remote system communicating with the voice-enabled device, to understand the spoken words that occur during the echo from the synthesized playback audio. In other words, voice-enabled devices often generate synthesized playback audio as a response to a query or command from a user of the voice-enabled device. For example, a user might ask the voice-enabled device, “What will the weather be like today?” When the voice-enabled device receives this query or question from the user, the voice-enabled device or the remote system communicating with the voice-enabled device must initially determine or process the spoken words from the user. By processing spoken words, voice-enabled devices can recognize that spoken words correspond to queries from users (e.g., about the weather), and as a query, users expect a response from the voice-enabled device.

[0024] Typically, speech-enabled devices use speech recognition systems (e.g., Automatic Speech Recognition (ASR) systems) to determine the context of spoken utterance. The speech recognition system receives audio signals or audio data and generates a transcription of text representing the characters, words, and / or sentences spoken in the audio signal. However, speech recognition can become more complex when the speech capture component of a speech-enabled device simultaneously receives echoes and / or distortions as part of one or more utterances spoken by a user to the speech-enabled device. For example, one or more microphones of the speech-enabled device are fed a portion of a synthesized playback audio signal as echoes or acoustic feedback. The echoes from the synthesized playback audio combine with one or more spoken utterances, causing the speech-enabled device to receive an audio signal with overlapping speech. Here, overlapping speech refers to an instance in the audio signal where the echo signal from the synthesized playback audio occurs at the same time (i.e., simultaneously or concurrently) with the one or more spoken utterances. When this overlapping speech occurs, the speech recognition system may experience difficulties processing the audio signal received at the speech-enabled device. In other words, overlapping speech can impair the ability of a speech recognition system to generate accurate transcriptions for the one or more spoken utterances. Without accurate transcriptions from the speech recognition system, a speech-enabled device may be unable to accurately respond to or respond to queries or commands from a user's spoken utterances. Alternatively, a speech-enabled device may want to avoid using its processing resources that attempt to interpret audible sounds that are actually from synthetic playback audio signals and / or echoes from the surrounding environment.

[0025] One method to prevent distortion or echo captured by the audio capture component of a voice-enabled device is to use an acoustic echo cancellation (AEC) system. In an AEC system, the audio signal is used to eliminate the echo associated with the synthesized playback audio signal. For example, in some configurations, the AEC system includes a neural network that learns how to eliminate echoes from the synthesized playback audio signal from examples of audio signals (e.g., examples of synthesized playback audio signals). While this method is generally successful in eliminating echoes, the AEC system may not be the most resource-efficient approach. In other words, to eliminate the echo of the synthesized playback audio signal, the synthesized playback audio signal, which will be output by the audio output component of the voice-enabled device, is sent to the AEC system. Unfortunately, the synthesized playback audio signal, as an audio file, is typically a relatively large file size. This relatively large file size introduces potential performance problems, such as latency, when communicating the synthesized playback audio signal between systems of voice-enabled devices. Furthermore, communicating and processing the relatively large file size can strain the processing resources of the voice-enabled device. To exacerbate these potential problems, the AEC system can reside remotely on a network server instead of on the device. For example, when speech recognition is implemented as a service, acoustic echo cancellation requires sending synthesized playback audio from the TTS system as a side input to the server hosting the speech recognition system. In cases where the AEC system and / or TTS system resides on a remote system, audio files may need to be transferred one or more times between the speech-enabled device and the remote system, subjecting the process to potential latency or other transmission issues.

[0026] Another approach that can be adopted by speech-enabled devices to remove unwanted echoes is a filtering system. In a filtering system, the system learns the speaker's vocal characteristics, enabling it to filter out components of the audio signal that are irrelevant to the speaker's voice. Here, the filtering system requires audio samples of the target speaker's spoken utterance to learn the target speaker's vocal characteristics. For example, the filtering system generates an embedding vector representing the target speaker's vocal characteristics. This embedding vector can then be used as auxiliary input to remove any captured audio signals that do not belong to the target speaker. However, using a filtering method requires additional auxiliary input to generate the embedding vector, as well as a learning process to develop the embedding vector. Furthermore, this method relies on the target speaker to provide audio samples and a sufficient number of audio samples to accurately learn the target speaker's vocal characteristics. While perhaps effective, these additional inputs can add costs to the speech-enabled device or the system load of the speech-enabled device.

[0027] To address some of these concerns using echo cancellation, speech-enabled devices can instead use a Text Echo Cancellation (TEC) system. In a TEC system, instead of using a synthesized playback audio signal that generates the echo to cancel it, the TEC system uses text to cancel the echo. Since the synthesized playback audio signal originates from input text (e.g., text in a TTS system), the input text used to generate the synthesized playback audio signal can be used by the TEC system to cancel the echo generated by the synthesized playback audio signal. Here, by using text, the TEC receives a text file that is smaller in size than the file used for the audio signal (e.g., the audio file used for synthesizing the playback audio signal). For example, the text file can be on the scale of data bytes, while the audio file can be on the scale of kilobytes, megabytes, or even larger (e.g., depending on signal quality). By using a text file that is smaller in size than the audio signal, this method can reduce latency because the audio signal does not need to communicate over the network, while simultaneously avoiding the consumption of processing time and / or resources to communicate the text file. Furthermore, TEC can facilitate the efficient use of processing resources because the TEC system can utilize text already being generated by a system such as a TTS system to produce the synthesized playback audio signal. Therefore, by using text, the TEC method can avoid problems caused by AEC systems, such as latency, or by filtering systems, such as training and the need for additional input.

[0028] refer to Figure 1 In some implementations, the voice environment 100 includes a user 10 who communicates spoken utterance 12 to a voice-enabled device 110 (also referred to as device 110 or user device 110). The user 10 (i.e., the speaker of utterance 12) can speak utterance 12 as a query or command to request a response from device 110. Device 110 is configured to capture the voices of one or more users 10 within the voice environment 100. Here, audio sounds can refer to the spoken utterance 12 of user 10, which serves as an audible query, command, or audible communication captured by device 110. A voice-enabled system of device 110 or a voice-enabled system associated with device 110 can respond to a query for a command by answering the query and / or causing the command to be executed.

[0029] Here, device 110 captures audio signals 202 (also referred to as audio data) of spoken words 12 of user 10. Device 110 can correspond to any computing device associated with user 10 and capable of receiving audio signals 202. Some examples of user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart appliances, and Internet of Things (IoT) devices, smart speakers, etc. Device 110 includes data processing hardware 112 and memory hardware 114 that communicates with and stores instructions that, when executed by data processing hardware 112, cause data processing hardware 112 to perform one or more operations. In some examples, device 110 includes one or more applications (i.e., software applications), each of which can utilize one or more voice processing systems 140, 150, 160, 200 associated with device 110 to perform various functions within the application. For example, device 110 includes an auxiliary application configured to communicate synthesized playback audio 154 to user 10 to assist user 10 in performing various tasks.

[0030] Device 110 further includes an audio subsystem having an audio capture device (e.g., a microphone) 116 for capturing spoken words 12 within a speech environment 100 and converting them into electrical signals, and a voice output device (e.g., a speaker) 118 for communicating audible audio signals (e.g., synthesized playback audio 154 from device 110). While in the illustrated example, device 110 implements a single audio capture device 116, without departing from the scope of this disclosure, device 110 may implement an array of audio capture devices 116, whereby one or more audio capture devices 116 in the array may not physically reside on device 110 but communicate with the audio subsystem (e.g., peripherals of device 110). For example, device 110 may correspond to a vehicle infotainment system that makes full use of a microphone array that spans the vehicle's location.

[0031] Furthermore, device 110 is configured to communicate with remote system 130 via network 120. Remote system 130 may include remote resources 132, such as remote data processing hardware 134 (e.g., a remote server or CPU) and / or remote memory hardware 136 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 132 to perform various functionalities related to speech processing and / or synthesis playback communication. For example, device 110 is configured to perform speech recognition using speech recognition system 140, text-to-speech conversion using TTS system 150, acoustic echo cancellation using AEC system 160, and / or text echo cancellation using TEC system 200. These systems 140, 150, 160, and 200 may reside on device 110 (referred to as on-device systems) or reside remotely (e.g., on remote system 130) but communicate with device 110. In some examples, some of these systems 140, 150, 160, 200 reside locally or on a device, while others reside remotely. In other words, any of these systems 140, 150, 160, 200 can be any combination of local or remote. For example, when systems 140, 150, 160, 200 are of considerable size or processing requirements, systems 140, 150, 160, 200 may reside in a remote system 130. However, when device 110 can support the size or processing requirements of one or more systems 140, 150, 160, 200, said one or more systems 140, 150, 160, 200, said one or more systems may reside on device 110 using data processing hardware 112 and / or memory hardware 114. Optionally, said one or more of systems 140, 150, 160, 200 may reside locally / on a device and remotely. For example, when a connection to network 120 is available between device 110 and remote system 130, one or more of systems 140, 150, 160, and 200 may be executed on remote system 130 by default. However, when the connection is lost or network 120 is unavailable, systems 140, 150, 160, and 200 are instead executed locally on device 110.

[0032] Speech recognition system 140 receives audio signal 202 as input and transcribes it into transcript 142 as output. Generally, by converting audio signal 202 into transcript 142, speech recognition system 140 allows device 110 to identify when spoken words 12 from user 10 correspond to a query, command, or some other form of audio communication. Transcription 142 refers to a sequence of text that device 110 can then use to generate a response to a query or command. For example, if user 10 asks device 110 the question “what will the weather be like today?”, device 110 will pass an audio signal corresponding to the question “what will the weather be like today” to speech recognition system 140. Speech recognition system 140 converts the audio signal into a transcript including the text “what will the weather be like today?”. Device 110 can then use the text or a portion of the text to determine a response to the query. For example, to determine the weather for a given day (i.e., today), device 110 might transmit text (e.g., “what will the weather be like today?”) or ask a search engine to identify portions of the text (e.g., “weather” and “today”). The search engine can then return one or more search results interpreted by device 110 to generate a response for user 10.

[0033] In some implementations, device 110 or a system associated with device 110 recognizes text 152 communicated by device 110 to user 10 as a response to a query of spoken utterance 12. Device 110 can then use TTS system 150 to convert text 152 into corresponding synthetic playback audio 154 so that device 110 can communicate with user 10 (e.g., audible communication with user 10) as a response to a query of spoken utterance 12. In other words, TTS system 150 receives text 152 as input and converts text 152 into synthetic playback audio 154 as output, where synthetic playback audio 154 is an audio signal that defines the audible rendition of text 152. In some examples, TTS system 150 includes a text encoder (e.g., text embedding) that processes text 152 into an encoded format (e.g., text embedding). Figures 2A to 2E(Text encoder 210). Here, the TTS system 150 can use a trained text-to-speech model to generate synthesized playback audio 154 from the encoded format of text 152. Once generated, the TTS system 150 communicates the synthesized playback audio 154 to device 110, allowing device 110 to output the synthesized playback audio 154. For example, device 110 outputs the synthesized playback audio 154 "today is sunny" at its speaker 118.

[0034] Continue to refer to Figure 1 When device 110 outputs synthesized playback audio 154, the synthesized playback audio 154 generates an echo 156 captured by audio capture device 116. Unfortunately, in addition to the echo 156, audio capture device 116 can also simultaneously capture another spoken word 12 from user 10. For example, Figure 1 The illustration depicts a scenario where, when device 110 outputs synthesized playback audio 154, user 10 inquires about the weather by stating "what about tomorrow?" in spoken words 12. Here, both spoken words 12 and echo 156 are simultaneously captured at audio capture device 116 to form audio signal 202. In other words, audio signal 202 includes overlapping audio signals, where a portion of spoken words 12 by user 12 overlaps with a portion of synthesized playback audio 154 output from speaker 118 of device 110. Figure 1 In the captured audio signal 202, portions of utterance 12 that overlap with each other and portions of synthesized playback audio 154 are shown as overlapping regions 204. For the overlapping audio signal 202, the speech recognition system 140 may be able to recognize the question corresponding to the weather inquiry “what about tomorrow” in the audio signal 202, because utterance 12 is mixed with the echo 156 of synthesized playback audio 154 to form the audio signal 202 received at device 110 (e.g., at audio capture device 116).

[0035] To prevent such problems, device 110 can communicate audio signal 202 to TEC system 200 before speech recognition system 140 processes it. By communicating audio signal 202 to TEC system 200 before speech recognition system 140, TEC system 200 is configured to generate enhanced audio signal 206, which removes echo 156 from the synthesized playback audio 154 or a portion thereof; effectively eliminating echo 156. That is, enhanced audio signal 206 refers to a modified version of audio signal 202 that enables speech recognition system 140 to recognize spoken utterance 12 contained in audio signal 202. Here, to generate enhanced audio signal 206, TEC system 200 receives overlapping audio signal 202 and text 152 corresponding to synthesized playback audio 154.

[0036] In some examples, device 110 includes an AEC system 160 in conjunction with TEC system 200. By including both systems 160 and 200, device 110 has the ability to cancel echoes regardless of when or not text is available for the cancellation process. In some implementations, device 110 determines whether to use AEC system 160 based on text indicating that at least a portion of the audio contributing to audio signal 202 is available. For example, device 110 is playing music that generates an echo at audio capture device 116 at the same time that user 10 speaks 12 to device 110. In this case, no text is available for the music being output by device 110, and therefore device 110 recognizes that no text is available and uses AEC system 160 to cancel the echo present at audio capture device 116. Conversely, Figure 1 The illustration shows device 110 outputting synthesized playback audio 154, where device 110 or some system associated with device 110 has generated text 152 corresponding to the synthesized playback audio 154. In this example, device 110 uses TEC system 200 instead of AEC system 160 to cancel echo 156 at audio capture device 116. Here, device 110 chooses TEC system 200 instead of AEC system 160 because TEC system 200 can potentially cancel / remove echo 156 more quickly based on the smaller text file size involved in the cancellation process at AEC system 160, compared to the larger audio file size involved in the cancellation process at TEC system 200.

[0037] refer to Figures 2A to 2DThe TEC system 200 is configured to receive an audio signal 202 including an overlapping region 204 and text 152 (also referred to as a text representation) for generating synthesized playback audio 154 as input. Using these inputs, the TEC system 200 is configured to generate an enhanced audio signal 206 by removing at least one segment of the synthesized playback audio 154 from the audio signal 202 within the overlapping region 204. To perform these operations, the TEC system 200 includes a combination of a text encoder 210, an attention mechanism 220, and an elimination neural network 230 (also referred to as network 230). The text encoder 210 may include a text encoder neural network.

[0038] Text encoder 210 is configured to receive text 152 corresponding to the synthesized playback audio 154 and encode the text 152 into encoded text referred to as text embedding representation 212 (or simply text embedding 212). TEC system 200 may include its own text encoder 210 (e.g., such as...). Figure 2A (as shown), or the text encoder 210 can be shared between the TEC 200 and the TTS system 150 (e.g., as shown). Figure 2D (As shown). The embedding generated by text encoder 210 refers to an n-dimensional mathematical vector representing text 152, where each dimension may correspond to a feature or characteristic of the text (e.g., a categorical variable). In some embodiments, text encoder 210 encodes the entire text 152 into a single fixed-dimensional text embedding 212 representing the synthesized playback audio 152. In other embodiments, text encoder 210 encodes text 152 into a sequence of text embeddings, where the embeddings are not of fixed length but have a length corresponding to multiple characters in text 152. For example, text embedding 212 is character embedding 214, 214. A-N The sequence of text embeddings. For words, or more specifically, for words in the English language, each word is divided into character units called letters. Thus, each character embedding 214 can represent a letter of the text 152, such that each character embedding 214 in the text embedding sequence corresponds to a letter in the twenty-six-letter English alphabet. Although the text embedding 212 is described relative to a twenty-six-dimensional vector used for the English alphabet, the text embedding 212 generated by the text encoder 210 can be any form of embedding used together with the elimination neural network 230 to generate the enhanced audio 206.

[0039] refer to Figure 2BText embedding 212 is a sequence of character embeddings 214, where each letter of text 152 is represented by a 26-dimensional vector. In other words, here, each dimension of the vector is assigned to a letter in the English alphabet. In this example, text 152 “today is sunny” is fourteen characters (including spaces), and based on this, text encoder 210 generates text embedding 212, which is a sequence of fourteen character embeddings 214. For illustration, the character embedding 214 for each letter in text 152 includes a value at its position in the alphabet. For example, the letter “T” is the 20th letter of the English alphabet, and the character embedding 214T of the letter “T” in text 152 “today is sunny” used to synthesize playback audio 154 is a vector in which the 20th dimension of the vector includes a value specifying the character as “T” (e.g., shown as “1”). The letter “O” is the 15th letter of the alphabet, and the 15th dimension of the character embedding for the letter “O” includes the value one. The letter "D" is the 4th letter of the alphabet, and its character embedding includes the value 1 in the 4th dimension. The letter "A" is the 1st letter of the alphabet, and its character embedding includes the value 1 in the 1st dimension. The letter "Y" is the 25th letter of the alphabet, and its character embedding includes the value 1 in the 25th dimension. The letter "I" is the 9th letter of the alphabet, and its character embedding includes the value 1 in the 9th dimension. The letter "S" is the 19th letter of the alphabet, and its character embedding includes the value 1 in the 19th dimension. The letter "U" is the 21st letter of the alphabet, and its character embedding includes the value 1 in the 21st dimension. The letter "N" is the 14th letter of the alphabet, and its character embedding includes the value 1 in the 14th dimension.

[0040] Attention mechanism 220 is configured to receive text embeddings 212 (e.g., a sequence of character embeddings 214) and determine elimination probabilities 222, which indicate the likelihood of a character associated with the text 152 corresponding to the synthesized playback audio 154 within the overlapping region 204. In other words, attention mechanism 220 can determine a corresponding elimination probability 222 for each character embedding 214 in the character embedding sequence 214 output from text encoder 210. Using this elimination probability 222, network 230 can focus on removing portions of the synthesized playback audio 154 to generate an enhanced audio signal 206. In other words, attention mechanism (e.g., attention mechanism 220) can correspond to an alignment model (e.g., a position-sensitive model) configured such that the neural network (e.g., network 230) can focus on specific components of the input (e.g., audio signal 202) to the neural network (e.g., network 230). Here, attention mechanism 220 directs network 230 to the location within the audio signal 202 where echo cancellation is required (i.e., the overlapping region 204). By using attention mechanism 220, TEC system 200 can understand where the alignment of text 152 of synthesized playback audio 154 occurs within audio signal 202. In other words, when TEC system 200 only receives text 152 corresponding to synthesized playback audio 154 and audio signal 202, TEC system 200 (specifically, ablation neural network 230) has no awareness of which part of text 152 corresponds to synthesized playback audio 154 occurring within the overlapping region 204 of audio signal 202. Therefore, in order to understand where the echo 156 from synthesized playback audio 154 mixes with the speech utterance 12 in audio signal 202, TEC system 200 can use attention mechanism 220.

[0041] In some examples, the attention mechanism 220 determines the cancellation probability 222 by stepping through each frame F of the audio signal 202. Here, the audio signal 202 comprises multiple frames F that divide the audio signal 202 into segments, F 1-n ,like Figure 2C As shown. For each frame F, the attention mechanism 220 determines whether the echo 156 from the synthesized playback audio 154 overlaps with the utterance 12 at that frame F. Figure 2CAs shown, the attention mechanism identifies three frames F corresponding to the overlapping region 204 where the echo 156 from the synthesized playback audio 154 overlaps with the utterance 12. In some configurations, when the attention mechanism 220 identifies the one or more frames F of the overlapping region 204, the attention mechanism 220 identifies / determines the one or more characters of the text 152 corresponding to the synthesized playback audio 154 of each frame F in the overlapping region 204 (e.g., the character "nny" shown as the word "sunny"). Using the specific characters of the text 152, the attention mechanism 220 generates a corresponding query Q for each identified character of the query text embedding 212 (e.g., three queries Q, Q...). N Q N Q Y The attention mechanism 220 operates to identify the value of the corresponding character (e.g., one or more values ​​represented by a character embedding vector). In other words, the attention mechanism 220 functions to map the query Q to a set of key-value (K,V) pairs in order to generate elimination probabilities 222. Here, key-value pairs (e.g., such as...) Figure 2B (As shown) corresponds to the key of a character within text 152, and the value of that key is a vector representation of the character embedding 214 of that particular character in text 152. When query Q identifies the character embedding 214 corresponding to a character in text 152 from a specific frame F, attention mechanism 220 assigns weights 224 to one or more values ​​of the character embedding 214. By assigning these attention weights 224 based on each frame F of the overlapping region 204, attention mechanism 220 forms a context vector defining the elimination probability 222 of the characters in text 152. Here, to implement this functionality, attention mechanism 220 can implement scaled dot product attention or multi-head attention. For example, in some examples, the attention function of attention mechanism 220, which assigns weights 224 to represent the elimination probability 222, is represented by the following equation:

[0042]

[0043] Where d k Corresponding to the dimension of the key in the key-value pair.

[0044] Network 230 operates in conjunction with attention mechanism 220 to generate enhanced audio 206. In some implementations, network 230 receives audio signal 202, text 152, and / or cancellation probability 222 as input from attention mechanism 220. Using cancellation probability 222, network 230 is able to identify where echo 156 from synthesized playback audio 154 should be cancelled or removed in audio signal 202, in order to modify audio signal 202 to form enhanced audio signal 206. In some examples, network 230 is a Long Short-Term Memory (LSTM) network with multiple LSTM layers. For example, attention mechanism 220 may reside on one or more of these LSTM layers.

[0045] Figure 2E This is an example of a TEC system 200 without the attention mechanism 220. In other words, the TEC system 200 includes a text encoder 210 and a network 230. Here, the text encoder 210 generates a text embedding 212, which is a single fixed-dimensional text embedding vector representing the entire text 152 corresponding to the synthesized playback audio 154. Here, the TEC system 200 repeatedly concatenates the fixed-dimensional text embedding 212 with the audio signal 202 for each frame F of the audio signal 202 at the network 230. By repeatedly concatenating the fixed-dimensional text embedding 212, the TEC system 200 acts like a filtering system by identifying where to filter out the fixed-dimensional text embedding 212 when one or more frames F of the audio signal 202 are similar to the fixed-dimensional text embedding 212. Although this process produces an enhanced audio signal 206, it lacks the alignment identified by the attention mechanism 220 and requires continuous processing resources to perform the concatenation at the network 230.

[0046] refer to Figure 3In some configurations, prior to the implementation of the TEC system 200 (i.e., prior to inference), a training process 300 trains the TEC system 200 (e.g., attention mechanism 220 and / or network 230). The training process 300 trains the TEC system 200 using multiple training examples 302, 302a-n. Each example of training example 302 includes a ground truth audio signal 310 corresponding to non-synthetic speech (e.g., a real speech sample spoken by a human user), a training overlapped audio signal 320, and training text 330. Here, the training overlapped audio signal 320 includes the ground truth audio signal 310 overlapping with the synthesized playback audio 154 (e.g., at least in part), and the training text 330 corresponds to the text representation of the synthesized playback audio 154 overlapping with the ground truth audio signal 310. By using multiple training examples 302, the training process 300 teaches the TEC system 200 (e.g., attention mechanism 220 and / or network 230) how to generate the enhanced audio signal 206 output by the TEC system 200 during inference. In other words, the training process 300 teaches the TEC system 200 to generate a real-value audio signal 310 as an enhanced audio signal 206 based on the training overlap audio signal 320 and the training text 330.

[0047] Figure 4 This is a flowchart illustrating an example arrangement of operations for a method 400 used to generate an enhanced audio signal 206 to eliminate echo 156. At operation 402, method 400 receives an overlapped audio signal 202 comprising audio (e.g., utterance 12) spoken by speaker 10 that overlaps with a segment of the synthesized playback audio 154. At operation 404, method 400 encodes a character sequence corresponding to the synthesized playback audio 154 into a text embedding representation 212 (interchangeably referred to as 'text embedding' 212). For each character in the character sequence, at operation 406, method 400 uses the text embedding representation 212 to generate a corresponding elimination probability 222, which indicates the likelihood of the corresponding character being associated with the audio spoken by speaker 10 (e.g., utterance 12) and the segment 156 of the synthesized playback audio 154 overlaid by the overlapped audio signal 202 (i.e., the segment 156 representing the echo). At operation 408, method 400 uses a cancellation network 230 configured to receive the overlapping audio signal 202 and a corresponding cancellation probability 222 generated for each character in the character sequence as input to generate an enhanced audio signal 206. The cancellation network 230 generates the enhanced audio signal 206 by removing segments of the synthesized playback audio 154 from the overlapping audio signal 202.

[0048] Figure 5This is a schematic diagram of an example computing device 500 that can be used to implement the systems (e.g., systems 130, 140, 150, 160, 200) and methods (e.g., method 400) described herein. The computing device 500 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The various components shown herein, their connections and relationships, and their functions are merely exemplary and not intended to limit the embodiments of the invention described and / or claimed in this document.

[0049] Computing device 500 includes a processor 510 (e.g., data processing hardware 510), a memory 520 (e.g., memory hardware 520), a storage device 530, a high-speed interface / controller 540 connected to the memory 520 and a high-speed expansion port 550, and a low-speed interface / controller 560 connected to a low-speed bus 570 and the storage device 530. Each of components 510, 520, 530, 540, 550, and 560 is interconnected using various buses and may be suitably mounted on a common motherboard or otherwise mounted. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or on storage device 530, to display graphical information of a graphical user interface (GUI) on an external input / output device such as a display 580 coupled to high-speed interface 540. In other embodiments, multiple processors and / or multiple buses, as well as multiple memories and multiple memory types, may be suitably used. In addition, multiple computing devices 500 can be connected, each providing a necessary part of the operation (e.g., as a server library, blade server group, or multiprocessor system).

[0050] Memory 520 stores information non-transiently within computing device 500. Memory 520 may be a computer-readable medium, a volatile memory cell, or a non-volatile memory cell. Non-transient memory 520 may be a physical device for storing programs (e.g., instruction sequences) or data (e.g., program state information) used by computing device 500, on a temporary or permanent basis. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used in firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase-change memory (PCM), and magnetic disks or magnetic tapes.

[0051] Storage device 530 provides mass storage for computing device 500. In some embodiments, storage device 530 is a computer-readable medium. In various embodiments, storage device 530 may be a floppy disk device, hard disk device, optical disk device, magnetic tape device, flash memory, or other similar solid-state storage device, or a device array, including devices in a storage area network or other configuration. In additional embodiments, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer or machine-readable medium, such as memory 520, storage device 530, or memory on processor 510.

[0052] High-speed controller 540 manages bandwidth-intensive operations of computing device 500, while low-speed controller 560 manages less bandwidth-intensive operations. This functional allocation is merely exemplary. In some embodiments, high-speed controller 540 is coupled to memory 520, display 580 (e.g., via a graphics processor or accelerator), and high-speed expansion port 550, which can accept various expansion cards (not shown). In some embodiments, low-speed controller 560 is coupled to storage device 530 and low-speed expansion port 590. Low-speed expansion port 590, which includes various communication ports (e.g., USB, Bluetooth, Ethernet, Wireless Ethernet), can be coupled to one or more input / output devices, such as keyboards, pointing devices, scanners, or network devices, such as switches or routers, for example, via network adapters.

[0053] As shown in the figure, the computing device 500 can be implemented in several different forms. For example, it can be implemented as a standard server 500a or multiple times as a laptop computer 500b in a set of such servers 500a, or as part of a rack server system 500c.

[0054] Various implementations of the systems and techniques described herein can be implemented in digital electronic and / or optical circuit systems, integrated circuit systems, specially designed ASICs (Application-Specific Integrated Circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementations in one or more computer programs executable and / or interpretable on a programmable system, said programmable system including at least one programmable processor, which may be dedicated or general-purpose, coupled to receive and transmit data and instructions from a storage system, at least one input device, and at least one output device.

[0055] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented in high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0056] The processes and logical flows described in this specification can be implemented by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The processes and logical flows can also be implemented by special-purpose logic circuit systems, such as FPGAs (Field-Programmable Gate Arrays) or ASICs (Application-Specific Integrated Circuits). For example, processors suitable for executing computer programs include both general-purpose and special-purpose microprocessors, as well as any one or more processors of any type of digital computer. Typically, the processor receives instructions and data from read-only memory or random access memory, or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include, or be operatively coupled to, receive data from or transfer data to, or both of, one or more mass storage devices—such as magnetic disks, magneto-optical disks, or optical disks—for storing data. However, a computer does not need to have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD-ROMs and DVD-ROMs. Processors and memory may be supplemented by or integrated into dedicated logic circuitry systems.

[0057] To provide interaction with the user, one or more aspects of this disclosure can be implemented on a computer with a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and optionally to a keyboard and a pointing device such as a mouse or trackball through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback, such as visual, auditory, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from the device used by the user; for example, by sending a webpage to a web browser on the user's client device in response to a request received from a web browser.

[0058] Several embodiments have been described. However, it will be understood that various modifications can be made without departing from the spirit and scope of this disclosure. Therefore, other embodiments are within the scope of the following claims.

Claims

1. A computer-implemented method for text echo cancellation, the method, when executed on data processing hardware, causing the data processing hardware to perform operations, the operations including: Receive overlapping audio signals, the overlapping audio signals including audio spoken by a speaker that overlaps with segments of synthesized playback audio; The character sequence corresponding to the synthesized playback audio is encoded into a text embedding representation; For each character in the character sequence, the text embedding representation is used to generate a corresponding elimination probability, the corresponding elimination probability indicating the likelihood of the corresponding character being associated with the segment of the synthetic playback audio that overlaps with the audio spoken by the speaker in the overlapping audio signal; as well as An enhanced audio signal is generated by using an elimination neural network to remove segments of the synthesized playback audio from the overlapping audio signal. The elimination neural network is configured to receive the overlapping audio signal and the corresponding elimination probability generated for each character in the character sequence as input.

2. The computer-implemented method according to claim 1, wherein, The text-to-speech (TTS) system converts the character sequence into synthesized speech that includes the synthesized playback audio.

3. The computer-implemented method according to claim 1, wherein, The text embedding representation includes a single, fixed-dimensional text embedding vector.

4. The computer-implemented method according to claim 1, wherein, Encoding the character sequence includes: encoding each character in the character sequence into a corresponding character embedding to generate a sequence of character embeddings.

5. The computer-implemented method according to claim 4, wherein: The overlapping audio signal includes a sequence of frames, each frame in the sequence corresponding to a portion of the audio spoken by the speaker that overlaps with the segment of the synthesized playback audio; as well as Generating the corresponding elimination probability for each character in the character sequence includes: when the corresponding character embedding corresponds to a frame in the frame sequence of the overlapping audio signal, applying weights to the corresponding character embedding using an attention mechanism.

6. The computer-implemented method according to claim 1, wherein, The operation further includes: training the elimination neural network on multiple training examples, each training example including: The real audio signal corresponding to non-synthetic speech; This includes a training overlapping audio signal that overlaps with the synthesized audio signal, including the real audio signal; and The corresponding text representation of the synthesized audio signal, wherein the text representation includes a sequence of characters.

7. The computer-implemented method according to claim 1, wherein, The text encoder of the text encoding neural network encodes the character sequence corresponding to the synthesized playback audio into the text embedding representation.

8. The computer-implemented method according to claim 7, wherein, The text encoder is shared by a text-to-speech (TTS) system configured to generate the synthesized playback audio from the character sequence.

9. The computer-implemented method according to claim 1, wherein, The elimination neural network includes a Long Short-Term Memory (LSTM) network, which has multiple LSTM layers.

10. The computer-implemented method according to any one of claims 1 to 9, wherein, The operation further includes receiving an indication that the text representation of the synthesized playback audio is available.

11. A system for text echo cancellation, comprising: Data processing hardware; as well as Memory hardware communicating with the data processing hardware, the memory hardware storing instructions, which, when executed on the data processing hardware, cause the data processing hardware to perform operations, the operations including: Receive overlapping audio signals, the overlapping audio signals including audio spoken by a speaker that overlaps with segments of synthesized playback audio; The character sequence corresponding to the synthesized playback audio is encoded into a text embedding representation; For each character in the character sequence, the text embedding representation is used to generate a corresponding cancellation probability, the corresponding cancellation probability indicating the likelihood of the corresponding character being associated with the segment of the synthesized playback audio overlaid by the audio spoken by the speaker in the overlapping audio signal; and An enhanced audio signal is generated by using an elimination neural network to remove segments of the synthesized playback audio from the overlapping audio signal. The elimination neural network is configured to receive the overlapping audio signal and the corresponding elimination probability generated for each character in the character sequence as input.

12. The system according to claim 11, wherein, The text-to-speech (TTS) system converts the character sequence into synthesized speech that includes the synthesized playback audio.

13. The system according to claim 11, wherein, The text embedding representation includes a single, fixed-dimensional text embedding vector.

14. The system according to claim 11, wherein, Encoding the character sequence includes: encoding each character in the character sequence into a corresponding character embedding to generate a sequence of character embeddings.

15. The system according to claim 14, wherein: The overlapping audio signal includes a sequence of frames, each frame in the sequence corresponding to a portion of the audio spoken by the speaker that overlaps with the segment of the synthesized playback audio; as well as Generating the corresponding elimination probability for each character in the character sequence includes: when the corresponding character embedding corresponds to a frame in the frame sequence of the overlapping audio signal, applying weights to the corresponding character embedding using an attention mechanism.

16. The system according to claim 11, wherein, The operation further includes: training the elimination neural network on multiple training examples, each training example including: The real audio signal corresponding to non-synthetic speech; This includes a training overlapping audio signal that overlaps with the synthesized audio signal, including the real audio signal; and The corresponding text representation of the synthesized audio signal, wherein the text representation includes a sequence of characters.

17. The system according to claim 11, wherein, The text encoder of the text encoding neural network encodes the character sequence corresponding to the synthesized playback audio into the text embedding representation.

18. The system according to claim 17, wherein, The text encoder is shared by a text-to-speech (TTS) system configured to generate the synthesized playback audio from the character sequence.

19. The system according to claim 11, wherein, The elimination neural network includes a Long Short-Term Memory (LSTM) network, which has multiple LSTM layers.

20. The system according to any one of claims 11 to 19, wherein, The operation further includes receiving an indication that the text representation of the synthesized playback audio is available.

Images