STFT-based Echo Muter

Abstract

A method (300) for STFT (Short-Time Fourier Transform)-based echo muting includes receiving a microphone signal (202) that is captured by a microphone and includes an acoustic echo (156) corresponding to audio content (154) from an acoustic speaker (118), and receiving a reference signal (158) that includes a frame sequence representing the audio content. For each frame, the method includes processing to generate an output signal frame (206) that removes acoustic echo from each frame using an acoustic echo remover (210) configured to receive each frame as input; and using a Double-talk Detector (220) to determine whether each frame contains a double-talk frame or an echo-only frame based on each frame and the output signal frame. The method further includes muting the output signal frame for each frame containing an echo-only frame, and performing speech processing on the output signal frame for each frame containing a double-talk frame.

Description

Technology Field

[0001] This specification relates to a Short-Time Fourier Transform (STFT)-based echo muter. Background Technology

[0002] A voice assistant device can generate synthesized playback audio and deliver it to one or more users within a voice environment. While the voice assistant device outputs the synthesized playback audio, its microphone may capture the synthesized playback audio as an acoustic echo while actively capturing the voice of the user speaking toward the device. Unfortunately, due to the acoustic echo occurring in the synthesized playback audio, it may be difficult for a speech recognizer to recognize the voice of the user occurring within the echo of the synthesized playback audio.

[0003] One aspect of the present disclosure provides a computer-implemented method for performing speech recognition using an STFT-based echo muter. When executed on data processing hardware, the computer-implemented method causes the data processing hardware to perform a set of operations including receiving a microphone signal containing an acoustic echo captured by a microphone. The acoustic echo corresponds to audio content being played on an acoustic speaker. The set of operations also includes receiving a reference signal containing a frame sequence representing audio content transmitted on a reference channel before the acoustic speaker plays the audio content. The set of operations further includes, for each frame of the frame sequence of the microphone signal: processing each frame of the microphone signal to generate each output signal frame that removes the acoustic echo from each frame of the microphone signal using an acoustic echo muter configured to receive each frame of the frame sequence of the reference signal as input. The set of operations further includes, using a Double-talk Detector (DTD), determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame based on each frame of the reference signal and each output signal frame. The operation set further includes an operation of muting each output signal frame for each frame of a frame sequence of microphone signals including echo-only frames. The operation set further includes an operation of performing speech processing on each output signal frame for each frame in a frame sequence of microphone signals including double-talk frames after muting each output signal frame for each frame in a frame sequence of microphone signals including echo-only frames.

[0004] Implementations of the present disclosure may include one or more of the following optional functions. In some implementations, a portion of the microphone signal further includes an audio signal representing a target voice captured by the microphone, and the target voice is spoken while the audio content is played on an acoustic speaker; and the set of operations further includes an operation to determine whether each frame of the microphone signal contains a double-talk frame or an echo-only frame when each frame of the microphone signal contains an audio signal representing the target voice. In some examples, performing speech processing includes performing speech recognition using an Automatic Speech Recognition (ASR) model. In some implementations, the set of operations further includes an operation to transform each frame of the microphone, reference, and output signals into a Short-Time Fourier Transform (STFT) domain before using a DTD to determine whether each frame of the microphone signal contains a double-talk frame or an echo-only frame.

[0005] In some examples, the operation of determining whether each frame of a microphone signal contains a double-talk frame or an echo-only frame includes: the operation of computing each first frame-level double-talk indicator based on the cross-correlation between each frame of the microphone signal and each frame of a reference signal using a DTD; the operation of computing each second frame-level double-talk indicator based on the cross-correlation between each frame of the microphone signal and each output signal frame using a DTD; the operation of determining whether at least one of each first or second frame-level double-talk indicator satisfies a double-talk threshold; and the operation of determining that each frame of the microphone signal contains said double-talk frame when at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold. In these examples, the operation of determining whether each frame of a microphone signal contains a double-talk frame or an echo-only frame further includes the operation of determining that each frame of the microphone signal contains the echo-only frame when both each first and second frame-level double-talk indicator does not satisfy the double-talk threshold. Each first frame-level double-talk indicator and each second frame-level double-talk indicator (both may be computed over a frequency subband of a predetermined range). Additionally or alternatively, the operation of determining whether at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold includes the operation of determining that at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold when the minimum value of each first and second frame-level double-talk indicator is less than the double-talk threshold.

[0006] In some implementations, the set of operations further includes, for each frame of a frame sequence of microphone signals: using a DTD, computing each first frame-level double-talk indicator based on the cross-correlation between each frame of a reference signal or one of each output signal frame and each frame of the microphone signal, and determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame is based on each first frame-level double-talk indicator. In such implementations, the set of operations further includes, for each frame of a frame sequence of microphone signals: using a DTD, computing each second frame-level double-talk indicator based on the cross-correlation between each frame of a reference signal or the other of each output signal frame and each frame of the microphone signal, and determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame is based further on each second frame-level double-talk indicator.

[0007] In some examples, the acoustic echo canceller includes a linear acoustic echo canceller. In some implementations, the data processing hardware, microphone, and acoustic speaker reside in the user computing device. In some examples, the operation of performing speech processing for each output signal frame for each frame in a sequence of microphone signals including double-talk frames includes the operation of performing speech processing for each output signal frame without performing acoustic echo suppression for each output signal frame.

[0008] Another aspect of the present disclosure provides a system for performing speech recognition using an STFT-based echo muter. The system includes data processing hardware and memory hardware that communicates with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform a set of operations including receiving a microphone signal containing an acoustic echo captured by a microphone. The acoustic echo corresponds to audio content played on an acoustic speaker. The set of operations further includes receiving a reference signal containing a frame sequence representing audio content transmitted on a reference channel before the acoustic speaker plays the audio content. The set of operations includes, for each frame in the frame sequence of the microphone signal: an operation to process each frame of the microphone signal to generate each output signal frame that removes the acoustic echo from each frame of the microphone signal using an acoustic echo muter configured to receive each frame of the frame sequence of the reference signal as input; The operation includes determining whether each frame of a microphone signal contains a double-talk frame or an echo-only frame based on each frame of a reference signal and each output signal frame using a DTD (Double-talk Detector); and muting each output signal frame for each frame of a frame sequence of a microphone signal containing the echo-only frame. The set of operations further includes, after muting each output signal frame for each frame in a frame sequence of a microphone signal containing the echo-only frame, performing voice processing on each output signal frame (206) for each frame in a frame sequence of a microphone signal (202) containing the double-talk frame.

[0009] This embodiment may include one or more of the following optional functions. In some implementations, a portion of the microphone signal further includes an audio signal representing a target voice captured by the microphone, and the target voice is spoken while the audio content is played on an acoustic speaker; and the set of actions further includes an action for determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame when each frame of the microphone signal contains an audio signal representing the target voice. In some examples, performing speech processing includes performing speech recognition using an Automatic Speech Recognition (ASR) model. In some implementations, the set of actions further includes an action for converting each frame of the microphone, reference, and output signals into a Short-Time Fourier Transform (STFT) domain before using a DTD to determine whether each frame of the microphone signal contains a double-talk frame or an echo-only frame.

[0010] In some examples, the operation of determining whether each frame of a microphone signal contains a double-talk frame or an echo-only frame includes: the operation of computing each first frame-level double-talk indicator based on the cross-correlation between each frame of the microphone signal and each frame of a reference signal using a DTD; the operation of computing each second frame-level double-talk indicator based on the cross-correlation between each frame of the microphone signal and each output signal frame using a DTD; the operation of determining whether at least one of each first or second frame-level double-talk indicator satisfies a double-talk threshold; and the operation of determining that each frame of the microphone signal contains said double-talk frame when at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold. In these examples, the operation of determining whether each frame of a microphone signal contains a double-talk frame or an echo-only frame further includes the operation of determining that each frame of the microphone signal contains the echo-only frame when both each first and second frame-level double-talk indicator does not satisfy the double-talk threshold. Each first frame-level double-talk indicator and each second frame-level double-talk indicator may both be computed over a frequency subband of a predetermined range. Additionally or alternatively, the operation of determining whether at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold includes the operation of determining that at least one of each first or second frame-level double-talk indicator satisfies the double-talk threshold when the minimum value of each first and second frame-level double-talk indicator is less than the double-talk threshold.

[0011] In some implementations, the set of operations further includes, for each frame of a frame sequence of microphone signals: using a DTD, computing each first frame-level double-talk indicator based on the cross-correlation between each frame of a reference signal or one of each output signal frame and each frame of the microphone signal, and determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame is based on each first frame-level double-talk indicator. In such implementations, the set of operations further includes, for each frame of a frame sequence of microphone signals: using a DTD, computing each second frame-level double-talk indicator based on the cross-correlation between each frame of a reference signal or the other of each output signal frame and each frame of the microphone signal, and determining whether each frame of the microphone signal contains a double-talk frame or an echo-only frame is based further on each second frame-level double-talk indicator (208b).

[0012] In some examples, the acoustic echo canceller includes a linear acoustic echo canceller. In some implementations, the data processing hardware, microphone, and acoustic speaker reside in the user computing device. In some examples, the operation of performing speech processing for each output signal frame for each frame in a sequence of microphone signals including double-talk frames includes the operation of performing speech processing for each output signal frame without performing acoustic echo suppression for each output signal frame.

[0013] Details of one or more embodiments of the present disclosure are described in the accompanying drawings and the description below. Other embodiments, features, and advantages will become apparent from the description, drawings, and claims. Brief explanation of the drawing

[0014] Figure 1 is a schematic diagram of an exemplary voice environment using an Acoustic Echo Cancellation (AEC) system. Figure 2 is a schematic diagram of an AEC system. Figure 3 is a flowchart of an exemplary operation array for implementing an acoustic echo removal system. FIG. 4 is a schematic diagram of an exemplary computing device that can be used to implement the system and method described in this specification. Similar reference symbols in various drawings represent similar elements. Specific details for implementing the invention

[0015] A voice support device can generate synthesized playback audio and deliver the synthesized playback audio to one or more users within a voice environment. Here, synthesized playback audio refers to audio generated by the voice support device, originating from the voice support device itself or a machine processing system connected to the voice support device, rather than from a person or other source of audible sound outside the voice support device. Generally, the voice support device generates synthesized playback audio using a text-to-speech (TTS) system. The TTS system converts text into an audio representation of the text. Here, the audio representation of the text is modeled to resemble a speech representation using human language.

[0016] While the audio output component of a voice assistant device (e.g., an acoustic speaker) outputs synthesized playback audio, the audio capture component of the voice assistant device (e.g., a microphone) can still actively capture (i.e., listen to) audio signals within the voice environment. This means that a portion of the synthesized playback audio output from the speaker is received as an acoustic echo by the audio capture component. The voice assistant device may also play other types of audio content, such as media content (e.g., music), which can be captured as an acoustic echo by the audio capture component while actively capturing audio signals within the voice environment. Unfortunately, due to the acoustic echo occurring in the playback audio content (e.g., synthetic audio content), it can be difficult for a speech recognizer implemented in the voice assistant device or in a remote system communicating with the voice assistant device to understand spoken utterances that occur during the echo of the synthesized playback audio. In other words, voice assistant devices often generate synthesized playback audio in response to queries or commands from the voice assistant device user. For example, a user might ask the voice assistant device, "What will the weather be like today?" When a voice assistant receives such a query or question from a user, the voice assistant or a remote system communicating with the voice assistant must first determine or process the user's voice utterance. By processing the voice utterance, the voice assistant can recognize that the voice utterance corresponds to a query from the user (e.g., a query about the weather) and that the user expects a response from the voice assistant as a query.

[0017] Generally, a voice assistant device uses a speech recognition system (e.g., an automatic speech recognition (ASR) system) to determine the content of speech utterances. The speech recognition system receives audio signals or audio data and generates a text transcript representing the characters, words, and / or sentences spoken in the audio signal. However, speech recognition can become more complex when the voice capture component of the voice assistant device receives echoes and / or distortions simultaneously with all or part of one or more utterances spoken by user(s) to the voice assistant device. For example, one or more microphones of the voice assistant device are fed with echoes or acoustic feedback from a portion of the synthesized playback audio signal. Due to the echoes of the synthesized playback audio combined with one or more utterances, the voice assistant device receives an audio signal containing overlapping speech. Here, overlapping speech refers to a double-talk event, which is an instance of an audio signal where acoustic echoes from the synthesized playback audio occur simultaneously (i.e., simultaneous or concurrently) with one or more utterances. During double-talk events, the speech recognition system may struggle to process audio signals received from the voice assistant. In other words, overlapping speech can impair the speech recognition system's ability to generate accurate transcripts for one or more speech utterances. Without accurate transcripts from the speech recognition system, the voice assistant may not respond accurately or at all to queries or commands regarding the user's speech utterances. Alternatively, the voice assistant may wish to avoid using processing resources to attempt to interpret synthesized playback audio signals and / or audible sounds, such as actual echoes from the surroundings.

[0018] One approach to preventing distortion or echo captured by the audio capture component of a voice assistant is to use an Acoustic Echo Cancellation (AEC) system. In an AEC system, the audio signal is used to remove echoes associated with the audio content being played by the acoustic speaker. However, the audio signal used to remove echoes in an AEC system inevitably generates residual echo, which can further degrade the performance of the speech recognition system. In wake word applications, where a predetermined word or phrase is spoken to trigger the voice assistant's speech recognition, the residual echo generated by the AEC system improves the system's ability to find the wake word. However, in applications without wake words, this residual echo negatively impacts the performance of the speech recognition system. One way to reduce this residual echo is to process the audio signal used by the AEC system using a post-filter (e.g., an echo suppressor). However, since speech recognition systems are generally sensitive to post-filter processing of the audio signal, using a post-filter becomes a suboptimal solution.

[0019] As illustrated in FIG. 1, in some implementations, a voice environment (100) includes a user (10) who transmits a voice utterance (12) to a voice support device (110) (also referred to as the device (110) or user device (110). The user (10) (i.e., the speaker of the utterance (12)) may speak the utterance (12) as a query or command to obtain a response from the device (110). The device (110) is configured to capture sound from one or more users (10) within the voice environment (100). Here, the audio sound may refer to the utterance (12) of the user (10) serving as an auditory query, a command to the device (110), voice communication captured by the device (110), etc. The voice support system of the device (110) or a system associated with the device (110) may perform a query for a command by responding to the query and / or causing the command to be executed.

[0020] Here, the device (110) receives a microphone signal (202) comprising an acoustic echo (156) captured by an audio capture device (116) (also called a microphone) and / or a speech (12) by the user (10). The acoustic echo (156) corresponds to audio content (154) played from an audio output device (118) (also called an acoustic speaker). The device (110) may correspond to any computing device associated with the user (10) and capable of receiving the microphone signal (202). Some examples of the user device (110) include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smartwatches), smart home appliances, Internet of Things (IoT) devices, smart speakers, etc. The device (110) comprises data processing hardware (112); It also includes memory hardware (114) that communicates with data processing hardware (112) and stores instructions that cause data processing hardware (112) to perform one or more operations when executed by data processing hardware (112). In some examples, the device (110) includes one or more applications (i.e., software applications), and each application may utilize one or more voice processing systems (140, 150, 200) associated with the device (110) to perform various functions within the application. For example, the device (110) includes an assistant application configured to deliver synthesized playback audio content (154) to the user (10) to assist the user (10) with various tasks. In other examples, the assistant application or media application is configured to play audio content (154) containing media content (e.g., music, radio talk show, podcast content, television / movie content).In an additional example, the application corresponds to an assistant application configured to deliver audio content (154) as synthetic voice for playback from an acoustic speaker (118) to communicate / converse with or support the user (10) regarding the performance of various tasks. For example, the assistant application may audibly output synthetic voice in response to a query / command submitted by the user (10) to the assistant application. In an additional example, the audio content (154) played from the acoustic speaker (118) corresponds without limitation to notifications / warnings such as timer end, incoming call notification, doorbell chime, audio message, etc.

[0021] The device (110) includes a microphone (116) for capturing audio data (14) within a voice environment (100) and converting it into an electric microphone signal (202), and an acoustic speaker (118) for communicating / outputting playback audio content (154) (e.g., synthesized playback audio content). In the illustrated example, the device (110) implements a microphone (116), but the device (110) may implement an array of microphones (116) without departing from the scope of the present disclosure, so that one or more microphones (116) of the array may not physically reside in the device (110) but may communicate with an interface / peripheral device of the device (110). For example, the device (110) may correspond to a vehicle infotainment system utilizing a microphone array located throughout the vehicle. Similarly, the acoustic speaker (118) may include one or more speakers that reside in and communicate with the device (110), or may include a combination in which one or more speakers reside in the device (110) and one or more other speakers are physically removed from the device (110) but communicate with the device (110).

[0022] Additionally, the device (110) may be configured to communicate with a remote system (130) via a network (120). The remote system (130) may include remote resources (resources) (132), such as remote data processing hardware (134) (e.g., a remote server or CPU) and / or remote memory hardware (136) (e.g., a remote database or other storage hardware). The device (110) may utilize the remote resources (132) to perform various functions related to speech processing and / or synthetic playback communication. For example, the device (110) may be configured to perform speech recognition using a speech recognition system (140) (e.g., using a speech recognition model (145)). Additionally, the device may be configured to convert text to speech using a TTS system (150) and to perform acoustic echo cancellation using an AEC system (200). These systems (140, 150, 200) may reside on the device (110) (referred to as on-device systems) or reside remotely (e.g., reside on a remote system (130)) and may communicate with the device (110). In some examples, some of these systems (140, 150, 200) reside locally or within the device, while others reside remotely. That is, any of these systems (140, 150, 200) may be local or remote in any combination. For example, if the size or processing requirements of the systems (140, 150, 200) are somewhat large, the systems (140, 150, 200) may reside on a remote system (130). However, if the device (110) can support the size or processing requirements of one or more systems (140, 150, 200), one or more systems (140, 150, 200) may reside in the device (110) using data processing hardware (112) and / or memory hardware (114).Optionally, one or more of the systems (140, 150, 200) may reside both locally / on the device and remotely. For example, one or more of the systems (140, 150, 200) may default to running on the remote system (130) when a connection to the network (120) between the device (110) and the remote system (130) is available. However, if the connection is lost or the network (120) is unavailable, the systems (140, 150, 200) instead run locally on the device (110).

[0023] A speech recognition system (140) receives an audio signal (202) as input and transcribes the audio signal as output to a transcription (142). Generally speaking, by converting the audio signal (202) into a transcription (142), the speech recognition system (140) enables the device (110) to recognize when an utterance (12) from a user (10) corresponds to a question, a command, or other form of audio communication. The transcription (142) refers to a sequence of text that the device (110) can use to generate a response to a query or command. For example, if the user (10) asks the device (110) "what will the weather be like today," the device (110) transmits an audio signal corresponding to the question "what will the weather be like today" to the speech recognition system (140). The voice recognition system (140) converts the audio signal into a transcription containing the text “What will the weather be like today?”. Then, the device (110) can use the text or parts of the text to determine a response to the query. For example, to determine the weather for the current day (i.e., today), the device (110) passes the text (e.g., “What will the weather be like today?”) or identifying parts of the text (e.g., “Weather” and “Today”) to a search engine. The search engine can then return one or more search results that the device (110) interprets to generate a response for the user (10).

[0024] In some embodiments, the device (110) or the system associated with the device (110) identifies text (152) that the device (110) will deliver to the user (10) in response to a query of voice utterance (12). Then, the device (110) may use a TTS system (150) to convert the text (152) into corresponding synthetic playback audio (154) for the device (110) to deliver to the user (10) (e.g., audibly to the user (10)) in response to a query of voice utterance (12). That is, the TTS system (150) receives the text (152) as input and converts the text (152) into an output of synthetic playback audio (154), wherein the synthetic playback audio (154) is an audio signal defining an audible representation of the text (152). In some examples, the TTS system (150) includes a text encoder that processes text (152) into an encoded format (e.g., text embedding). Here, the TTS system (150) may use a trained text-to-speech model to generate synthesized playback audio (154) from the encoded format of the text (152). Once generated, the TTS system (150) transmits the synthesized playback audio (154) to a device (110) so that the device (110) can output the synthesized playback audio (154). For example, the device (110) outputs the synthesized playback audio (154) "today is sunny" from the speaker (118) of the device (110).

[0025] As illustrated in FIG. 1, when the device (110) outputs synthesized playback audio (154) (e.g., synthesized voice), the synthesized playback audio (154) generates an echo (156) that is captured by the microphone (116). Unfortunately, in addition to the echo (156), the microphone (116) may also simultaneously capture another voice utterance (12) from the user (10) corresponding to the target voice directed at the device (110). For example, FIG. 1 illustrates the user (10) asking more about the weather via voice utterance (12) directed at the device (110) by saying "what about tomorrow?" when the device (110) outputs synthesized playback audio content (154). In particular, the user (10) speaks an utterance (12) as part of a continuous conversation scenario in which the device (110) keeps the microphone (116) open and activates the voice recognition system (140) so that the user (10) can provide a subsequent query for recognition by the voice recognition system (140) without requiring the user (10) to speak a hotword (e.g., a predetermined word or phrase that triggers the device (110) to call voice recognition when detected). If the utterance (12) spoken by the user (10) contains a hotword, the echo (156) helps the voice recognition system (140) convert the audio signal (14) into a transcription. On the other hand, if the device (110) does not require the user (10) to speak a hot word to perform voice recognition, the user device (110) can process a microphone signal (202) including an echo (156) to allow the user device (110) to process its own playback audio (154) output from the speaker (118).

[0026] Here, both the voice utterance (12) and the echo (156) are simultaneously captured by the microphone (116) to form a microphone signal (202). That is, the microphone signal (202) includes a portion containing only the echo (156) corresponding to the playback audio content (154) output from the speaker (118) before the user (10) speaks the utterance (12), an overlapping portion (e.g., an overlapping area (204)) in which the utterance (12) spoken by the user (10) overlaps with a portion of the playback audio content (154) output from the speaker (118), and a portion containing only the utterance (12) spoken by the user (12) after the acoustic speaker (118) stops outputting the playback audio content (154).

[0027] As illustrated in FIG. 1, the overlapping region (204) of the captured microphone signal (202) corresponds to a double-talk event representing an instance where a portion of the utterance (12) and a portion of the synthesized playback audio (154) overlap each other in the captured microphone signal (202). During a double-talk event, the speech recognition system (140) may have trouble recognizing a subsequent query (12) corresponding to a weather query, "what about tomorrow," in the audio signal (202) captured by the microphone (116) because the utterance (12) is mixed with the echo (156) of the synthesized playback audio content (154). As described above, one method to reduce the echo (156) of the microphone signal (202) is to process the microphone signal (202) with an echo suppressor filter. However, the speech recognition system (140) has difficulty processing this type of filtered microphone signal (202).

[0028] To solve this problem, the device (110) includes an AEC system (200) that processes a microphone signal (202) and provides the output to a speech recognition system (140). The AEC system (200) (Fig. 2) includes an acoustic echo canceller (210), a double-talk detector (220), and an echo muter (230), and is configured to mute audio output signal frames identified only by echo, while allowing audio frames containing double-talk to pass through for processing by the speech recognition system (140). That is, the AEC system (200) receives a microphone signal (202) containing a follow-on query (12) mixed with an echo (156). For each audio frame of the microphone signal, the AEC system (200) processes the audio frame and determines whether the audio frame is echo only or contains double-talk.

[0029] Referring to FIG. 2, an acoustic echo canceller (210) of an AEC system (200) is configured to receive a microphone signal (202) that may simultaneously include a reference signal (158) corresponding to each frame of target voice (12) directed toward the device and each frame representing playback audio (154) captured by the microphone (12). For each frame of the frame sequence, the acoustic echo canceller (210) generates an output signal frame (206) by processing the microphone signal (202) using the reference signal (158) to remove the echo (156) of the microphone signal (202). In some examples, the AEC system (200) converts each frame of the microphone signal (202), the reference signal (158), and the output signal (206) into the STFT (Short-Time Fourier Transform) domain using Equation 1, and

[0030]

[0031] Here, L represents a frame hop, and w[n] represents an analysis window with N possible points, and l represents the frame index, and k represents the subband index.

[0032] Afterwards, the double-talk detector (220) receives the microphone signal m[n](202)( M(l,k) Indicated by ), reference signal x[n](158)( X(l,k (indicated by ) and residual echo output signal r[n] (206)( R(l,k) Each frame in the STFT domain of (indicated by) is received to determine whether each frame contains a double-talk frame or an echo-only frame. To achieve this, the double-talk detector (220) uses two double-talk indicators (208) per subband in each frame. The two double-talk indicators (208a, 208b) can be expressed as in Equation 2, and

[0033]

[0034]

[0035] Here, c PQ (k) represents the complex cross-correlation for each subband k at a lag of zero in the STFT domain of the playback audio frames (154) P(l,k) and Q(l,k). Here, P(l,k) and Q(l,k) are associated with time-domain signals (p[n] and q[n]), respectively. Complex cross-correlation ( c PQ (k)) can be expressed as in mathematical equation 3, and

[0036]

[0037] Here, E{·} represents the mathematical expectation, and ·* represents the complex conjugate. In addition, the energy of subband k can be expressed as in Equation 4.

[0038]

[0039] The cross-correlation for each subband determines the energy of the subband using the exponentially weighted average over the STFT domain using Equation 5, and

[0040]

[0041]

[0042] Here, ρ represents the exponential weighting factor (i.e., the forgetting factor). For example, an exponential weighting factor ρ of 0.9 can balance estimation accuracy and response time. Using this, Equation 2 can be re-formulated and expressed as Equation 6.

[0043]

[0044]

[0045] In some examples, the first double-talk indicator (208a) is computed by the cross-correlation between the microphone signal (202) and the reference signal (158), while the second double-talk indicator (208b) is computed by the cross-correlation between the microphone signal (202) and the output signal (206).

[0046] The double-torque detector (220) uses indicators and subband energies to calculate double-torque indicators (208a, 208b) for each frame by computing (calculating) a weighted average of subband indicators for a limited number of subbands. For example, subbands between 700 Hz and 2400 Hz can be used to determine double-torque indicators (208a, 208b). Each double-torque indicator (208a, 208b) is weighted by the energy of the residual of a given subband by cross-correlating each frame of the frame sequence to determine the energy of the band. A higher residual energy of the subband indicates a higher probability of double-torque being present, and the double-torque indicator (208a, 208b) indicates that the frame contains double-torque. The first double-torque indicator (208a) can be calculated as in Equation 7.

[0047]

[0048] Likewise, the second double-torque indicator (208b) can be calculated as in Equation 8.

[0049]

[0050] Once the double-torque detector (220) computes the double-torque indicators (208a, 208b) as outputs, this is provided to the echo muter (230) as an input to determine whether at least one of the double-torque indicators (208a, 208b) satisfies the double-torque threshold. In some examples, the double-torque detector (220) computes only one of the first or second double-torque indicators (208a, 208b) as an input to the echo muter (230) to determine whether the double-torque threshold is satisfied. Given a threshold τ, the double-torque detector (220) determines whether each frame is echo-only, which must be muted, or contains double-torque that must be passed through. This is determined as in Equation 9.

[0051]

[0052] If at least one of the double-talk indicators (208a, 208b) satisfies the double-talk threshold, each frame is passed on for processing by the speech recognition system (140). If neither of the double-talk indicators (208a, 208b) satisfies the double-talk threshold, each frame is identified as echo-only and muted (i.e., not passed on for processing by the speech recognition system). Requires a minimum value of the double-talk indicators (208a, 208b) to satisfy the double-talk threshold, preferring passing through frames over muting frames.

[0053] The double-talk threshold τ can be calculated using a data-driven approach. Metadata is collected from utterances to split the utterances into echo-only intervals and double-talk intervals. Next, a frame-level double-talk indicator (208a, 208b) is calculated for all frames of the training dataset. For a given threshold τ, the misclassification rate of echo-only frames and double-talk frames is calculated. This can be repeated for a range of threshold τ with a target that identifies a threshold τ that results in a misclassification of double-talk frames of 10% or less.

[0054] FIG. 3 includes a flowchart of an exemplary array of operations for a method (300) for performing speech recognition using an STFT-based echo muter (230). In operation (302), the method (300) includes receiving a microphone signal (202) containing an acoustic echo (156) captured by a microphone (116). The acoustic echo (156) corresponds to audio content (154) played from an acoustic speaker (118). In operation (304), the method (300) includes the step of receiving a reference signal (158) containing a frame sequence representing audio content (154) transmitted from a reference channel before the acoustic speaker (118) plays the audio content (154).

[0055] For each frame of a frame sequence of a microphone signal (202), the method (300) includes the step of processing each frame of the microphone signal (202) to generate each output signal frame (206) that removes acoustic echo (156) from each frame of the microphone signal (202) by using an acoustic echo remover (210) configured to receive each frame of a frame sequence of a reference signal (158) as input in operation (306). In operation (308), the method (300) includes the step of using a double-talk detector (220) to determine whether each frame of the microphone signal (202) contains a double-talk frame or an echo-only frame based on each frame of the reference signal (158) and each output signal frame (206). For each frame of a frame sequence of a microphone signal (202) including an echo-only frame, the method (300) in operation (310) includes the step of muting each output signal frame (206). In operation (312), after muting each output signal frame (206) for each frame in a frame sequence of a microphone signal (202) including an echo-only frame, the method (300) includes the step of performing voice processing on each output signal frame (206) for each frame of a sequence of frames of a microphone signal (202) including a double-talk frame.

[0056] FIG. 4 is a schematic diagram of an exemplary computing device (400) that may be used to implement the system and method described in this document. The computing device (400) is intended to represent various forms of digital computers, such as laptops, desktops, workstations, PDAs, servers, blade servers, mainframes, and other suitable computers. The components, connections, relationships, and functions shown herein are for illustrative purposes only and do not limit the implementation of the invention described and / or claimed herein.

[0057] The computing device (400) includes a processor (410), memory (420), a storage device (430), a high-speed interface / controller (440) connected to the memory (420) and a high-speed expansion port (450), a low-speed bus (470), and a low-speed interface / controller (460) connected to the storage device (430). Each component (410, 420, 430, 440, 450 and 460) is interconnected using various buses and can be mounted on a common motherboard or in a suitable other way. A processor (410) (also referred to as "data processing hardware (410)" which may include data processing hardware (112) of a user computing device (110) or data processing hardware (134) of a remote system (130)) can process instructions for execution within a computing device (400), including instructions stored in memory (420) or a storage device (430), to display graphic information for a graphical user interface (GUI) on an external input / output device, such as a display (480) connected to a high-speed interface (440). In other implementations, multiple processors and / or multiple buses may be appropriately used with multiple memories and memory types. Additionally, multiple computing devices (400) may be connected, and each device provides a part of the necessary operation (e.g., a server bank, a blade server group, or a multiprocessor system).

[0058] Memory (420) stores information non-transiently within a computing device (400). Memory (420) may be a computer-readable medium, volatile memory unit(s), or non-volatile memory unit(s). Non-transient memory (420) may be a physical device used to store a program (e.g., instruction sequence) or data (e.g., program state information) temporarily or permanently for use by the computing device (400). Examples of non-volatile memory include, but are not limited to, flash memory and ROM (read-only memory) / PROM (programmable read-only memory) / EPROM (erasable programmable read-only memory) / EEPROM (electronically erasable programmable read-only memory) (e.g., generally used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, RAM (Random Access Memory), DRAM (Dynamic Random Access Memory), SRAM (Static Random Access Memory), PCM (Phase Change Memory), and disks or tapes.

[0059] The storage device (430) may provide a large-capacity storage device for the computing device (400). In some implementations, the storage device (430) is a computer-readable medium. In various different implementations, the storage device (430) may be a floppy disk device, a hard disk device, an optical disk device or a tape device, a flash memory or other similar solid-state memory device, or a device array including a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information medium. The computer program product includes instructions that perform one or more methods such as those described above when executed. The information medium is a computer or machine-readable medium such as memory (420), the storage device (430), or memory (410) on a processor.

[0060] The high-speed controller (440) manages bandwidth-intensive operations for the computing device (400), while the low-speed controller (460) manages low-bandwidth intensive operations. These assignments of duties are merely examples. In some implementations, the high-speed controller (440) is connected to a high-speed expansion port (450) capable of accommodating memory (420), a display (480) (e.g., via a graphics processor or accelerator), and various expansion cards (not shown). In some implementations, the low-speed controller (460) is connected to a storage device (430) and a low-speed expansion port (490). The low-speed expansion port (490), which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be connected to one or more input / output devices, such as a keyboard, pointing device, scanner, or networking device such as a switch or router, for example, via a network adapter.

[0061] The computing device (400) can be implemented in various forms as illustrated in the drawing. For example, it can be implemented as a standard server (400a) or, multiple times from a group of such servers (400a), as part of a laptop computer (400b) or a rack server system (400c).

[0062] Various implementations of the systems and technologies described herein may be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more executable and / or interpretable computer programs on a programmable system comprising a storage system and at least one programmable processor, which may be specialized or general-purpose, coupled to receive data and instructions from at least one input device and at least one output device and to transmit data and instructions.

[0063] A software application (i.e., a software resource) may refer to computer software that enables a computing device to perform operations. In some examples, a software application may be referred to as an "application," "app," or "program." Examples of applications include system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and game applications.

[0064] Non-transient memory may be a physical device used to store programs (e.g., sequences of instructions) or data (e.g., program state information) temporarily or permanently for use by a computing device. Non-transient memory may be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Phase Change Memory (PCM), and disks or tapes.

[0065] Such computer programs (also referred to as programs, software, software applications, or code) contain machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” mean any computer program product, non-transient computer-readable medium, device, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as machine-readable signals. The term “machine-readable signal” means any signal used to provide machine instructions and / or data to a programmable processor.

[0066] The processes and logic flows described in this specification may be performed by one or more programmable processors, also known as data processing hardware, and may perform functions by executing one or more computer programs to operate on input data and generate outputs. The processes and logic flows may also be performed by special-purpose logic circuits, for example, field programmable gate arrays (FPGAs) or application-specific integrated circuits (ASICs). Processors suitable for executing computer programs include, for example, both general-purpose and special-purpose microprocessors and one or more processors of all types of digital computers. Generally, processors receive instructions and data from read-only memory or random access memory, or both. Essential elements of a computer are a processor that executes instructions and one or more memory devices that store instructions and data. Generally, a computer will include or be operablely combined with one or more mass storage devices for storing data, for example, magnetic, magneto-optical disks, or optical disks, to receive data or transmit data, or both. However, a computer does not require such devices. Computer-readable media suitable for storing computer program instructions and data include, for example, semiconductor memory devices, for example, EPROM, EEPROM, and flash memory devices; Magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and all forms of non-volatile memory, media, and memory devices such as CD-ROM and DVD-ROM disks are included. Processors and memory may be complemented or integrated by special-purpose logic circuits.

[0067] To provide interaction with a user, one or more aspects of the present invention may be implemented in a computer having a display device for displaying information to a user, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, and optionally a keyboard and pointing device, such as a mouse or trackball, thereby allowing the user to provide input to the computer. Interaction with the user may also be provided using other types of devices. For example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback. Input from the user may be received in any form, including sound, voice, or tactile input. Additionally, the computer may interact with the user by exchanging documents with the device used by the user. For example, it sends a web page to a web browser on the user client device in response to a request received from a web browser.

[0068] Many implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the invention. Accordingly, other implementations are within the scope of the following claims.

Claims

Claim 1 A method (300) implemented in a computer that causes the data processing hardware (410) to perform a set of operations when executed in the data processing hardware (410), wherein the set of operations comprises: an operation of receiving a microphone signal (202) containing an acoustic echo (156) captured by a microphone (116) - wherein the acoustic echo (156) corresponds to audio content (154) played from an acoustic speaker (118); and an operation of receiving a reference signal (158) containing a frame sequence representing audio content (154) transmitted from a reference channel before the acoustic speaker (118) plays the audio content (154). For each frame (202) in the frame sequence of the microphone signal above: an operation of processing each frame of the microphone signal (202) to generate each output signal frame (206) of an output signal that removes acoustic echo (156) from each frame of the microphone signal (202) using an acoustic echo remover (210) configured to receive each frame of the frame sequence of the reference signal (158) as input; and an operation of determining at the frame-level whether each frame of the microphone signal (202) contains a double-talk frame or an echo-only frame, based on the STFT domain representation of each frame of the reference signal (158) and the STFT domain representation of each output signal frame (206), using a DTD (Double-talk Detector) (220) that processes a plurality of subbands in the STFT (Short-Time Fourier Transform) domain. And for each frame of the frame sequence of the microphone signal (202) including the echo-dedicated frame, the operation of muting each output signal frame (206);The operation includes, after muting each output signal frame (206) for each frame in the frame sequence of the microphone signal (202) including the echo-only frame, performing voice processing on each output signal frame (206) for each frame in the frame sequence of the microphone signal (202) including the double-talk frame, but not performing voice processing on each muted output signal frame; and the operation of determining whether each frame of the microphone signal (202) includes a double-talk frame or an echo-only frame comprises, using a DTD (220), computing each first frame-level double-talk indicator (208a) based on the cross-correlation between the subband of the STFT domain representation of each frame of the microphone signal (202) and the subband of the STFT domain representation of each frame of the reference signal (158); A computer-implemented method comprising: computing each second frame-level double-talk indicator (208b) based on the cross-correlation between the subband of the STFT domain representation of each frame of the microphone signal (202) and the subband of the STFT domain representation of each output signal frame (206) using a DTD (220); determining whether at least one of each first or second frame-level double-talk indicator (208a, 208b) satisfies a double-talk threshold; and determining that each frame of the microphone signal (202) contains said double-talk frame when at least one of each first or second frame-level double-talk indicator (208a, 208b) satisfies the double-talk threshold. Claim 2 A computer-implemented method according to claim 1, wherein a portion of the microphone signal (202) further comprises an audio signal representing a target voice (12) captured by a microphone (116), and the target voice (12) is uttered while audio content (154) is played on an acoustic speaker (118); and the set of operations further comprises an operation determining whether each frame of the microphone signal (202) includes the double-talk frame or the echo-only frame when each frame of the microphone signal (202) includes the audio signal representing the target voice (12). Claim 3 A computer-implemented method according to claim 1, wherein the operation of performing voice processing includes the operation of performing voice recognition using an automatic speech recognition (ASR) model (145). Claim 4 delete Claim 5 delete Claim 6 A computer-implemented method according to claim 1, wherein the operation of determining whether each frame (202) of a microphone signal includes a double-talk frame or an echo-only frame further includes the operation (202) of determining that each frame of a microphone signal includes the echo-only frame when both the first and second frame-level double-talk indicators (208a, 208b) do not satisfy a double-talk threshold. Claim 7 In claim 1, a computer-implemented method wherein each first frame-level double-torque indicator (208a) and each second frame-level double-torque indicator (208b) are both computed over a frequency subband of a predetermined range. Claim 8 A computer-implemented method according to claim 1, wherein the operation of determining whether at least one of each first or second frame-level double-torque indicator (208a, 208b) satisfies a double-torque threshold includes the operation of determining that at least one of each first or second frame-level double-torque indicator (208a, 208b) satisfies the double-torque threshold when the minimum value of each first and second frame-level double-torque indicator (208a, 208b) is smaller than the double-torque threshold. Claim 9 delete Claim 10 delete Claim 11 In claim 1, the method implemented by a computer includes an acoustic echo remover (210) comprising a linear acoustic echo remover. Claim 12 In claim 1, the data processing hardware (410), microphone (116), and acoustic speaker (118) are implemented as a computer residing in a user computing device (110). Claim 13 A computer-implemented method according to claim 1, wherein the operation of performing voice processing for each output signal frame (206) for each frame in a sequence of microphone signals (202) including double-talk frames includes the operation of performing voice processing for each output signal frame (206) without performing acoustic echo suppression for each output signal frame (206). Claim 14 A system comprising: data processing hardware (410); and memory hardware (114, 136) communicating with the data processing hardware (410), wherein the memory hardware (114, 136) stores instructions that cause the data processing hardware (410) to perform a method of any one of claims 1 through 3, 6 through 8, and 11 through 13 when executed on the data processing hardware (410). Claim 15 delete Claim 16 delete Claim 17 delete Claim 18 delete Claim 19 delete Claim 20 delete Claim 21 delete Claim 22 delete Claim 23 delete Claim 24 delete Claim 25 delete Claim 26 delete

Images