Systems and methods for echo reduction using speech decomposition

Abstract

A method includes performing a first decomposition operation on a transformed input speech signal at a first neural network to generate speech components of the transformed input speech signal. The transformed input speech signal includes a frequency domain transformed near-end speech component stacked with a frequency domain transformed far-end speech component. The method also includes performing a second decomposition operation on the transformed input speech signal at a second neural network to generate non-speech components of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. The method also includes merging the speech component with the non-speech component at a third neural network to generate a transformed output speech signal.

Description

[0001] I. Cross-Reference to Related Applications

[0002] This application claims the benefit of priority to commonly owned Greek Provisional Patent Application No. 20220100350 filed on April 27, 2022, the contents of which are expressly incorporated herein by reference in their entirety. II. Technical field

[0003] The present disclosure generally relates to echo cancellation.

[0004] III. Related Technology

[0005] Technological advances have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including wireless phones (such as mobile and smart phones, tablet computers, and laptop computers) that are small, lightweight, and easy for users to carry. These devices can communicate voice packets, data packets, or both through wired or wireless networks. In addition, many such devices incorporate additional functionality, such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications, such as web browser applications, which can be used to access the Internet. As such, these devices can include significant computing capabilities.

[0006] One common use of wireless devices is voice communication. As a non-limiting example, during a phone call, a first user of a wireless device may speak into a microphone of the wireless device to communicate with a second user. However, when the first user speaks into the microphone, in some scenarios, the user's voice may be affected by an echo. For example, when the voice from the second user is output to the first user via the speaker of the wireless device, the microphone may inadvertently capture the voice from the second user. Therefore, by capturing the voice from the second user, an unintentional echo may be generated.

[0007] Typically, a single architecture or module is used to process user speech for echo cancellation. As a non-limiting example, a single network can process speech having both a speech component and a non-speech component to eliminate echo characteristics and suppress noise. However, because the speech component and the non-speech component have completely different probability distributions, using a single network may be inefficient and may reduce the speech quality of the final output speech. For example, by applying the same weights and coefficients to process the speech component and the non-speech component in a single network, the speech quality of at least one component may be impaired. IV. Summary of the Invention

[0008] According to certain aspects, a device includes a first neural network configured to perform a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component. The device also includes a second neural network configured to perform a second decomposition operation on the transformed input speech signal to generate a non-speech component of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. The device also includes a third neural network configured to combine the speech component with the non-speech component to generate a transformed output speech signal.

[0009] According to another specific aspect, a method includes performing a first decomposition operation on a transformed input speech signal at a first neural network to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component. The method also includes performing a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. The method also includes combining the speech component with the non-speech component at a third neural network to generate a transformed output speech signal.

[0010] According to another specific aspect, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform a first decomposition operation on a transformed input speech signal at a first neural network to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component. The instructions also cause the one or more processors to perform a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. The instructions further cause the one or more processors to combine the speech component with the non-speech component at a third neural network to generate a transformed output speech signal.

[0011] According to another specific aspect, a device includes a component for performing a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain. The device also includes a component for performing a second decomposition operation on the transformed input speech signal to generate a non-speech component of the transformed input speech signal. The component for performing the first decomposition operation and the component for performing the second decomposition operation perform echo cancellation on the transformed input speech signal. The device also includes a component for merging the speech component with the non-speech component to generate a transformed output speech signal. V. Description of the Figures

[0012] Figure 1 is a diagram of a specific illustrative example of a system configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network.

[0013] Figure 2 is a diagram of a specific illustrative example of a system configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network.

[0014] Figure 3 is a diagram of a specific illustrative example of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a convolutional u-net architecture.

[0015] Figure 4 is a diagram of a specific illustrative example of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a recurrent u-net architecture.

[0016] Figure 5 is a diagram of a specific illustrative example of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a recurrent layer architecture.

[0017] Figure 6 is a block diagram illustrating a specific implementation of an integrated circuit configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network.

[0018] Figure 7 Depicted are implementations of a mobile device configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network.

[0019] Figure 8 Depicted are implementations of a portable electronic device configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network.

[0020] Fig. 9Described are implementations of a wearable electronic device configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network.

[0021] Fig.10 Detailed description of the invention The invention relates to a wireless speaker and a voice activated device configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network.

[0022] Fig.11 Depicted is a specific implementation of a head mounted device configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network.

[0023] Fig.12 Depicted are implementations in which a vehicle is configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network.

[0024] Fig.13 Another implementation of a vehicle configured to reduce echo associated with input speech using a voice neural network and a non-voice neural network is depicted.

[0025] Fig.14 is a flow chart of a particular example of a method for reducing echo associated with input speech using a voiced neural network and an unvoiced neural network.

[0026] Fig.15 is a diagram of a specific example of components of a device configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network.

[0027] Fig.16 is a block diagram of a particular illustrative example of a device operable to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network. VI. Specific Implementation Methods

[0028] An electronic device (e.g., a mobile device, a head-mounted device, etc.) may include at least one microphone configured to capture a first user's voice from a first user. Typically, the first user's mouth is close to (e.g., near) the microphone. However, in addition to capturing the first user's voice from the first user, the microphone may also capture noise in the surrounding environment of the first user. The first user's voice (and the surrounding environmental noise) captured by the microphone may be classified as "near-end voice". In some cases, the microphone may also capture a second user's voice output by a speaker associated with the electronic device. As a non-limiting example, if the first user and the second user are participating in a voice call, in addition to capturing the near-end voice, the microphone may also capture the second user's voice from the second user output by the speaker. The second user's voice (and any surrounding noise) output by the speaker may be classified as "far-end voice". The microphone may generate a near-end voice signal based on the captured near-end voice. However, as described above, because the microphone may inadvertently capture the far-end voice output by the speaker, the near-end voice signal generated by the microphone may include the captured far-end voice component.

[0029] The technology described herein utilizes a combination of trained neural networks to reduce (or eliminate) echoes associated with near-end speech signals, particularly echoes associated with unintentionally captured far-end speech components. For example, if far-end speech (e.g., speech from a second user) is captured and sent back to the second user, the second user may hear the echo. In order to reduce the echo, the near-end speech signal may be provided to an echo cancellation system, which includes a first transform unit, a second transform unit, a combination unit, a first neural network (e.g., a voice network), a second neural network (e.g., a non-voice network), and a third neural network (e.g., a merging network). The first transform unit may be configured to perform a transform operation on the near-end speech signal to generate a transformed near-end speech signal (e.g., a frequency domain version of the near-end speech signal). Therefore, the transformed near-end speech signal corresponds to a transformed version of the near-end speech, and may also include a residual transformed version of the far-end speech (based on the far-end speech unintentionally captured by the microphone).

[0030] In addition, a far-end audio signal indicating the far-end speech from the loudspeaker may be transformed by a second transformation unit to generate a transformed far-end speech signal. The transformed far-end speech signal and the transformed near-end speech signal are provided to a combination unit, and the combination unit may be configured to generate a transformed input speech signal based on the transformed far-end speech signal and the transformed near-end speech signal. For example, the transformed input speech signal may include a near-end speech component (based on the transformed near-end speech signal) transformed in the frequency domain stacked with a far-end speech component (based on the transformed far-end speech signal) transformed in the frequency domain.

[0031] The transformed input speech signal is provided to a first neural network and a second neural network. The first neural network may perform a first decomposition operation on the transformed input speech signal to generate a speech component. For example, the first neural network may apply a speech mask or identify transform coefficients (e.g., fast Fourier transform (FFT) coefficients) to separate and extract the speech component from the transformed input speech signal. In some implementations, after extracting the speech component, the first neural network may process the speech component to improve gain, reduce noise, reduce echo, etc. The second neural network may perform a second decomposition operation on the transformed input speech signal to generate a non-speech component. For example, the second neural network may apply a non-speech mask or identify transform coefficients (e.g., FFT coefficients) to separate and extract the non-speech component from the transformed microphone signal. In some implementations, after extracting the non-speech component, the second neural network may process the non-speech component to reduce gain, reduce noise, reduce echo, etc. In general, a large portion of the echo may contribute to the non-speech component. Therefore, in order to reduce the echo, the second neural network may significantly reduce the gain of the non-speech component to reduce the echo. The third neural network may combine the processed speech component and the processed non-speech component to generate a transformed output speech signal having a reduced amount of noise and echo (eg, an echo-canceled signal indicative of clean speech).

[0032] Thus, the techniques described herein improve the quality of speech decomposition and reconstruction by using multiple neural networks to process different components of a transformed input speech signal. For example, a speech component may be processed using a first neural network, and an unspeech component may be processed using a second neural network. Thus, a single neural network does not have to process different speech parts (e.g., speech parts and unspeech parts) with completely different probability distributions, which enables improved speech quality and weight efficiency.

[0033] Unless expressly limited by its context, the term "generating" is used herein to indicate any of its ordinary meanings (such as calculating or otherwise producing). Unless expressly limited by its context, the term "calculating" is used herein to indicate any of its ordinary meanings (such as computing, evaluating, smoothing, and / or selecting from a plurality of values). Unless expressly limited by its context, the term "obtaining" is used herein to indicate any of its ordinary meanings, such as calculating, deriving, receiving (e.g., from another component, block, or device), and / or retrieving (e.g., from a memory register or storage element array).

[0034] Unless expressly limited by its context, the term "generating" is used to indicate any of its ordinary meanings (such as calculating, generating, and / or providing). Unless expressly limited by its context, the term "providing" is used to indicate any of its ordinary meanings (such as calculating, generating, and / or producing). Unless expressly limited by its context, the term "coupling" is used to indicate a direct or indirect electrical or physical connection. If the connection is indirect, there may be other blocks or components between the "coupled" structures. For example, a speaker may be acoustically coupled to a nearby wall via an intervening medium (e.g., air) that enables waves (e.g., sound) to propagate from the speaker to the wall (or vice versa).

[0035] As indicated by its specific context, the term "configuration" may be used with reference to a method, an apparatus, a device, a system, or any combination thereof. When the term "comprising" is used in this specification and claims, it does not exclude other elements or operations. The term "based on" (as in "A is based on B") is used to indicate any of its ordinary meanings, including the case (i) "based at least on" (e.g., "A is at least based on B") and, where appropriate in a particular context, (ii) "equal to" (e.g., "A is equal to B"). The case (i) where A is based on B includes at least based on, which may include a configuration where A is coupled to B. Similarly, the term "responsive to" is used to indicate any of its ordinary meanings, including "at least responsive to". The term "at least one" is used to indicate any of its ordinary meanings, including "one or more". The term "at least two" is used to indicate any of its ordinary meanings, including "two or more".

[0036] Unless otherwise indicated by a specific context, the terms "device" and "equipment" are used generally and interchangeably. Unless otherwise indicated, any disclosure of the operation of a device with specific features is also explicitly intended to disclose methods with similar features (and vice versa), and any disclosure of the operation of a device according to a specific configuration is also explicitly intended to disclose methods according to similar configurations (and vice versa). Unless otherwise indicated by a specific context, the terms "method", "process", "procedure" and "technology" are used generally and interchangeably. The terms "element" and "module" can be used to indicate a part of a larger configuration. The term "packet" can correspond to a data unit including a header portion and a payload portion. Any incorporation of a portion of a reference file should also be understood to be incorporated into the definitions of the terms or variables referenced within the portion, where such definitions appear elsewhere in the file, as well as any drawings referenced in the incorporated portion.

[0037] As used herein, the term "communication device" refers to an electronic device that can be used for voice and / or data communications over a wireless communication network. Examples of communication devices include speaker bars, smart speakers, cellular phones, personal digital assistants (PDAs), handheld devices, head-mounted devices, wireless modems, laptop computers, personal computers, etc.

[0038] Certain aspects are described herein with reference to the accompanying drawings. In the description, common features are designated throughout the drawings by common reference numerals. In some of the drawings, multiple instances of a particular type of feature are used. Although the features are physically and / or logically different, the same reference numeral is used for each feature, and the different instances are distinguished by adding a letter to the reference numeral. When referring to features as a group or type herein (e.g., when no reference is made to a particular one of the features), the reference numeral is used without a distinguishing letter. However, when a particular feature among multiple features of the same type is referred to herein, the reference numeral is used with a distinguishing letter. For example, reference numerals are used to identify the features of the same type. Figure 1 , a plurality of transform units are illustrated and associated with reference numerals 132A and 132B. When referring to a specific one of the transform units, such as transform unit 132A, a distinguishing letter "A" is used. However, when referring to any one of the transform units or referring to the transform units as a group, reference numeral 132 is used without a distinguishing letter.

[0039] Figure 1 is a diagram of a particular illustrative example of a system 100 configured to reduce echo associated with a user's speech using a speech neural network and a non-speech neural network. For example, the system 100 may employ a first trained neural network architecture (e.g., the first neural network 134) to separate and process a speech component of the user's speech. The system 100 may also employ a second trained neural network architecture (e.g., the second neural network 136) to separate and process a non-speech component of the user's speech. As described below, the outputs of the two neural network architectures may be combined to produce a version of the user's speech with a reduced amount of echo.

[0040] exist Figure 1 In FIG. 1 , a first user 102 is close to a first microphone 106 and a speaker 110. In some implementations, the first microphone 106 and the speaker 110 may be integrated into a first device, such as a first mobile phone or a first head-mounted device. The speech from the first user 102 may be captured by the first microphone 106. In addition, the noise in the surrounding environment of the first user 102 may be captured by the first microphone 106. Figure 1 , the speech from the first user 102 (and noise from the surrounding environment) captured by the first microphone 106 is characterized as near-end speech 112.

[0041] In addition, Figure 1 , the second user 104 is close to the second microphone 108. In some specific implementations, the second microphone 108 can be integrated into a second device, such as a second mobile phone or a second head-mounted device. The voice from the second user 104 can be captured by the second microphone 108. In addition, the noise in the surrounding environment of the second user 104 can be captured by the second microphone 108. As described below, from the perspective of the first user 102 (or the first microphone 106), the voice from the second user 104 (and the noise from the surrounding environment) captured by the second microphone 108 is characterized as far-end speech 114A.

[0042] Depending on the implementation, the first user 102 and the second user 104 may participate in a communication, such as a voice call or a video call. During the communication, in addition to capturing the near-end voice 112, the first microphone 106 may also inadvertently capture the far-end voice 114B originating from the second user 104. For example, the second microphone 108 may capture the far-end voice 114A and generate a far-end voice signal 116 indicating the far-end voice 114A. The far-end voice signal 116 may be provided to the speaker 110, and the speaker 110 may output the far-end voice 114B. The far-end voice 114B may be substantially similar to the far-end voice 114A; however, a property change or distortion may occur during the processing of the far-end voice signal 116, which results in a certain difference between the far-end voice 114B as output by the speaker 110 and the far-end voice 114A as spoken by the second user 104. For example, the far-end speech signal 116 may undergo additional processing at a device associated with the first user 102, a device associated with the second user 104, or both, which may cause subtle property changes or distortions.

[0043] Because the far-end speech 114B is output by the speaker 110, in addition to the user 102 hearing the far-end speech 114B, the far-end speech 114B may also be unintentionally captured by the first microphone 106. Therefore, the first microphone 106 may capture the near-end speech 112 and the far-end speech 114B, which may exhibit additional changes associated with the propagation of the far-end speech 114B from the speaker 110 to the first microphone 106, such as attenuation, delay, reflection, etc. In response to capturing the near-end speech 112 (and unintentionally capturing a portion of the far-end speech 114B), the first microphone 106 may be configured to generate a near-end speech signal 120.

[0044] One disadvantage of capturing far-end speech 114B is the generation of echo, such as double talk. For example, if the first microphone 106 captures far-end speech 114B (e.g., speech from the second user 104) in addition to the near-end speech 112, the far-end speech 114B may be sent back to the second user 104 in the form of an echo during the communication. Since the speech of the first user 102 and the speech of the second user 104 are more similar to each other than the ambient noise, it may be very difficult to remove the speech of the second user 104 from the speech of the first user 102 in the output of the microphone 106 using conventional techniques such as adaptive linear filtering. In contrast, the techniques described herein reduce the amount of echo sent using separate trained neural networks for speech components and non-speech components, and can eliminate double talk based on the pitch difference between the speech of users 102 and 104. Specifically, the system 100 includes an echo cancellation system 130 that uses separate trained neural networks for speech components and non-speech components and is operable to reduce or cancel echo (eg, double talk) caused by the first microphone 106 capturing far-end speech 114B.

[0045] For illustration, the near-end speech signal 120 is provided to the echo cancellation system 130. The echo cancellation system 130 includes a transform unit 132A, a transform unit 132B, a combination unit 133, a first neural network 134, a second neural network 136, and a third neural network 138. The transform unit 132A may be configured to perform a transform operation on the near-end speech signal 120 to generate a transformed near-end speech signal 142. As described herein, a "transform operation" may correspond to a fast Fourier transform (FFT) operation, a Fourier transform operation, a discrete cosine transform (DCT) operation, or any other transform operation that transforms a time domain signal into a frequency domain signal (as used herein, "frequency domain" may refer to any such transform domain, including a feature domain). Thus, the transform unit 132A may transform the near-end speech signal 120 from a time domain signal into a frequency domain signal. Thus, the transformed near-end speech signal 142 may include a frequency domain near-end speech component (e.g., a frequency domain representation of the near-end speech 112). The transformed near-end speech signal 142 is provided to the combining unit 133 .

[0046] The far-end speech signal 116 may also be provided to the echo cancellation system 130. The transform unit 132B may be configured to perform a transform operation on the far-end speech signal 116 to generate a transformed far-end speech signal 144. Therefore, the transform unit 132B may transform the far-end speech signal 116 from a time domain signal to a frequency domain signal. Therefore, the transformed far-end speech signal 144 may include a frequency domain far-end speech component (e.g., a frequency domain representation of the far-end speech 114A). The transformed far-end speech signal 144 is also provided to the combining unit 133.

[0047] The combining unit 133 may be configured to concatenate, interleave, or otherwise aggregate or combine the transformed near-end speech signal 142 with the transformed far-end speech signal 144 to generate a transformed input speech signal 145. The transformed input speech signal 145 may include a frequency-domain transformed near-end speech component (based on the transformed near-end speech signal 142) stacked with a frequency-domain transformed far-end speech component (based on the transformed far-end speech signal 144). The transformed input speech signal 145 is provided to the first neural network 134 and the second neural network 136.

[0048] The first neural network 134 is configured to perform a first decomposition operation on the transformed input speech signal 145 to generate a speech component 150 of the transformed input speech signal 145. For example, the first neural network 134 may correspond to a speech subnetwork that is trained to apply a speech mask (or identify transform coefficients) to separate and extract the speech component 150 from the transformed input speech signal 145. Because the speech component 150 generally represents the near-end speech 112, the first neural network 134 may be trained to perform additional processing on the speech component 150, such as increasing the gain of the speech component 150. The speech component 150 is provided to the third neural network 138. As described with respect to Figures 3 to 5 The first neural network 134 may have one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0049] Based on using the transform coefficients of the transformed far-end speech signal 144 (e.g., the transform coefficients of the frequency-domain transformed far-end speech component) as a reference signal indicative of the far-end speech 114B captured by the microphone 106, the first neural network 134 may be trained to attenuate or eliminate the component of the transformed input speech signal 145 corresponding to the far-end speech 114B in the voice component 150. Thus, the first neural network 134 may be trained to use this information to perform echo cancellation for the voice component 150. Although the first neural network 134 is described as performing various functions, such as speech / non-speech decomposition, applying gain, and performing echo cancellation, it should be understood that the first neural network 134 may perform any or all of these functions as a single combined operation rather than as a series of discrete operations.

[0050] The second neural network 136 is configured to perform a second decomposition operation on the transformed input speech signal 145 to generate a non-speech component 152 of the transformed input speech signal 145. For example, the second neural network 136 may correspond to a non-speech subnetwork that is trained to apply a non-speech mask (or identify transform coefficients) to separate and extract the non-speech component 152 from the transformed input speech signal 145. The second neural network 136 is also trained to use the transform coefficients of the transformed far-end speech signal 144 received in the transformed input speech signal 145 as a reference signal to attenuate or eliminate the component of the transformed input speech signal 145 in the non-speech component 152 that corresponds to the far-end speech 114B. Because the non-speech component 152 generally represents the far-end speech 114B captured by the first microphone 106, the second neural network 136 may be trained to perform additional processing on the non-speech component 152, such as reducing the gain of the non-speech component 152. The non-speech component 152 is provided to the third neural network 138. As described with respect to Figures 3 to 5 The second neural network 136 may have one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0051] The third neural network 138 is configured to combine the speech component 150 and the non-speech component 152 to generate the transformed output speech signal 146. According to a specific implementation, the third neural network 138 may apply an unconditional unweighted sum of the speech component 150 and the non-speech component 152 to generate the transformed output speech signal 146. According to another specific implementation, the third neural network 138 may apply weights to the components 150, 152. As a non-limiting example, the third neural network 138 may apply a first set of weights to the elements of the speech component 150 and a second set of weights (different from the first set of weights) to the non-speech component 152. In this example, the weighted components may be combined, such as via an element-by-element sum of the corresponding weighted elements. The transformed output speech signal 146 may correspond to an echo-cancelled signal indicating clean speech (e.g., a clean version of the near-end speech 112) with a reduced amount of noise and echo.

[0052] Thus, the techniques described herein improve the quality of speech decomposition and reconstruction by using multiple neural networks 134, 136 to process different corresponding components of the transformed input speech signal 145. For example, the speech component 150 may be processed using the first neural network 134, and the non-speech component 152 may be processed using the second neural network 136. Thus, a single neural network does not have to process different speech portions (e.g., speech portions and non-speech portions) with different statistical values, which enables improved speech quality and weight efficiency. Additionally, the techniques described herein may reduce or eliminate echo (e.g., double talk) caused by the first microphone 106 capturing the far-end speech 114B, compared to conventional techniques (e.g., adaptive linear filtering). For example, the techniques described herein reduce the amount of echo sent using separate trained neural networks for speech components and non-speech components, and may eliminate double talk based on the pitch difference between the voices of the users 102 and 104.

[0053] Figure 2 is a diagram of a particular illustrative example of a system 200 configured to reduce echo associated with input speech using a voiced neural network and a non-voiced neural network. The system 200 includes a first neural network 134, a second neural network 136, and a third neural network 138. Depending on the implementation, the neural networks 134 to 138 may be integrated into one or more processors.

[0054] exist Figure 2 , a transformed input speech signal 145 is provided to a first neural network 134 and a second neural network 136. The first neural network 134 is configured to perform a first decomposition operation and echo cancellation on the transformed input speech signal 145 to generate a speech component 150 of the transformed input speech signal 145. For example, the first neural network 134 may correspond to a speech sub-network trained to apply a speech mask (or identify transform coefficients) to separate and extract the speech component 150 from the transformed input speech signal 145. The speech component 150 is provided to a third neural network 138.

[0055] The second neural network 136 is configured to perform a second decomposition operation and echo cancellation on the transformed input speech signal 145 to generate a non-speech component 152 of the transformed input speech signal 145. For example, the second neural network 136 may correspond to a non-speech sub-network trained to apply a non-speech mask (or identify transform coefficients) to separate and extract the non-speech component 152 from the transformed input speech signal 145. The non-speech component 152 is also provided to the third neural network 138.

[0056] The third neural network 138 is configured to combine the speech component 150 and the non-speech component 152 to generate a transformed output speech signal 146. The transformed output speech signal 146 may correspond to an echo-cancelled signal indicative of clean speech (e.g., a clean version of the near-end speech 112) with a reduced amount of noise and echo.

[0057] Thus, the techniques described herein improve the quality of speech decomposition and reconstruction by using multiple neural networks 134, 136 to process different components of the transformed input speech signal 145. For example, the speech component 150 may be processed using the first neural network 134, and the non-speech component 152 may be processed using the second neural network 136. Thus, a single neural network does not have to process different speech portions (e.g., speech portions and non-speech portions) having completely different probability distributions, which enables improved speech quality and weight efficiency. Additionally, the techniques described herein may reduce or eliminate echo (e.g., double talk) caused by the first microphone 106 capturing the far-end speech 114B, compared to conventional techniques (e.g., adaptive linear filtering). For example, the techniques described herein reduce the amount of echo sent using separate trained neural networks for speech components and non-speech components, and may eliminate double talk based on the pitch difference between the voices of the users 102 and 104.

[0058] Figure 3 is a diagram of a specific illustrative example 300 of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a convolutional u-net architecture. Figure 3 Example 300 includes a combination unit 133 and a neural network 301. The neural network 301 has a convolutional u-net architecture and may correspond to the first neural network 134, the second neural network 136, or both.

[0059] exist Figure 3 , the transformed near-end speech signal 142 and the transformed far-end speech signal 144 are provided to the combination unit 133. The combination unit 133 may be configured to cascade the transformed near-end speech signal 142 and the transformed far-end speech signal 144 to generate a transformed input speech signal 145. The transformed input speech signal 145 may include a frequency-domain transformed near-end speech component (based on the transformed near-end speech signal 142) stacked with a frequency-domain transformed far-end speech component (based on the transformed far-end speech signal 144). The transformed input speech signal 145 is provided to the neural network 301.

[0060] The neural network 301 includes a convolution block 302, a convolution bottleneck 304, and a transposed convolution block 306. The transformed input speech signal 145 is provided to the convolution block 302, which may include a plurality of sets of convolution layers configured to perform a series of downsampling operations on the transformed input speech signal 145 to generate a convolution block output 310. Figure 3 , information from downsampling (e.g., the output of each stage of downsampling performed by the convolution block 302) may be provided to the transposed convolution block 306 via a skip connection. The convolution block output 310 is provided to the convolution bottleneck 304. The convolution bottleneck 304 may include one or more convolution layers configured to generate a convolution bottleneck output 312 based on the convolution block output 310. The convolution bottleneck output 312 is provided to the transposed convolution block 306, which may include multiple sets of convolution layers configured to perform a series of upsampling operations on the convolution bottleneck output 312 in conjunction with information received via the skip connections (e.g., each stage of upsampling concatenates the output of the previous stage of upsampling with the output of the corresponding stage of downsampling) to generate a component 350 based on the convolution bottleneck output 312. The component 350 may correspond to the speech component 150 or the non-speech component 152.

[0061] Figure 4 is a diagram of a specific illustrative example 400 of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a recurrent u-net architecture. Figure 4 Example 400 includes a combination unit 133 and a neural network 401. The neural network 401 has a recurrent u-net architecture and may correspond to the first neural network 134, the second neural network 136, or both.

[0062] exist Figure 4 , the transformed near-end speech signal 142 and the transformed far-end speech signal 144 are provided to the combination unit 133. The combination unit 133 may be configured to cascade the transformed near-end speech signal 142 and the transformed far-end speech signal 144 to generate a transformed input speech signal 145. The transformed input speech signal 145 may include a frequency-domain transformed near-end speech component (based on the transformed near-end speech signal 142) stacked with a frequency-domain transformed far-end speech component (based on the transformed far-end speech signal 144). The transformed input speech signal 145 is provided to the neural network 401.

[0063] The neural network 401 includes a convolution block 402, a long short-term memory (LSTM) / gated recurrent unit (GRU) bottleneck 404, and a transposed convolution block 406. The transformed input speech signal 145 is provided to the convolution block 402, which may include multiple sets of convolution layers configured to perform a series of downsampling operations on the transformed input speech signal 145 to generate a convolution block output 410. Figure 4, information from downsampling (e.g., the output of each stage of downsampling performed by the convolution block 402) may be provided to the transposed convolution block 406 via a skip connection. The convolution block output 410 is provided to the bottleneck 404. The bottleneck 404 may include one or more convolution layers configured to generate a bottleneck output 412 based on the convolution block output 410. The bottleneck output 412 is provided to the transposed convolution block 406, which may include multiple sets of convolution layers configured to perform a series of upsampling operations on the bottleneck output 412 in conjunction with information received via the skip connection (e.g., each stage of upsampling cascades the output of the previous stage of upsampling with the output of the corresponding stage of downsampling) to generate a component 450 based on the bottleneck output 412. The component 450 may correspond to the speech component 150 or the non-speech component 152.

[0064] Figure 5 is a diagram of a specific illustrative example 500 of performing a decomposition operation on stacked and transformed near-end speech and far-end speech using a recurrent layer architecture. Figure 5 Example 500 includes combination unit 133 and neural network 501. Neural network 501 has a recurrent layer architecture and may correspond to first neural network 134, second neural network 136, or both.

[0065] exist Figure 5 , the transformed near-end speech signal 142 and the transformed far-end speech signal 144 are provided to the combination unit 133. The combination unit 133 may be configured to cascade the transformed near-end speech signal 142 and the transformed far-end speech signal 144 to generate a transformed input speech signal 145. The transformed input speech signal 145 may include a near-end speech component (based on the transformed near-end speech signal 142) transformed in the frequency domain stacked with a far-end speech component (based on the transformed far-end speech signal 144) transformed in the frequency domain. The transformed input speech signal 145 is provided to the neural network 501.

[0066] The neural network 501 includes three GRU layers 502, 504, 506. The transformed input speech signal 145 is provided to the GRU layer 502. The GRU layer 502 processes the transformed input speech signal 145 to generate a GRU layer output 510. The GRU layer 504 processes the GRU layer output 510 to generate a GRU layer output 512. The GRU layer 506 processes the GRU layer output 512 to generate a component 550 that may correspond to a speech component 150 or a non-speech component 152. For illustration, the GRU layers 502, 504, and 506 may be trained to directly generate speech masks or speech (which may be learned or predefined in some transformed domain).

[0067] Although the recurrent layer architecture of the neural network 501 is illustrated as including three GRU layers, in other implementations, the neural network 501 may include stacked recurrent neural network (RNN) layers, LSTM layers, GRU layers, or any combination thereof. Although three recurrent layers are illustrated, in other implementations, any number of recurrent layers may be used.

[0068] Figure 6 6 is a block diagram illustrating an implementation 600 of an integrated circuit 602 configured to reduce echo associated with input speech using a speech neural network and a non-speech neural network. The integrated circuit 602 includes one or more processors 610 that include the echo cancellation system 130. The integrated circuit 602 also includes a signal input terminal 604 (such as a bus interface) to enable receiving a near-end speech signal 120. The integrated circuit 602 includes a signal output terminal 606 (such as a bus interface) to enable output of an output speech signal 620. The output speech signal 620 may correspond to a time domain version of the transformed output speech signal 146. For example, the one or more processors 610 may perform an inverse transform operation on the transformed output speech signal 146 to generate the output speech signal 620 provided to the signal output 606. The integrated circuit 602 enables echo cancellation for stacked and transformed near-end speech and far-end speech, such as Figure 1 Described in.

[0069] As an illustrative, non-limiting example, Figure 7 A specific implementation 700 of a mobile device 702, such as a phone or tablet, is depicted. The mobile device 702 includes a display screen 704, a microphone 706, and a speaker 708. For illustration, the microphone 706 may correspond to the first microphone 106, and the speaker 708 may correspond to Figure 1 110. Components of the one or more processors 610, including the echo cancellation system 130, are integrated into the mobile device 702 and are illustrated using dashed lines to indicate internal components that are generally not visible to a user of the mobile device 702. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 706 that is played from the speaker 708. For example, during a video or voice call with a remote participant using another mobile device, the echo cancellation system 130 reduces or removes the remote participant's voice from the audio captured by the microphone 706 so that the remote participant's voice is not transmitted back to the remote participant.

[0070] Figure 8 An implementation 800 of a portable electronic device corresponding to a camera device 802 is depicted. The camera device 802 includes a microphone 806 and a speaker 808. For illustration, the microphone 806 may correspond to the first microphone 106, and the speaker 808 may correspond to Figure 1 110. Components of the one or more processors 610, including the echo cancellation system 130, are integrated into the camera device 802 and are illustrated using dashed lines to indicate internal components that are generally not visible to a user of the camera device 802. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio played from the speaker 808 that is captured by the microphone 806. For example, the camera device 802 may be used to capture a video recording that includes audio. In some scenarios, a user may use the microphone 806 to insert an audio annotation into the video recording during playback of the video recording. The echo cancellation system 130 reduces or removes audio from the video recording from the audio captured by the microphone 806 so that the audio annotation does not have an echo of the playback audio.

[0071] Fig. 9 A specific implementation 900 of a wearable electronic device 902 (illustrated as a "smart watch") is depicted. The wearable electronic device 902 is coupled to or includes a display screen 904 to display video data. In addition, the wearable electronic device 902 includes a microphone 906 and a speaker 908. For illustration, the microphone 906 may correspond to the first microphone 106, and the speaker 908 may correspond to Figure 1 110. Components of the one or more processors 610, including the echo cancellation system 130, are integrated into the wearable electronic device 902 and are illustrated using dashed lines to indicate internal components that are generally not visible to a user of the wearable electronic device 902. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 906 that is played from the speaker 908. For example, during a voice call with a remote participant using another mobile device, the echo cancellation system 130 reduces or removes the remote participant's voice from the audio captured by the microphone 906 so that the remote participant's voice is not transmitted back to the remote participant.

[0072] Fig.10 1000 is a specific implementation of a wireless speaker and voice activated device 1002. The wireless speaker and voice activated device 1002 may have wireless network connectivity and be configured to perform auxiliary operations. One or more processors 610 are included in the wireless speaker and voice activated device 1002 and include the echo cancellation system 130. In certain aspects, the wireless speaker and voice activated device 1002 includes one or more microphones 1038 and one or more speakers 1036, and also includes a display device 1004 or is coupled to the display device for video playback. The one or more microphones 1038 may correspond to the first microphone 106, and the one or more speakers 1036 may correspond to Figure 1The speaker 110 of the wireless speaker and voice activated device 1002. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio played from one or more speakers 1036 captured by one or more microphones 1038. For example, when the speaker 1036 is playing audio (e.g., music, podcasts, etc.), the user can use the one or more microphones 1038 to issue a verbal command to the wireless speaker and voice activated device 1002. The echo cancellation system 130 reduces or removes the audio output by the speaker 1036 from the audio captured by the microphone 1038 to improve the voice recognition of the verbal command. In response to the verbal command, the wireless speaker and voice activated device 1002 can perform an auxiliary operation, such as via executing a voice activation system (e.g., an integrated auxiliary application). The auxiliary operation can include adjusting the temperature, playing media content (such as stored or streaming audio and video content), turning on the lights, etc. For example, the auxiliary operation is performed in response to receiving a command after a keyword or key phrase (e.g., "Hello, assistant").

[0073] Fig.11 An implementation 1100 of a portable electronic device corresponding to a virtual reality, augmented reality, or mixed reality head mounted device 1102 is depicted. A visual interface device 1104 is positioned in front of the user's eyes to enable a video associated with an augmented reality, mixed reality, or virtual reality scene to be displayed to the user while wearing the head mounted device 1102. The head mounted device 1102 includes a microphone 1106 and a speaker 1108. For illustration, the microphone 1106 may correspond to the first microphone 106, and the speaker 1108 may correspond to Figure 1 110. Components of the one or more processors 610, including the echo cancellation system 130, are integrated into the head mounted device 1102. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 1106 and played from the speaker 1108. For example, if the user is using the head mounted device 1102 to experience an immersive multi-participant open world virtual reality (VR) scene, the speaker 1108 may output audio from other participants of the VR scene. The echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 1106 and played from the speaker 1108. For example, the echo cancellation system 130 reduces or removes the voices of other participants from the audio captured by the microphone 1106 so that the voices of other participants are not sent back to the participant via double talk during the conversation.

[0074] Fig.12An implementation 1200 is depicted in which the echo cancellation system 130 corresponds to or is integrated within a vehicle 1202, illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). The vehicle 1202 includes a display device 1204, a microphone 1206, and a speaker 1208. For illustration, the microphone 1206 may correspond to the first microphone 106, and the speaker 1208 may correspond to Figure 1 110. In some implementations, the vehicle 1202 is manned (e.g., carrying a pilot, one or more passengers, or both), and the display device 1204 is inside a cabin of the vehicle 1202. Components of the one or more processors 610, including the echo cancellation system 130, are integrated into the vehicle 1202. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 1206 that is played from the speaker 1208. For example, during communication with a flight controller, the echo cancellation system 130 reduces or removes the flight controller's voice from the audio captured by the microphone 1206 so that the flight controller's voice is not sent back to the flight controller.

[0075] Fig.13 13. An implementation 1300 is depicted of a vehicle 1302 illustrated as a car that includes an echo cancellation system 130, a display device 1320, a microphone 1334, and a speaker 1336. The microphone 1334 may correspond to the first microphone 106, and the speaker 1336 may correspond to the first microphone 106. Figure 1 The components of the one or more processors 610, including the echo cancellation system 130, are integrated in the vehicle 1302. In a specific example, the echo cancellation system 130 operates to perform echo cancellation to reduce or remove audio captured by the microphone 1334 that is played from the speaker 1336. For example, during a voice call with a remote participant, the echo cancellation system 130 reduces or removes the voice of the remote participant from the audio captured by the microphone 1334 so that the voice of the remote participant is not transmitted back to the remote participant.

[0076] Fig.14 is a flow chart of a specific example 1400 of a method for reducing echo associated with input speech using a voiced neural network and a non-voiced neural network. In various implementations, the method 1400 may be performed by Figure 1 Echo cancellation system 130, Figure 1 System 100, Figure 2 System 200, Figure 3 One or more components of example 300, Figure 4 One or more components of example 400, Figure 5 One or more components of example 500, Figure 6 IC 602 or Figures 7 to 13 Executed by one or more of any of the devices.

[0077] The method 1400 includes, at block 1402, performing a first decomposition operation on a transformed input speech signal at a first neural network to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain. For example, referring to Figure 1 , the first neural network 134 performs a first decomposition operation on the transformed input speech signal 145 to generate a speech component 150 of the transformed input speech signal 145. According to some specific implementations of the method 1400, the first neural network 134 applies a speech mask to separate and extract the speech component 150 from the transformed input speech signal 145. The transformed input speech signal 145 may include a frequency domain near-end speech component (based on the transformed near-end speech signal 142) stacked with a frequency domain transformed far-end speech component (based on the transformed far-end speech signal 144).

[0078] The method 1400 also includes, at block 1404, performing a second decomposition operation on the transformed input speech signal at the second neural network to generate a non-speech component of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. For example, referring to Figure 1 , the second neural network 136 performs a second decomposition operation on the transformed input speech signal 145 to generate a non-speech component 152 of the transformed input speech signal 145. According to some specific implementations of the method 1400, the second neural network 136 applies a non-speech mask to separate and extract the non-speech component 152 from the transformed input speech signal 145.

[0079] The method 1400 also includes, at block 1406, combining the speech component and the non-speech component at a third neural network to generate a transformed output speech signal. Figure 1 , the third neural network 138 combines the speech component 150 and the non-speech component 152 to generate a transformed output speech signal 146.

[0080] According to some implementations, method 1400 may include performing a first transformation operation on the near-end speech signal to generate a transformed near-end speech signal. Figure 1 , the transformation unit 132A may perform a transformation operation on the near-end speech signal 120 to generate a transformed near-end speech signal 142. According to some specific implementations, the method 1400 may include performing a second transformation operation on the far-end speech signal to generate a transformed far-end speech signal. Figure 1, the transformation unit 132B may perform a transformation operation on the far-end speech signal 116 to generate a transformed far-end speech signal 144. According to some specific implementations, the method 1400 may also include cascading the transformed near-end speech signal and the transformed far-end speech signal to generate a transformed input speech signal. Figure 1 , the combining unit 133 may cascade the transformed near-end speech signal 142 and the transformed far-end speech signal 144 to generate a transformed input speech signal 145 .

[0081] The method 1400 improves the quality of speech decomposition and reconstruction by using multiple neural networks 134, 136 to process different components of the transformed input speech signal 145. For example, the speech component 150 may be processed using the first neural network 134, and the non-speech component 152 may be processed using the second neural network 136. Thus, a single neural network does not have to process different speech portions (e.g., speech portions and non-speech portions) having completely different probability distributions. In addition, compared to conventional techniques (e.g., adaptive linear filtering), the method 1400 may reduce or eliminate echo (e.g., double talk) caused by the first microphone 106 that captures the far-end speech 114B. For example, the techniques described herein reduce the amount of echo sent using separate trained neural networks for speech components and non-speech components, and may eliminate double talk based on the pitch difference between the voices of users 102 and 104.

[0082] Fig.14 The method 1400 may be implemented by an FPGA device, an ASIC, a processing unit (such as a CPU), a DSP, a controller, another hardware device, a firmware device, or any combination thereof. For example, Fig.14 The method 1400 may be performed by a processor executing instructions, such as reference Fig.15 As described in the processor 1510.

[0083] Fig.15A specific implementation 1500 is depicted in which a device 1502 includes one or more processors 1510 that include components of the echo cancellation system 130. For example, the one or more processors 1510 include a transform unit 132A, a transform unit 132B, a combination unit 133, a first neural network 134, a second neural network 136, and a third neural network 138. The one or more processors 1510 also include an inverse transform unit 1532 that is configured to perform an inverse transform operation (e.g., an inverse fast Fourier transform (IFFT) operation, an inverse discrete cosine transform (IDCT) operation, etc.) on the transformed output speech signal 146 to generate an output speech signal 620. The device 1502 also includes an input interface 1504 (e.g., one or more buses or wireless interfaces) configured to receive an input signal (such as the near-end speech signal 120), and an output interface 1506 (e.g., one or more buses or wireless interfaces) configured to output a signal (such as the output speech signal 620). As an illustrative, non-limiting example, device 1502 may correspond to a system on a chip or other modular device that can be integrated into other systems to provide data encoding, such as within a mobile phone, another communication device, an entertainment system, or a vehicle. According to some implementations, device 1502 may be integrated into a server, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, a motor vehicle (such as a car), or any combination thereof.

[0084] In the illustrated implementation 1500, the device 1502 includes a memory 1520 (e.g., one or more memory devices) containing instructions 1522, and one or more processors 1510 are coupled to the memory 1520 and configured to execute the instructions 1522 from the memory 1520. For example, executing the instructions 1522 causes the one or more processors 1510 (e.g., the transform unit 132A) to perform a first transform operation on the near-end speech signal 120 to generate a transformed near-end speech signal 142. Executing the instructions 1522 also causes the one or more processors 1510 (e.g., the transform unit 132B) to perform a second transform operation on the far-end speech signal 116 to generate a transformed far-end speech signal 144. Executing the instructions 1522 may also cause the one or more processors 1510 (e.g., the combining unit 133) to cascade the transformed near-end speech signal 142 with the transformed far-end speech signal 144 to generate a transformed input speech signal 145. As described above, the first neural network 134 may generate the speech component 150, the second neural network 136 may generate the non-speech component 152, and the third neural network 138 may combine the speech component 150 with the non-speech component 152 to generate the transformed output speech signal 146. The inverse transform unit 1532 may be configured to perform an inverse transform operation (e.g., an inverse fast Fourier transform (IFFT) operation, an inverse discrete cosine transform (IDCT) operation, etc.) on the transformed output speech signal to generate the output speech signal 620.

[0085] refer to Fig.16 , depicts a block diagram of a particular exemplary implementation of a device and generally designated 1600. In various implementations, the device 1600 may have more Fig.16 More or fewer components may be illustrated. In an exemplary implementation, the device 1600 may execute the reference Figures 1 to 15 One or more of the operations described.

[0086] In certain implementations, device 1600 includes processor 1606 (e.g., a CPU). Device 1600 may include one or more additional processors 1610 (e.g., one or more DSPs, one or more GPUs, or a combination thereof). Processor 1610 includes components of echo cancellation system 130, such as first neural network 134, second neural network 136, and third neural network 138. In some implementations, processor 1610 includes additional components, such as transform unit 132A, transform unit 132B, combination unit 133, inverse transform unit 1532, etc. According to some implementations, processor 1610 includes a speech and music coder-decoder (CODEC) (not shown). In these implementations, components of echo cancellation system 130 may be integrated into the speech and music CODEC.

[0087] The device 1600 also includes a memory 1686 and a CODEC 1634. The memory 1686 may include instructions 1656 that are executable by one or more additional processors 1610 (or processor 1606) to implement the functionality described herein. The device 1600 may include a modem 1640 coupled to an antenna 1690 via a transceiver 1650.

[0088] Device 1600 may include a display 1628 coupled to display controller 1626. Speaker 1696 and microphone 1694 may be coupled to CODEC 1634. Depending on the implementation, speaker 1696 corresponds to Figure 1 Depending on the implementation, the microphone 1694 corresponds to Figure 1 The CODEC 1634 may include a digital-to-analog converter (DAC) 1602 and an analog-to-digital converter (ADC) 1604. In a particular implementation, the CODEC 1634 may receive an analog signal from the microphone 1694, convert the analog signal to a digital signal using the analog-to-digital converter 1604, and provide the digital signal to the processor 1610. The processor 1610 may process the digital signal. In a particular implementation, the processor 1610 may provide the digital signal to the CODEC 1634. The CODEC 1634 may convert the digital signal to an analog signal using the digital-to-analog converter 1602, and may provide the analog signal to the speaker 1696.

[0089] In a particular implementation, device 1600 may be included in a system-in-package or system-on-chip device 1622. In a particular implementation, memory 1686, processor 1606, processor 1610, display controller 1626, CODEC 1634, and modem 1640 are included in the system-in-package or system-on-chip device 1622. In a particular implementation, input device 1630 and power supply 1644 are coupled to the system-in-package or system-on-chip device 1622. In addition, in a particular implementation, as shown in FIG. Fig.16 As illustrated, the display 1628, input device 1630, speaker 1696, microphone 1694, antenna 1690, and power supply 1644 are external to the system-in-package or system-on-chip device 1622. In certain implementations, each of the display 1628, input device 1630, speaker 1696, microphone 1694, antenna 1690, and power supply 1644 can be coupled to a component (such as an interface or controller) of the system-in-package or system-on-chip device 1622. In some implementations, the device 1600 includes additional memory that is external to the system-in-package or system-on-chip device 1622 and coupled to the system-in-package or system-on-chip device 1622 via an interface or controller.

[0090] Device 1600 may include a smart speaker (e.g., processor 1606 may execute instructions 1656 to run a voice-controlled digital assistant application), a speaker, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a DVD player, a tuner, a camera, a navigation device, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, a vehicle, or any combination thereof.

[0091] In conjunction with the described implementation, an apparatus includes means for performing a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal. The transformed input speech signal includes a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain. The means for performing the first decomposition operation includes the first neural network 134, the echo cancellation system 130, the processor 610, the processor 1510, the processor 1610, one or more other circuits or components configured to perform the first decomposition operation, or any combination thereof.

[0092] The apparatus also includes means for performing a second decomposition operation on the transformed input speech signal to generate a non-voice component of the transformed input speech signal. The means for performing the first decomposition operation and the means for performing the second decomposition operation perform echo cancellation on the transformed input speech signal. The means for performing the second decomposition operation includes the second neural network 136, the echo cancellation system 130, the processor 610, the processor 1510, the processor 1610, one or more other circuits or components configured to perform the second decomposition operation, or any combination thereof.

[0093] The apparatus also includes means for combining the speech component and the non-speech component to generate a transformed output speech signal. For example, the means for combining includes the third neural network 138, the echo cancellation system 130, the processor 610, the processor 1510, the processor 1610, one or more other circuits or components configured to combine the speech component and the non-speech component, or any combination thereof.

[0094] In some implementations, a non-transitory computer-readable medium includes instructions that, when executed by one or more processors, cause the one or more processors to perform a first decomposition operation on a transformed input speech signal (e.g., the transformed input speech signal 145) at a first neural network (e.g., the first neural network 134) to generate a speech component (e.g., the speech component 150) of the transformed input speech signal. The transformed input speech signal includes a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component. Execution of the instructions further causes the one or more processors to perform a second decomposition operation on the transformed input speech signal at a second neural network (e.g., the second neural network 136) to generate a non-speech component (e.g., the non-speech component 152) of the transformed input speech signal. The first neural network and the second neural network perform echo cancellation on the transformed input speech signal. Execution of the instructions further causes the one or more processors to combine the speech component with the non-speech component at a third neural network (e.g., the third neural network 138) to generate a transformed output speech signal (e.g., the transformed output speech signal 146).

[0095] Specific aspects of the present disclosure are described below in various sets of related embodiments:

[0096] Example 1

[0097] A device comprising: a first neural network configured to perform a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal, the transformed input speech signal comprising a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain; a second neural network configured to perform a second decomposition operation on the transformed input speech signal to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; and a third neural network configured to combine the speech component with the non-speech component to generate a transformed output speech signal.

[0098] Example 2

[0099] The apparatus of embodiment 1, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0100] Example 3

[0101] The apparatus of embodiment 1 or 2, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0102] Example 4

[0103] The apparatus according to any one of embodiments 1 to 3, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

[0104] Example 5

[0105] According to the device described in any one of embodiments 1 to 4, the device includes: a first transformation unit, which is configured to perform a first transformation operation on a near-end speech signal to generate a transformed near-end speech signal; a second transformation unit, which is configured to perform a second transformation operation on a far-end speech signal to generate a transformed far-end speech signal; and a combination unit, which is configured to cascade the transformed near-end speech signal and the transformed far-end speech signal to generate the transformed input speech signal.

[0106] Example 6

[0107] According to any one of embodiments 1 to 5, the device further includes a microphone configured to capture near-end speech to generate the near-end speech signal.

[0108] Example 7

[0109] According to the device of embodiment 6, the device also includes a speaker configured to output a far-end voice associated with the far-end voice signal, wherein the speaker is close to the microphone.

[0110] Example 8

[0111] The device of any one of embodiments 1 to 7, wherein the first neural network, the second neural network, and the third neural network are integrated into a mobile device.

[0112] Example 9

[0113] The apparatus according to any one of embodiments 1 to 8, wherein the first neural network is configured to apply a speech mask to separate and extract the speech component from the transformed input speech signal.

[0114] Example 10

[0115] The apparatus according to any one of embodiments 1 to 9, wherein the second neural network is configured to apply a non-speech mask to separate and extract the non-speech component from the transformed input speech signal.

[0116] Embodiment 11

[0117] A method, the method comprising: performing a first decomposition operation on a transformed input speech signal at a first neural network to generate a speech component of the transformed input speech signal, the transformed input speech signal comprising a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain; performing a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; and combining the speech component with the non-speech component at a third neural network to generate a transformed output speech signal.

[0118] Example 12

[0119] The method of embodiment 11, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0120] Example 13

[0121] The method of embodiment 11 or 12, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0122] Embodiment 14

[0123] A method according to any one of embodiments 11 to 13, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

[0124] Embodiment 15

[0125] According to the method described in any one of Embodiments 11 to 14, the method further includes: performing a first transformation operation on a near-end speech signal to generate a transformed near-end speech signal; performing a second transformation operation on a far-end speech signal to generate a transformed far-end speech signal; and cascading the transformed near-end speech signal with the transformed far-end speech signal to generate the transformed input speech signal.

[0126] Example 16

[0127] According to the method of embodiment 15, the method also includes capturing near-end speech to generate the near-end speech signal.

[0128] Embodiment 17

[0129] A method according to any one of embodiments 11 to 16, wherein the first neural network is configured to apply a speech mask to separate and extract the speech component from the transformed input speech signal.

[0130] Embodiment 18

[0131] A method according to any one of embodiments 11 to 17, wherein the second neural network is configured to apply a non-speech mask to separate and extract the non-speech component from the transformed input speech signal.

[0132] Embodiment 19

[0133] A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: perform a first decomposition operation on a transformed input speech signal at a first neural network to generate a speech component of the transformed input speech signal, the transformed input speech signal comprising a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain; perform a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; and combine the speech component with the non-speech component at a third neural network to generate a transformed output speech signal.

[0134] Embodiment 20

[0135] The non-transitory computer readable medium of embodiment 19, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0136] Embodiment 21

[0137] The non-transitory computer readable medium of claim 19 or 20, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

[0138] Embodiment 22

[0139] A non-transitory computer readable medium according to any one of embodiments 19 to 21, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

[0140] Embodiment 23

[0141] A non-transitory computer-readable medium according to any one of embodiments 19 to 22, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: perform a first transformation operation on a near-end speech signal to generate a transformed near-end speech signal; perform a second transformation operation on a far-end speech signal to generate a transformed far-end speech signal; and cascade the transformed near-end speech signal with the transformed far-end speech signal to generate the transformed input speech signal.

[0142] Embodiment 24

[0143] A non-transitory computer readable medium according to any one of embodiments 19 to 23, wherein the first neural network applies a voice mask to separate and extract the voice component from the transformed input speech signal.

[0144] Embodiment 25

[0145] A non-transitory computer readable medium according to any one of embodiments 19 to 24, wherein the second neural network applies a non-speech mask to separate and extract the non-speech component from the transformed input speech signal.

[0146] Embodiment 26

[0147] A device, the device comprising: a component for performing a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal, the transformed input speech signal comprising a near-end speech component transformed in the frequency domain stacked with a far-end speech component transformed in the frequency domain; a component for performing a second decomposition operation on the transformed input speech signal to generate a non-speech component of the transformed input speech signal, wherein the component for performing the first decomposition operation and the component for performing the second decomposition operation perform echo cancellation on the transformed input speech signal; and a component for combining the speech component and the non-speech component to generate a transformed output speech signal.

[0148] Embodiment 27

[0149] A device as in embodiment 26, wherein the means for performing the first decomposition operation and the means for performing the second decomposition operation perform noise reduction on the transformed input speech signal.

[0150] Embodiment 28

[0151] According to the device described in embodiment 26 or 27, the device also includes: a component for performing a first transformation operation on a near-end speech signal to generate a transformed near-end speech signal; a component for performing a second transformation operation on a far-end speech signal to generate a transformed far-end speech signal; and a component for cascading the transformed near-end speech signal and the transformed far-end speech signal to generate the transformed input speech signal.

[0152] Embodiment 29

[0153] The apparatus of any one of embodiments 26 to 28, wherein the means for performing the first decomposition operation applies a speech mask to separate and extract the speech component from the transformed input speech signal.

[0154] Embodiment 30

[0155] A device according to any one of embodiments 26 to 29, wherein the means for performing the second decomposition operation applies an unvoiced mask to separate and extract the unvoiced component from the transformed input speech signal.

[0156] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. The skilled person may implement the described functionality in different ways for each specific application, and such specific implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0157] The steps of the method or algorithm described in conjunction with the specific implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0158] The previous description of the disclosed implementations is provided to enable those skilled in the art to make or use the disclosed implementations. Various modifications to these implementations will be apparent to those skilled in the art, and the principles defined herein may be applied to other implementations without departing from the scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the specific implementations shown herein, but should be given the broadest scope possible consistent with the principles and novel features as defined by the appended claims.

Claims

1. A device, comprising: a first neural network configured to perform a first decomposition operation on a transformed input speech signal to generate a speech component of the transformed input speech signal, the transformed input speech signal comprising a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component; a second neural network configured to perform a second decomposition operation on the transformed input speech signal to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; and A third neural network is configured to combine the speech component with the non-speech component to generate a transformed output speech signal.

2. The apparatus of claim 1, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

3. The apparatus of claim 1, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

4. The apparatus of claim 1, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

5. The device according to claim 1, further comprising: a first transform unit configured to perform a first transform operation on the near-end speech signal to generate a transformed near-end speech signal; a second transform unit, the second transform unit being configured to perform a second transform operation on the far-end speech signal to generate a transformed far-end speech signal; and A combining unit is configured to cascade the transformed near-end speech signal and the transformed far-end speech signal to generate the transformed input speech signal. 6 . The device of claim 5 , further comprising a microphone configured to capture near-end speech to generate the near-end speech signal. 7 . The device of claim 6 , further comprising a speaker configured to output a far-end speech associated with the far-end speech signal, wherein the speaker is proximate to the microphone.

8. The device of claim 1, wherein the first neural network, the second neural network, and the third neural network are integrated into a mobile device.

9. The apparatus of claim 1, wherein the first neural network is configured to apply a speech mask to separate and extract the speech component from the transformed input speech signal.

10. The apparatus of claim 1, wherein the second neural network is configured to apply a non-speech mask to separate and extract the non-speech component from the transformed input speech signal.

11. A method comprising: performing a first decomposition operation on a transformed input speech signal at a first neural network to generate a voice component of the transformed input speech signal, the transformed input speech signal comprising a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component; performing a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; as well as The speech component and the non-speech component are combined at a third neural network to generate a transformed output speech signal.

12. The method of claim 11, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

13. The method of claim 11, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

14. The method of claim 11, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

15. The method according to claim 11, further comprising: performing a first transform operation on the near-end speech signal to generate a transformed near-end speech signal; performing a second transform operation on the far-end speech signal to generate a transformed far-end speech signal; as well as The transformed near-end speech signal and the transformed far-end speech signal are concatenated to generate the transformed input speech signal.

16. The method of claim 15, further comprising capturing near-end speech to generate the near-end speech signal.

17. The method of claim 11, wherein the first neural network is configured to apply a speech mask to separate and extract the speech component from the transformed input speech signal.

18. The method of claim 11, wherein the second neural network is configured to apply a non-speech mask to separate and extract the non-speech component from the transformed input speech signal.

19. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: performing a first decomposition operation on a transformed input speech signal at a first neural network to generate a voice component of the transformed input speech signal, the transformed input speech signal comprising a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component; performing a second decomposition operation on the transformed input speech signal at a second neural network to generate a non-speech component of the transformed input speech signal, wherein the first neural network and the second neural network perform echo cancellation on the transformed input speech signal; as well as The speech component and the non-speech component are combined at a third neural network to generate a transformed output speech signal.

20. The non-transitory computer-readable medium of claim 19, wherein the first neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

21. The non-transitory computer-readable medium of claim 19, wherein the second neural network has one of a recurrent layer architecture, a convolutional u-net architecture, or a recurrent u-net architecture.

22. The non-transitory computer readable medium of claim 19, wherein the first neural network and the second neural network perform noise reduction on the transformed input speech signal.

23. The non-transitory computer readable medium of claim 19, wherein the instructions, when executed by the one or more processors, further cause the one or more processors to: performing a first transform operation on the near-end speech signal to generate a transformed near-end speech signal; performing a second transform operation on the far-end speech signal to generate a transformed far-end speech signal; and The transformed near-end speech signal and the transformed far-end speech signal are concatenated to generate the transformed input speech signal.

24. The non-transitory computer-readable medium of claim 19, wherein the first neural network applies a speech mask to separate and extract the speech component from the transformed input speech signal.

25. The non-transitory computer readable medium of claim 19, wherein the second neural network applies an unspeech mask to separate and extract the unspeech component from the transformed input speech signal.

26. An apparatus, comprising: means for performing a first decomposition operation on a transformed input speech signal to generate a voice component of the transformed input speech signal, the transformed input speech signal comprising a frequency-domain transformed near-end speech component stacked with a frequency-domain transformed far-end speech component; means for performing a second decomposition operation on the transformed input speech signal to generate a non-voice component of the transformed input speech signal, wherein the means for performing the first decomposition operation and the means for performing the second decomposition operation perform echo cancellation on the transformed input speech signal; and Means for combining the voiced component with the non-voiced component to generate a transformed output speech signal.

27. The apparatus of claim 26, wherein the means for performing the first decomposition operation and the means for performing the second decomposition operation perform noise reduction on the transformed input speech signal.

28. The apparatus according to claim 26, further comprising: means for performing a first transformation operation on the near-end speech signal to generate a transformed near-end speech signal; means for performing a second transformation operation on the far-end speech signal to generate a transformed far-end speech signal; and Means for concatenating the transformed near-end speech signal with the transformed far-end speech signal to generate the transformed input speech signal.

29. The apparatus of claim 26, wherein the means for performing the first decomposition operation applies a speech mask to separate and extract the speech component from the transformed input speech signal.

30. The device of claim 26, wherein the means for performing the second decomposition operation applies an unvoiced mask to separate and extract the unvoiced component from the transformed input speech signal.

Images