Pre-conditioning audio for machine awareness

Abstract

The invention relates to machine-aware preconditioning audio. An apparatus and method for pre-conditioning audio for machine awareness. The machine perception is different from the human perception, and processing parameters for the machine-aware application (e.g., voice-to-text processing) are different than processing parameters for the human-aware application (e.g., voice communication). These different parameters may produce pre-conditioned audio that is worsened with respect to human perception and improved with respect to machine perception.

Description

[0001] This application is a divisional application of the invention patent application with application number 202080060167.0, application date August 27, 2020, and invention name “Pre-adjusted audio for machine perception”.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims priority to European Patent Application No. 19194685.4 filed on August 30, 2019 and U.S. Provisional Patent Application No. 62 / 893,972 filed on August 30, 2019, which are incorporated herein by reference. Technical Field

[0004] The present disclosure relates to audio processing, and more particularly to machine perception of audio. Background Art

[0005] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims in this application and are not admitted to be prior art by inclusion in this section.

[0006] An important use of audio processing is to process audio in a way that improves (or at least influences in a way that is acceptable to human listeners) human perception. For example, the presence of echo can be very distracting to a human, so an audio processing system may include aggressive echo cancellation to improve the (perceived) performance of the audio processing system. This performance improvement may be measured qualitatively (e.g., through listener surveys), quantitatively (e.g., through various perceptual metrics), etc.

[0007] Humans are not the only intended audience for audio communications. Machines can also perceive audio. Machine perception of audio can often be referred to as Automatic Speech Recognition (ASR). ASR includes activities such as wake-up word detection, speech-to-text processing, speech recognition, speaker identification, emotion recognition, language recognition, etc. Summary of the invention

[0008] Although human perception and machine perception are similar, there are many differences between human perception and machine perception. As explained in detail below, these differences result in different audio processing parameters for machine perception than for human perception. In fact, these different parameters often result in processed audio that is worse according to human perception, while the performance of machine perception is improved.

[0009] Embodiments described herein relate to pre-conditioning audio for machine perception.

[0010] According to an embodiment, a method processes audio for machine perception. The method includes receiving an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The method also includes preconditioning the audio signal according to machine perception parameters to generate a preconditioned audio signal, wherein the machine perception parameters are different from human perception parameters. The method also includes performing machine perception, including automatic speech recognition, on the preconditioned audio signal to generate a machine perception output.

[0011] The human perception parameter may correspond to a first echo cancellation parameter having a convergence of 100 to 200 ms, and the machine perception parameter may correspond to a second echo cancellation parameter having a convergence of less than 50 ms. The human perception parameter may correspond to a first noise suppression target that is more than 20 dB below the speech level, and the machine perception parameter may correspond to a second noise suppression target that is 10 to 15 dB below the speech level. The human perception parameter may correspond to comfort noise, and the machine perception parameter may correspond to a masking noise that is 20 to 25 dB greater than the comfort noise.

[0012] According to one aspect, a method of processing audio for machine perception is presented. The method may include receiving an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The method may include preconditioning the audio signal by adding noise to the audio signal to generate a preconditioned audio signal. The method may include performing machine perception, including automatic speech recognition, on the preconditioned audio signal to generate a machine perception output.

[0013] The added noise may have the characteristics of pink or white noise. Within this document, the added noise may also be denoted as masking noise. This masking noise may be added continuously to speech without any detection or gating around the presence and detection of speech. The masking noise may have the characteristics of pink or (spectral) white noise, with a spectral shape similar to the profile of Hoth noise, but at a level more than 10 dB higher than Hoth noise. Such noise levels in listening tests may be considered intrusive, artificial, and an overall degradation to the human listening experience. However, the addition and blurring of other lower level noise that may be present in the recording, or artifacts that may be caused by noise suppression, may have a net positive impact on the performance of automatic speech recognition.

[0014] The uniformity of the noise field of the preconditioned audio signal may be increased compared to the uniformity of the noise field of the audio signal. In this way, the added noise may add ambiguity to the preconditioned audio signal.

[0015] According to another aspect, a method for processing audio for human perception and for machine perception is presented. The method may include receiving an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The method may include preprocessing the audio signal for human perception by reducing echo residual to generate a preprocessed audio signal. The method may include preconditioning the audio signal for machine perception by reducing echo residual to generate a preconditioned audio signal. The echo residual of the preconditioned audio signal is higher than the echo residual of the preprocessed audio signal. The method may include performing machine perception, including automatic speech recognition, on the preconditioned audio signal to generate a machine perception output. In other words, when generating a preconditioned audio signal for machine perception, the degree of echo reduction is reduced.

[0016] In the method, pre-processing the audio signal for human perception may include pre-processing according to a first echo cancellation parameter. In addition, pre-conditioning the audio signal for machine perception may include pre-processing the audio signal according to a second echo cancellation parameter. The second echo cancellation parameter may have a convergence smaller than the convergence of the first echo cancellation parameter. More specifically, the first echo cancellation parameter may have a convergence of 100 to 200 ms, and the second echo cancellation parameter may have a convergence of less than 50 ms. For example, the first echo cancellation parameter may correspond to a first echo amount less than -60 dB, and the second echo cancellation parameter may correspond to a second echo amount of -40 to -20 dB.

[0017] According to another embodiment, a non-transitory computer-readable medium stores a computer program that, when executed by a processor, controls an apparatus to perform a process including one or more of the above methods.

[0018] According to another embodiment, an apparatus processes audio for machine perception. The apparatus includes a processor and a memory. The processor is configured to control the apparatus to receive an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The processor is configured to control the apparatus to precondition the audio signal according to machine perception parameters to generate a preconditioned audio signal, wherein the machine perception parameters are different from human perception parameters. The processor is configured to control the apparatus to perform machine perception, including automatic speech recognition, on the preconditioned audio signal to generate a machine perception output.

[0019] The apparatus may include similar details as discussed herein with respect to the method.

[0020] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations.

[0021] According to yet another aspect, an apparatus for processing audio for machine perception is described. The apparatus may include a processor and a memory. The processor may be configured to control the apparatus to receive an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The processor may be configured to control the apparatus to pre-condition the audio signal by adding noise to the audio signal to generate a pre-conditioned audio signal. The processor may be configured to control the apparatus to perform machine perception, including automatic speech recognition, on the pre-conditioned audio signal to generate a machine perception output.

[0022] The added noise may have characteristics of pink or white noise.The homogeneity of the noise field of the preconditioned audio signal may be increased compared to the homogeneity of the noise field of the audio signal.

[0023] According to another aspect, a device for processing audio for human perception and for machine perception is proposed. Similarly, the device may include a processor and a memory. The processor may be configured to control the device to receive an audio signal, wherein the audio signal corresponds to audio that has been captured by a device. The processor may be configured to control the device to pre-process the audio signal for human perception by reducing echo residual to generate a pre-processed audio signal. The processor may be configured to control the device to pre-condition the audio signal for machine perception by reducing echo residual to generate a pre-conditioned audio signal, wherein the echo residual amount of the pre-conditioned audio signal is higher than the echo residual amount of the pre-processed audio signal. The processor may be configured to control the device to perform machine perception, including automatic speech recognition, on the pre-conditioned audio signal to generate a machine perception output.

[0024] The processor may be configured to control the device to pre-process the audio signal for human perception by pre-processing according to the first echo cancellation parameter. The processor may be configured to control the device to pre-adjust the audio signal for machine perception by pre-adjusting according to the second echo cancellation parameter. The second echo cancellation parameter may have a convergence smaller than the convergence of the first echo cancellation parameter. The first echo cancellation parameter may have a convergence of 100 to 200 ms, and the second echo cancellation parameter may have a convergence of less than 50 ms. Here, the first echo cancellation parameter may correspond to a first echo amount less than -60 dB, and the second echo cancellation parameter may correspond to a second echo amount of -40 to -20 dB. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 is a block diagram of the audio processing system 100 .

[0026] Figure 2 is a block diagram of the audio processing system 200 .

[0027] Figure 3 is a block diagram of an audio processing system 300 .

[0028] Figure 4 is a block diagram of the pre-conditioner 400 .

[0029] Figure 5 is a flow chart of a method 500 for machine-perceptual pre-conditioning audio.

[0030] Figure 6 is a block diagram of an audio processing system 600 .

[0031] Figure 7 is a block diagram of the pre-conditioner 700 . DETAILED DESCRIPTION

[0032] This article describes techniques for machine-aware pre-conditioning of audio. In the following description, for the purpose of explanation, many examples and specific details are set forth to provide a thorough understanding of the present disclosure. However, it is apparent to those skilled in the art that the present disclosure, as defined by the claims, may include some or all of the features in these examples alone, or in combination with other features described below, and may also include modifications and equivalents of the features and concepts described herein.

[0033] In the following description, various methods, processes, and procedures are described in detail. Although specific steps may be described in a certain order, such order is primarily for convenience and clarity. Certain steps may be repeated more than once, may occur before or after other steps (even if such steps are otherwise described in another order), and may occur in parallel with other steps. A step may be performed after a first step only if the first step must be completed before the second step can be started. This will be specifically noted where the context does not clearly indicate otherwise.

[0034] In this document, the terms "and", "or" and "and / or" are used. These terms should be understood to have an inclusive meaning. For example, "A and B" may at least mean the following: "both A and B", "at least both A and B". As another example, "A or B" may at least mean the following: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and / or B" may at least mean the following: "A and B", "A or B". When an exclusive or is intended, it will be specifically noted (e.g., "either A or B", "at most one of A and B").

[0035] Overview

[0036] As will be discussed in more detail below, many existing audio processing systems are tuned to sound pleasant to human listeners (e.g., pre-processing the audio before transmission, etc.). Examples of such processing parameters for human perception include reducing echo residual and lowering background noise. Compared to these processing parameters for human perception, the embodiments discussed herein relate to techniques for improving the performance of machine perception systems (e.g., automatic speech analysis systems based on machine learning algorithms, etc.).

[0037] Allowing more echo leakage and background noise is beneficial to the performance of such machine perception systems, even though it may sound unpleasant to human listeners. Removing background noise in communication systems is believed to be beneficial to human perception, but removing background noise may result in missing syllables in speech signals, misleading machine learning algorithms when inferring content and further deteriorating the performance of automated speech analysis systems.

[0038] In general, the techniques described herein provide a multi-layered approach (referred to as "preconditioning") to improve processing parameters of automated speech analysis systems in the presence of echo and noise. This multi-layered processing framework allows for more echo residual and background noise, and adds masking noise, compared to systems designed for human perception, so that the captured signal is understandable and predictable for machine perception systems.

[0039] Human Perception of Audio

[0040] Audio processing systems designed for human communication generally include components that take the signal from a microphone (usually in the presence of noise, which may come from the room or from the actual device containing the microphone) and convert it into clean speech that another person can listen to. Such audio processing components can generally be referred to as "pre-processing" components. These pre-processing components include techniques such as echo cancellation, echo suppression, noise suppression, noise gating, and automatic gain control (AGC).

[0041] Echo cancellation generally operates to remove the output of a device that includes a microphone (e.g., the device includes a loudspeaker, whose output is also detected by the microphone). Echo suppression generally operates to compensate for the inability of the echo canceller to completely eliminate the echo. Noise suppression generally operates on stationary noise or non-stationary noise in the background (e.g., fans, traffic, typing noise, etc.). When no speech is occurring at all (e.g., as detected by a voice activity detector), noise gating can be used to completely mute the signal. AGC generally applies gain to increase or decrease the speech level to a predefined level.

[0042] None of these techniques are mathematically fixed; they all have controllable parameters that will be set for a specific use case (usually duplex voice communications). How they are configured will be influenced by the preferences of the vendor and / or tuner. In general, however, preprocessing systems are tuned so that human perception metrics are improved (or at least not adversely affected to an unacceptable degree). For example, people experience their own echo in conversation as detracting from the ability to convey the message. The echo cancellers and suppressors in the preprocessing system will be tuned to never allow any echo to the voice communication system. There are always trade-offs in the tuning process, and this strict echo-free strategy may come at the expense of occasionally truncating the beginning of a word or attenuating parts of the signal when the system has low confidence that it contains an echo.

[0043] As another example, people in a duplex communication link may be bothered by background noise. In a communication call, multiple people are transmitting their background noise, which further affects understanding. Noise gating is often implemented in pre-processing systems to completely mute the audio when no speech is detected in order to mitigate background noise (and also reduce bandwidth requirements on the communication link).

[0044] Machine perception of audio

[0045] Speech recognition systems also benefit from preprocessing to improve their performance. For example, by removing many non-speech sounds from the signal prior to the ASR, we do not need to include superfluous training data in the ASR to effectively make it an echo canceller or AGC. We would argue that the cost of learning a network to perform these functions and the cost of collecting large amounts of training data to tune the network greatly outweighs the cost of doing the preprocessing - turning off preprocessing is not an option. We argue that a system designed to facilitate human-to-human communication may well be inadvertently tuned to reduce the performance of a human-to-machine system.

[0046] Given that the term "preprocessing" is commonly associated with duplex human-to-human communications, we will use the term "preconditioning" to refer to systems that have been designed as the front end of a speech detection or speech recognition system. Example techniques in preprocessing systems may include noise injection, relaxed echo cancellation, and relaxed echo suppression.

[0047] Noise injection generally adds noise (either broadband or within a single subband) when the signal drops below a certain level. The added noise adds ambiguity to the ASR. (Such an operation may be contrasted with preprocessing for human perception, where noise is reduced rather than added.) Furthermore, for an ASR, making the noise disappear completely represents a situation that may never occur in its training set.

[0048] Relaxed echo cancellation and suppression is implemented in pre-conditioning systems (as opposed to systems that use pre-processing for human perception). For example, an ASR may not care at all if there is an occasional echo in the signal, but may have a problem if the beginning of a word is cut off. Such operations can be contrasted with pre-processing for human perception, where ensuring that echo is removed is particularly important (e.g., in a duplex human-to-human communication system). For machine perception, the pre-conditioner allows more echo, thereby enabling more real speech to enter the ASR.

[0049] Many pre-conditioning system components can be viewed as pre-processing system components that change intent using different tuning strategies in order to favor the behavior of the ASR. Furthermore, a pre-conditioning system may not necessarily include an echo canceller - it may exist after the codec purely to enhance the corrupted audio before feeding it to the ASR.

[0050] In general, "preconditioning" refers to the application of signal processing prior to ASR to improve its performance. Preconditioning can take input from a wide variety of sources, including transmitted audio signals, digitally stored audio signals, microphone audio signals, recorded signals from analog storage media such as cassette tapes, and so on. A preconditioning system can also be used as a component of a duplex communication system (e.g., a system that also includes a preprocessing system). It can receive one or more of the following signals, such as a near-end microphone feed, a near-end preprocessed microphone feed (i.e., speech that has been preprocessed for human perception), a far-end microphone feed, other sources, and so on.

[0051] Therefore, there may be similarities between pre-conditioning systems and pre-processing systems. For example, both a local microphone and a local speaker output from a system may be fed in as inputs. Typically, an echo canceller will use these to try to remove localized talk from the captured signal.

[0052] The Difference Between Human and Machine Perception of Audio

[0053] Although similar processing operations can be used when processing audio for both human perception and machine perception, there can be differences in the types of processing operations performed and the parameters used in each processing operation. The following is an example of the differences between human perception and machine perception.

[0054] Human perception is affected by the cognitive load of the communication, while machine perception is adaptable to difficult communications. For example, a noisy or intermittent communication channel can be tiring for a human listener, but the processing power of a machine can be increased to overcome this problem. Human perception is affected by the emotional state of the listener, but machine perception has little self-awareness. Human perception is easily distracted by stimuli outside of the communication, but machine perception focuses on the specific input being analyzed. Human perception is "stateful" and uses years of experience to add context to the communication, but machine perception can have finite state (or even zero state).

[0055] Human perception is greatly affected by self-echo, but machine perception can easily ignore self-echo. For humans, echoes greater than 50 milliseconds must be attenuated to the order of 60dB for normal communication.

[0056] Human perception involves coupled speech and listening systems, but machine perception can be achieved using decoupled speech and listening systems. Speaking and listening at the same time is difficult for humans, but it may be easy for machines.

[0057] Human perception generally determines meaning based on a single pass, but machine perception can determine meaning by using multiple passes to evaluate information. For example, humans generally hear a sound and deduce the meaning from it, but a machine can store a verbatim copy of the sound and evaluate it multiple times (e.g., using a successively more computationally intensive evaluation process).

[0058] Adjustments for machine perception

[0059] Given the above differences between human perception and machine perception, many adjustments can be made to systems targeting machine perception (compared to those used in systems targeting human perception). As mentioned above, these adjustments can be referred to as "preconditioning" (to distinguish them from "preprocessing" used in systems targeting human perception). Three forms of preconditioning are described in further detail below: noise suppression, speech inactivity detection, and echo scrambling.

[0060] Noise Suppression

[0061] As mentioned above, human perception is negatively affected by noise much more than machine perception. To address this noise issue, preprocessing for human perception includes significant noise reduction. This may result in truncation of the front of the speech. (Truncation to reduce noise is an acceptable tradeoff for human perception, as humans will infer information based on the overall situation and context to fill in the truncated sound.)

[0062] In contrast, machines don’t respond well to silence because they may try to infer information where it doesn’t exist. (Machines are much less able to use the whole picture and context to infer missing information.) To address this, preconditioners for machine perception inject higher levels of noise (higher than the target noise level when preconditioning for human perception). Injecting noise introduces ambiguity into the ASR in situations where the preconditioner is confident that speech is not present. This has the benefit that the system allows the noise suppression component to add more junk bits of the signal to the audio (because it will be masked by the noise) to increase the variation that the preconditioner doesn’t cut off the front end of speech.

[0063] In short, humans need voices that are much louder than background noise. Machines don’t.

[0064] Voice inactivity detection

[0065] In preprocessors for human perception, voice inactivity detection is typically implemented using a low-latency activity detector that applies gain to the audio signal. When the activity detector detects voice activity, the gain is close to 1, and the audio signal passes through the other components of the system (e.g., transmitter, etc.). When the activity detector does not detect voice activity, the gain is quickly dropped to 0 to gate out any background noise in the audio signal before providing the modified audio signal to other components. Since most preprocessing for human perception is implemented in communication systems, low latency is an important consideration so that speech is detected quickly and allowed to pass, and the signal is gated out when speech stops to prevent noise smearing.

[0066] In contrast, in preconditioners for machine perception, low latency is not a typical requirement. The activity detector in the preconditioner includes delay and holdover components to add some look-ahead, ramp-in, and ramp-out to reduce glitches entering the ASR that may be misidentified as transient-like phonemes (e.g., "ti", "ka", etc.). Because of the additional context in the activity detector, it may be more sensitive (compared to the case of human perception) to reduce the possibility of clipping the front end of the speech.

[0067] Echo Scrambling

[0068] Preprocessors targeting human perception are typically targeted at duplex communication use cases and include an echo canceller and an echo suppressor. The echo canceller is tuned to smoothly adapt the incoming audio signal (e.g., speech) to prevent spurious background noise from changing the behavior of the echo canceller and causing strange artifacts in the audio signal. This smooth adaptation results in slower adaptation of all echo paths, even the primary echo path that is targeted for cancellation. Due to system nonlinearities, the echo canceller cannot cancel all echoes; at best, the echo canceller can only cancel about 20 dB of self-echo. For duplex communication, 60 dB of self-echo cancellation is typically required to achieve acceptable performance for human perception, so the preprocessor uses an echo suppressor to remove the unwanted echo remaining after the echo canceller.

[0069] In contrast, in a preconditioner for machine perception, echo cancellation may be performed much faster than the smooth adaptation used in preconditioning for human perception. This faster echo cancellation may result in reference coupling and artificial echo bleed-through, which are acceptable in machine perception but unacceptable in human perception. Furthermore, for machine perception, echo suppression may be performed using heavier suppression gain smoothing than those used in preconditioning for human perception. This heavier suppression gain smoothing results in a whiter noise spectrum, which is acceptable in machine perception but unacceptable in human perception.

[0070] In summary, these modifications to echo processing in machine-aware preconditioners may be referred to as echo scrambling (rather than echo cancellation or echo suppression).

[0071] Figure 1 1 is a block diagram of an audio processing system 100. The audio processing system 100 includes a processor 102, a memory 104, a microphone 106, a loudspeaker 108, one or more other components 110, and a bus 112. The audio processing system 100 may include other components that are not shown (for simplicity). The audio processing system 100 generally performs preconditioning of an audio signal 120, performs machine perception on the preconditioned audio signal, and generates a machine perception output 130.

[0072] The processor 102 generally controls the operation of the audio processing system 100. The processor 102 may execute one or more computer programs that implement the various signal processing and preconditioning functions described herein.

[0073] The memory 104 generally stores data used by the processor 102 as part of operating the audio processing system 100. The memory 104 may store computer programs executed by the processor 102 and data processed by the processor 102. The memory 104 may store data corresponding to the audio signal 120, the preconditioned audio signal, the machine perception output 130, and any intermediate data used in processing these signals.

[0074] The microphone 106 generally receives audio input. The audio input may correspond to an audio signal 120. Alternatively, the audio signal 120 may be stored by the memory 104 or received by one of the other components 110 (eg, a receiver component).

[0075] The microphone 108 can output an audio output. For example, the memory 104 can store the audio data output by the microphone 108, or one of the other components 110 (eg, a receiver component) can receive the audio data output by the microphone 108.

[0076] Other components 110 generally provide additional functionality for the audio processing system 100. Other components 110 may include input components (eg, wired receivers, wireless receivers, etc.), output components (eg, wired transmitters, wireless transmitters, etc.), or combined input / output components.

[0077] The bus 112 generally connects the components of the audio processing system 100. Although shown as a single bus, the bus 112 may be configured as multiple interconnected buses.

[0078] The audio processing system 100 can be implemented in various types of devices. One example device is a smart speaker. A smart speaker is a wireless speaker and voice command device with an integrated virtual assistant that provides interactive actions and hands-free activation with the help of a "hot word" (or several "hot words"). Some smart speakers can also act as smart devices that utilize Wi-Fi, Bluetooth, and other wireless protocol standards to extend uses beyond audio playback, such as controlling home automation devices. Smart speakers use machine perception to recognize "hot words" or for other ASR purposes. Example smart speakers include the Amazon Echo TM Devices, Apple HomePod TM Devices and Google Home TM equipment.

[0079] Another example device is a mobile phone. The mobile phone includes the audio processing system 100 to perform pre-conditioning for machine perception, such as when the mobile phone implements a voice-controlled virtual assistant. Since the mobile phone also performs a duplex communication function, the audio processing system 100 also performs pre-processing for human perception. The processing parameters of the audio processing system 100 for pre-conditioning are different from the processing parameters for pre-processing.

[0080] Further details of the audio processing system 100 are provided below.

[0081] Figure 2 2 is a block diagram of an audio processing system 200. The audio processing system 200 may be composed of the audio processing system 100 (see Figure 1 ) is implemented, for example, by the processor 102. The audio processing system 200 includes a pre-conditioner 210 and an ASR component 220.

[0082] The preconditioner 210 receives the audio signal 120 (see Figure 1 ), performs preconditioning of the audio signal 120 according to the machine perception parameter and generates a preconditioned audio signal 230. The machine perception parameter is different from the human perception parameter. For example, the machine perception parameter may be noise injection, which is different from noise reduction for human perception. As another example, the machine perception parameter may be a delay or retention in an activity detector, which is different from a low-latency activity detector for human perception. As another example, the machine perception parameter may be echo scrambling, which is different from echo cancellation or echo suppression for human perception.

[0083] The ASR component 220 performs machine perception on the preconditioned audio signal 230 to generate a machine perception output 130 (see Figure 1 ). Machine perception can be automatic speech recognition, wake-up word detection, speech-to-text processing, speaker identification, emotion detection, language detection, etc.

[0084] The device implementing the audio processing system 200 can then perform an action in response to the machine perception output 130. For example, when a wake-up word is detected, the device can perform a wake-up operation. When performing speech-to-text processing, the generated text can be stored, sent, or displayed.

[0085] Figure 3 3 is a block diagram of an audio processing system 300. The audio processing system 300 may be composed of the audio processing system 100 (see Figure 1 ) is implemented, for example, by the processor 102. The audio processing system 300 may be a component of a smart speaker. The audio processing system 300 includes a loudspeaker 310, a microphone 320, an echo canceller 330, a pre-conditioner 340, and an ASR component 350.

[0086] Microphone 310 may be similar to microphone 108 (see Figure 1 ). The loudspeaker 310 outputs a loudspeaker output 360 based on the loudspeaker signal 362. For example, when the audio processing system 300 is implemented in a smart speaker, the loudspeaker output 360 may correspond to a music selection.

[0087] Microphone 320 may be similar to microphone 106 (see Figure 1 ). Microphone 320 captures audio 363 and generates a captured audio signal 364, which includes an echo component and a capture component. The echo component corresponds to the loudspeaker output 360 that has been output by the loudspeaker 310, and the capture component corresponds to an audio source other than the loudspeaker 310. For example, the capture component may be generated by a person speaking in the same room as the audio processing system 300.

[0088] The echo canceller 330 performs echo cancellation on the captured audio signal 364 using the microphone signal 362 to generate the audio signal 120 (see Figure 1 ).

[0089] Pre-conditioner 340 may be similar to pre-conditioner 210 (see Figure 2 ), and preconditioning is performed on the audio signal 120 in a manner similar to that described above with respect to the audio processing system 200 to generate a preconditioned audio signal 230.

[0090] ASR component 350 may be similar to ASR component 220 (see Figure 2 ), and performs machine perception on the preconditioned audio signal 230 (using the loudspeaker signal 362 as a reference signal) in a manner similar to that described above with respect to the audio processing system 200 to generate the machine perception output 130 .

[0091] Figure 4 400. The preconditioner 400 may be provided by the audio processing system 100 (see Figure 1 ) is implemented, for example, by the processor 102. The pre-conditioner 400 may be used as the pre-conditioner 210 (see Figure 2 ), pre-regulator 340 (see Figure 3 ) etc. The preconditioner 400 includes a cleaning component 402 , a noise filler 404 , an echo scrambler 406 , and an inactivity detector 408 .

[0092] The cleaning component 402 receives the audio signal 120 (see Figure 1-3), performs cleaning on the audio signal 120, and generates a cleaned audio signal 420. The cleaning component 402 generally operates as a weak pre-processor. The cleaning component 402 includes an echo canceller 410 and an echo suppressor 412. The echo canceller 410 performs echo cancellation on the audio signal 120 using a reference signal 421, and then the echo suppressor 412 performs echo suppression on the signal obtained by the echo cancellation. For example, the reference signal 421 may correspond to the loudspeaker signal 362 (see Figure 3 ).

[0093] Compared to a preprocessor for human perception, the cleaning component 402 can have many differences. The cleaning component 402 can have no gain control or level adjustment. The echo canceller 410 can perform much faster echo cancellation, so that this can cause reference coupling and artificial echo leakage in the cleaned audio signal 420. The echo suppressor 412 can perform much less echo suppression, so that the burst leakage and residual echo after the canceller remain in the cleaned audio signal 420. The cleaning component 402 can perform much less noise suppression than the human listener would otherwise need. The cleaning component 402 can perform less regularization. (Regularization is a post-filtering of the gain coefficients in the preprocessor, which makes the algorithm sound more natural, but may cause information loss at the beginning and end of speech.) In other embodiments, the cleaning component 402 can be omitted from the preconditioner 400.

[0094] Additional details are provided when discussing the echo cancellation functionality of the cleaning component 402. The general design of an echo canceller involves an adaptive prediction system in which a time-varying filter attempts to predict how much of the signal presented at the microphone is to be provided to the speaker based on a reference signal. In the case of a multi-channel system with more than one output speaker, such a system is known to degrade and suffer from local minima in the adaptation because multiple output signals from the speakers are combined at a single microphone where the signal is to be predicted. Ideally, the filter should reach a steady state representing a set of transfer functions from the speakers to the microphones, leaving a residual echo that approximates a similar time envelope and decaying version of the echo presented at the microphone. At any time, the adaptation rate can be increased in order to more aggressively find a good prediction that reduces the energy in the residual signal. However, when the tuning is too aggressive, two problems will occur, the system will track into spatial degradation more quickly (this will occur when some speaker outputs are correlated), and the adaptation filter will also find spurious temporary correlations between the output signal (reference) and the incoming signal component in the microphone representing the desired speech to be detected. The first mechanism will cause a sudden jump in echo leakage when spatial steering of the output on multiple channels occurs. The second mechanism will introduce a coloration of the desired signal which amounts to a mix of versions of certain components of the reference.

[0095] As is well known in the art, tuning of an echo canceller is a delicate balance between adaptation speed and the ability to prevent erroneous adaptation to a desired signal (local talk). In general, the time constant used for conventional echo cancellation corresponds to a convergence (adaptation) time of about 100 ms or more. Since the detection of local talk may be erroneous for several frames of audio, typically 5-20 ms frames, the adaptation is deliberately slowed down when a local signal is present to avoid adaptation divergence. However, in the case of ASR, we can tolerate temporary instability of the adaptation filter to a greater extent. This divergence or instability can cause short-term frame leakage. These are very problematic for the user, but can be effectively ignored by a properly trained ASR. In the case of some embodiments of the echo canceller 410, we suggest that adaptation times in the range of 10-50 ms are applicable and more suitable for ASR, and in one embodiment, an adaptation range of 20 ms is suitable.

[0096] In other respects, the echo canceller 410 can be similar to other echo cancellers discussed herein (e.g., Figure 3 echo canceller 330, etc.).

[0097] The noise filler 404 generates the noise 422 according to the noise parameter, and the preconditioner 400 adds the noise 422 to the cleaned audio signal 420 to generate the preconditioned audio signal 230 (see Figure 2-3 ). The noise 422 increases the uniformity of the noise field of the preconditioned audio signal 230 compared to the cleaned audio signal 420. The noise 422 may be applied to the cleaned audio signal 420 when the signal drops below a certain level across the entire frequency band or within a single sub-band. The noise 422 adds ambiguity to the preconditioned audio signal 230.

[0098] The concept of comfort noise was largely introduced in the prior art as a means of inserting low level noise fill during periods of complete inactivity, in some cases where the transmission system might otherwise not be transmitting or be completely silent. The comfort noise system switches between the decoded signal and the generated low level noise. Typical comfort noise levels are generated and presented at levels more than 20dB below the average speech level when viewed from a long term spectrum. The shape of the ideal comfort noise is similar to the spectrum of speech or background room noise, with one specification being known as Hoth noise. It represents the level of noise in a typical quiet room and is generally considered detectable but not significant to the average listener. In some cases such noise is added continuously, but at a level that appears natural to speech conversations.

[0099] Compared to the long term average of speech and the typical noise floor in the recording, we can see a difference of about 20 dB in the long term spectrum, which represents the signal to noise ratio of speech activity, given the dynamic range and burstiness of speech, which ranges between 20-35 dB. Typically in noisy environments, a noise suppressor is needed to suppress the noise to achieve this desired goal.

[0100] In this example, we propose a noise level that is only about 10-15 dB below the root mean square (RMS) speech level. To achieve this, we typically need to add a certain amount of noise, a process we call noise filling or masking noise. This masking noise can be added to the speech continuously without any detection or gating around the presence and detection of speech. Masking noise has the characteristics of pink or spectral white noise, with a spectral shape similar to the profile of Hoth noise, except at a level 10dB or more higher than Hoth noise. This noise level in listening tests has been found to be intrusive, artificial, and an overall degradation to the human listening experience. However, the addition and blurring of other lower level noise that may be present in the recording, or artifacts that may be caused by noise suppression, has a net positive impact on the performance of automatic speech recognition.

[0101] In one embodiment, the noise filling is a constant addition of a random noise signal with spectral shaping as described above.

[0102] In other embodiments, we introduce the concept of a filling method that adds spectrally shaped noise such that at any point in time, the spectral output of the system does not fall below this target spectral shape. By creating a target output level that is 10-15 dB higher than in traditional speech and coding systems with traditional comfort noise potential, this process is more effective in blurring out any background suppressed noise and artifacts.

[0103] In this detail, we see that the noise filler 404 adds or fills in noise levels that are as much as 20-30 dB higher than the noise levels considered for well-conditioned audio for human listening. This aspect of pre-conditioning was unexpected and was found to be beneficial for automatic speech recognition.

[0104] The echo scrambler 406 receives the reference signal 421 and blurs the echo in the reference signal 421 according to the suppression gain parameter to generate a blurred reference signal 423, and the preconditioner 400 adds the blurred reference signal 423 to the cleaned audio signal 420 to generate the preconditioned audio signal 230. The suppression gain parameter increases the smoothing of the suppression gain and increases the whiteness of the noise spectrum of the blurred reference signal 423 (compared to the noise spectrum of the reference signal 421). The blurred reference signal 423 adds the echo to the cleaned audio signal 420 that has been removed by the cleaning component 402. This process of smoothing the gain helps to reduce excessive variations in the time or spectral response that may result from attempting to separate or completely remove perceived noise. This blurring may also be referred to as "regularization" and may be implemented as described in U.S. Patent No. 10,311,891 or U.S. Patent No. 8,712,076.

[0105] Note that the noise filler 404 and the echo scrambler 406 are additive (e.g., using the signal combiner 430) after the cleaning component 402 because they operate to fill back the signal and energy. However, this added signal is in a random or randomized form. One view of this is that the noise filler 404 and the echo scrambler 406 help propagate ambiguity by replacing something in the part that the cleaning component 402 may have stripped or suppressed in the cleaned audio signal 420. The cleaned audio signal 420 output from the cleaning component 402 may produce a sense of false belief that the machine perception system does not exist. That is, if the cleaning component 402 suppresses the signal passing through it, it may be displayed downstream that something may be missing at the input (e.g., microphone). However, preprocessing generally designed for human perception may attempt to hide or mask problems (residual echoes), and therefore remove the possibility of the presence of certain audio signals-in this sense, the system has adjusted the ambiguity (whether there is any signal after the echo) to certainty (the cleaned audio signal 420 is suppressed and / or muted). The addition of the noise filler 404 and the echo scrambler 406 is a means of adding background noise that covers most of the conditions where the pre-processing system may have introduced processing artifacts (such as suppressed audio fragments). This idea of ​​adding noise to properly propagate uncertainty into downstream machine perception is a noteworthy feature of the embodiments described herein, and can be seen to be quite different and contradictory to many existing systems, as we show and suggest by example embodiments where pre-conditioning for operation and ideally training will add a noise component that (paradoxically) serves to improve the overall performance of the machine perception system.

[0106] The inactivity detector 408 receives the audio signal 120, performs activity detection on the audio signal 120 according to a delay parameter and a gating parameter, and generates a gain 424, which the preconditioner 400 applies to the cleaned audio signal 420 (e.g., using a mixer 440) to generate the preconditioned audio signal 230. The delay parameter introduces a delay when preconditioning the audio signal 120, and the gating parameter introduces a ramp-in and ramp-out when preconditioning the audio signal 120. The inactivity detector 408 is more sensitive than an inactivity detector for human perception, and uses more delay and more look-ahead (according to the delay parameter) to apply more aggressive gating (according to the gating parameter).

[0107] Figure 5 is a flow chart of a method 500 for machine-perceptually processing audio. One or more steps of the method 500 may be performed by the audio processing system 100 (see Figure 1 ), audio processing system 200 (see Figure 2 ), audio processing system 300 (see Figure 3 ), pre-regulator 400 (see Figure 4 ) is executed, for example, by a processor executing one or more computer programs.

[0108] At 502, an audio signal is received. The audio signal corresponds to audio that has been captured by a device. For example, the audio signal may correspond to an audio signal 120 captured by a microphone 106 (see Figure 1 As another example, the audio signal may correspond to data representing audio that has been stored in the memory 104 or has been captured by a device other than the audio processing system 100 .

[0109] At 504, the audio signal is preconditioned according to the machine perception parameters to generate a preconditioned audio signal. The machine perception parameters are different from the human perception parameters. For example, the preconditioner 400 (see Figure 4 ) can perform preconditioning of the audio signal 120 to generate the preconditioned audio signal 230 according to parameters controlling one or more of the cleaning component 402, the noise filler 404, the echo scrambler 406, and the inactivity detector 408. As described above for each component, the parameters used by the preconditioner 400 are different from the parameters for the human perception system.

[0110] At 506, machine perception is performed on the preconditioned audio signal to generate a machine perception output. For example, the ASR component 220 (see Figure 2) can perform machine perception operations on the pre-conditioned audio signal 230, such as automatic speech recognition, wake-up word detection, speech-to-text processing, emotion detection, etc. Based on the machine perception output, the device implementing method 500 can perform other related actions, such as waking up, outputting text, implementing commands provided via voice, and so on.

[0111] Figure 6 600. The audio processing system 600 may be composed of the audio processing system 100 (see Figure 1 ) is implemented, for example, by the processor 102. The audio processing system 600 is similar to the audio processing system 300, or can be regarded as the audio processing system 300 (see Figure 3 ). The audio processing system 600 receives audio from a microphone (eg, Figure 1 The captured audio signal 664 of the microphone 106) and provided to the loudspeaker (e.g., Figure 1 The audio processing system 600 includes a beamformer 610, an echo canceller 630, a preconditioner 640, and an ASR component 650.

[0112] The beamformer 610 receives the captured audio signal 664, performs beamforming on the captured audio signal 664, and generates a beamformed audio signal 665. In general, the beamformer 610 combines signals from multiple microphones (not shown) into another spatial signal representation containing fewer channels, so the beamformer 610 can be considered an optional (or additional) component. Although the beamformer 610 is described as performing beamforming, the beamformer 610 can include other linear elements that can process the captured audio signal 664 as an alternative or in addition to beamforming.

[0113] The echo canceller 630 receives the beamformed audio signal 665, the echo cancellation parameters 635 from the preconditioner 640, and the microphone signal 662. According to the echo cancellation parameters 635, the echo canceller generally performs echo cancellation by using the microphone signal 662 to cancel the echo from the microphone and captured by the microphone in the beamformed audio signal 665. The echo cancellation parameters 635 enable the echo cancellation to be adapted according to the results of the preconditioner 640. For example, the echo cancellation parameters can adjust the echo cancellation parameters over time, such as by freezing or resetting the adaptation. Compared with the echo cancellation of the preprocessor for human perception, the echo canceller 630 adapts faster (e.g., an adaptation time of 20-50 ms). In addition, compared with the echo cancellation of the preprocessor for human perception, the echo canceller 630 may not have a double talk detector, or its double talk detector may be less sensitive (e.g., only -10 dB). The echo canceller 630 generates an audio signal 666 (see Figure 1 Alternatively, the echo canceller 630 may be similar to one or more other echo cancellers discussed herein (e.g., Figure 3 Echo canceller 330, Figure 4 echo canceller 410, etc.).

[0114] The preconditioner 640 receives the audio signal 666 and the loudspeaker signal 662, performs preconditioning on the audio signal 666, and generates a preconditioned audio signal 667 (see Figure 2 The preconditioner 640 also generates echo cancellation parameters 635 based on the preconditioning results. Compared to the preprocessor (which generally targets human perception), the preconditioner 640 targets machine perception and contains different nonlinear and time-varying additive elements. In other aspects, the preconditioner 640 can be similar to one or more other preconditioners discussed herein (e.g., Figure 2 Pre-regulator 210, Figure 3 Pre-regulator 340, Figure 4 pre-conditioner 400, etc.).

[0115] The ASR component 650 is similar to that described above (e.g., with respect to Figure 2 The audio processing system 200, Figure 3 The ASR component 650 may be configured to perform machine perception on the preconditioned audio signal 667 in a manner similar to the audio processing system 300 of the present invention to generate the machine perception output 130. Alternatively, the ASR component 650 may be configured to perform machine perception on the preconditioned audio signal 667 in a manner similar to the audio processing system 300 of the present invention to generate the machine perception output 130. Figure 2 ASR component 220, Figure 3 ASR component 350, etc.).

[0116] Figure 7 700. The preconditioner 700 may be provided by the audio processing system 100 (see Figure 1 ) is implemented, for example, by the processor 102. The pre-conditioner 700 can be used as the pre-conditioner 210 (see Figure 2 ), pre-regulator 340 (see Figure 3 ) etc. Alternatively, pre-conditioner 700 may be similar to pre-conditioner 400 (see Figure 4 ). The preconditioner 700 receives the audio signal 120 (see Figure 1-3 ), performs preconditioning, and generates a preconditioned audio signal 230 (see Figure 2-3 ). The preconditioner 700 includes a delay element 702 , a noise predictor 704 , an echo predictor 706 , a combiner 708 , a mixer 710 , a voice activity detector (VAD) 712 , a mixer 714 , an echo scrambler 716 , and a noise filler 718 .

[0117] The delay element 702 delays the audio signal 120, producing a delayed audio signal 730. The delay element 702 can implement a delay between 100-200 ms. This delay allows for better noise prediction and echo prediction, because the time delay in the prediction allows for non-causal prediction. This delay is generally longer than the delay used in preprocessors for human perception applications (e.g., which are typically less than 100 ms), because in many cases, such as for duplex communication applications, longer delays are annoying to humans.

[0118] The noise predictor 704 performs noise prediction on the audio signal 120 to generate a noise prediction result 732. Generally speaking, the noise predictor 704 operates to find the level of the current background noise in the audio signal 120 by looking through a large amount of history to find the minimum level seen on the spectrum. This gives an indication of a stationary noise floor, ignoring bursts like speech.

[0119] The echo predictor 706 uses a reference signal 734 (eg, corresponding to a loudspeaker signal such as Figure 3 362 of them Figure 6 662, etc.) to perform echo prediction on the audio signal 120 and generate an echo prediction result 736. In general, the echo predictor 706 is connected to an echo canceller (e.g., Figure 3 330 of them Figure 6630, etc. in the audio signal 120) shares similarities in how it operates, but it attempts to predict the noise power within a frequency band. For example, rather than effectively trying to predict a band-limited time-domain signal, the echo predictor 706 attempts to predict the energy level in a particular frequency band given what the device is playing. The echo predictor 706 operates on the echo cancellation data - so it attempts to predict what the echo canceller missed. The echo prediction result 736 represents a prediction of the power level of the echo in the audio signal 120.

[0120] Combiner 708 and mixer 710 work together to suppress noise and echo in delayed audio signal 730 using noise prediction result 732 and echo prediction result 736. In general, combiner 708 and mixer 710 provide a limited or compressed form of delayed audio signal 730 so as to slowly move into the noise floor (as the confidence that the signal contains meaningful data approaches 0). Compared to a preprocessor for human perception, the noise suppression is much smaller, for example, about 10-15 dB below the RMS speech level (or about 10-15 dB less than the signal-to-noise ratio improvement (SNRI)). (In a preprocessor for human perception, the corresponding noise suppression is about 20-30 dB below the RMS speech level.) In addition, the echo suppression is much smaller than a preprocessor for human perception, for example, 5-10 dB below the noise level (or 15-20 dB below the sound level). (In a preprocessor for human perception, the corresponding echo suppression is approximately 50-60 dB below the speech level.) As a result of the noise suppression and echo suppression, the mixer 710 generates a modified audio signal 738 .

[0121] A voice activity detector (VAD) 712 performs voice activity detection on the modified audio signal 738 to generate a VAD result 740. The VAD 712 implements what may be referred to as a soft VAD, which has a slower time constant than the time constant in the VAD used in the preprocessor for human perception applications. (For example, the time constant in the VAD of the preprocessor for human perception may be 20-60 ms of ramp-up, while the time constant for the VAD 712 may be 100-200 ms of ramp-up.) The VAD 712 may also involve a delay of 100-200 ms (e.g., to improve detection accuracy), while the VAD used in the preprocessor for human perception applications (e.g., duplex communication) generally uses a delay of less than 100 ms. The VAD 712 is not gated at approximately 6 dB, and the attenuation is between 6-10 dB. In addition, the VAD 712 may not fade out as part of the VAD process, because the abrupt end of silence or activity and the drop in level due to the VAD operation will not be perceived as an annoyance by the machine. (In contrast, VAD in preprocessors for human perception applications is typically faded out within a few seconds as part of the VAD process.)

[0122] The mixer 714 performs mixing on the modified audio signal 738 using the VAD result 740 to generate a further modified audio signal 742 .

[0123] The echo scrambler 716 performs echo scrambling on the further modified audio signal 742 using the echo prediction result 736 and generates an echo scrambled audio signal 744. In general, the echo scrambling is added to the power of the further modified audio signal 742 in a time and frequency varying manner. The goal of the echo scrambler 716 depends on the frequency spectrum of the echo prediction result 736.

[0124] The noise filler 718 performs noise filling on the echo-scrambled audio signal 744 to generate the preconditioned audio signal 230. In general, the noise filler 718 adds noise to achieve additive filling of a target power level of the echo-scrambled audio signal 744. For example, in a specific frequency band, when the power level is below the noise target level, the noise filler 718 adds noise, and when the power level is above the noise target level, the noise filler 718 does not add noise. In addition, the noise filler 718 operates to fill a fixed spectrum, while the target of the echo scrambler 716 depends on the spectrum of the echo prediction result 736.

[0125] An example implementation of the echo scrambler 716 and the noise filler 718 is as follows. In each frequency band "band" on all frequency bands, over all intervals "f" of each frequency band, the signal power P1(f) is calculated according to the following formula:

[0126]

[0127] where P1(f) is the signal power in each bin in the band, P1 is the signal power in the band, P2(band) is the target power level for the band (defined further below), P3(band) is the signal power level for the band, z is a random scalar or complex number with unit variance and zero mean, and window(band, f) is the window function.

[0128] The z-function can be any compact random distribution as long as it is suitably stationary in time and decorrelated in frequency and time. Example functions include the normal distribution function or a complex number with a Rayleigh distribution amplitude, depending on the nature of the transform interval, whether real or complex. The z-function can be implemented using shift registers, multiplication / addition / modular random number generators with uniform distribution and unit power, etc.

[0129] Window function window (band, f) is a group of coefficients, and it is used for the original transformation set of frequency interval to be reduced to the frequency band set of interval generally on perceptual scale (such as mel scale, Bark scale or equivalent rectangular bandwidth (ERB) spacing).Some attributes of this weighting are generally the preservation of energy from interval, because the summation of window (band, f) on all frequency intervals f is constant, and is normally unified.This expression of perceptual filter bank mapping from the linear frequency interval of a large number of intervals to the frequency band of less number is known in the art.For example, in one embodiment, system is mapped downward to 61 frequency bands (the width of each frequency band is about 10% of the center frequency) from one group of 320 intervals (20ms frame under 16 kHz).

[0130] The P2(band) of the echo scrambler 716 and the noise filler 718 are different, and are respectively called P2echo(band) and P2noise(band). For the echo scrambler 716, P2echo(band) is defined according to the following formula:

[0131]

[0132] where PowerPredictedEcho(band) is the power level of the predicted echo in the band and ConstantEchoScrambleOffset(band) is the offset constant power level of the echo scrambler in the band.

[0133] For the noise filler 718, P2noise(band) is defined as follows:

[0134]

[0135] Among them, ConstantNoiseTargetProfile(band) corresponds to the constant noise target profile.

[0136] Thus, the echo scrambler 716 can be viewed as adding fuzzy noise with a similar envelope to the echo, generally masking it. The offset can be frequency dependent, typically between 5-15 dB. The noise filler 718 adds fuzzy noise to fill the signal to a given power level and can be performed after the echo scrambler 716 (e.g. Figure 7 Note that if P2(band) is less than P3(band), nothing is added.

[0137] In this way, the pre-conditioner 700 obscures the remaining speckle and noise in the audio signal 120 with occasional bursts of high and unpleasant (from a human perception perspective) noise levels, but they are generally short in time and narrow in frequency (e.g., known as "barks") and are therefore unlikely to be recorded by the ASR as meaningful phonemes or words.

[0138] Summarize

[0139] In summary, although the embodiments of the preconditioner described herein (e.g., Figure 2 Pre-regulator 210, Figure 3 Pre-regulator 340, Figure 4 Pre-regulator 400, Figure 6 Pre-regulator 640, Figure 7The preconditioner 700, etc.) has similarities to existing comfort noise generators (CNGs) or preprocessors, but has significant differences because CNGs are tuned for human perception, while preconditioners are tuned for machine perception. Specific embodiments of preconditioners may have one or more of these differences from CNGs. For example, in CNGs, echo cancellation is tuned to a convergence of 100-200 ms, and frequent double-talk detection in the voice local area is reduced from nominal to -20 dB; while in preconditioners, echo cancellation is tuned to a convergence of less than 50 ms, and double-talk detection can be as small as detecting any residual energy within 6-10 dB of the predicted echo. As another example, in CNG, echo suppression targets subaudible levels (e.g., -60 to -40 dB, or 10-20 dB below the noise level); whereas in a preconditioner, echo suppression depends on the amount of echo present, typically suppressing in the range of 0-20 dB so that the residual echo is about the same level (or slightly below) as the inserted noise fill used to mask the residual artifacts. In this way, the residual echo is targeted to be just below the noise fill, since further suppression is not necessary, and only benefits human listeners who may be distracted by hearing delayed speech-like signals deep in the noise. As another example, in CNG, noise suppression targets perceptually low levels (e.g., more than 20 dB below the speech level, or greater than 10 dB of the SNRI); whereas in a preconditioner, noise suppression targets 10-15 dB below the speech level, and typically only a small (e.g., less than 10 dB) SNRI, or in some cases no suppression at all. As another example, in CNG, suppression is applied nonlinearly to do most of the cleaning, which always reduces signal energy, while in a preconditioner, a masking noise filler works by conditionally adding noise based on the energy present in a given frequency band, and generally substantially increases the energy in the pass signal after some initial noise removal. This added noise has the characteristics of a very steady or "electrical hiss" type of noise (which can be annoying to listeners in CNG applications). As another example, in CNG, when used in communications applications, the voice activity detection response has low latency (e.g., 20-50 ms), while in a preconditioner, the voice activity detector has a latency of greater than 100 ms (or as high as 200-500 ms if immediate low latency ASR and response is not required). The system can detect speech and look back at the data to re-determine where the activity started - this is generally done by passing the audio signal through a delay path and calculating the gain and voice activity decision to apply from the older audio data. As another example, in CNG, comfort noise is either on / off or always additive; whereas in a preconditioner, the noise filler is present continuously throughout the signal (although perhaps only added to specific frequency bands when they are below the noise filler's target energy).

[0140] Implementation details

[0141] The embodiments may be implemented in hardware, executable modules stored on a computer-readable medium, or a combination of the two (e.g., a programmable logic array). Unless otherwise specified, the steps performed by the embodiments need not be inherently related to any particular computer or other device, although they may be related in certain embodiments. Specifically, various general-purpose machines may be used with programs written according to the teachings of this article, or more specialized devices (e.g., integrated circuits) may be more conveniently constructed to perform the required method steps. Therefore, the embodiments may be implemented in one or more computer programs executed on one or more programmable computer systems, each of which includes at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The program code is applied to the input data to perform the functions described herein and generate output information. The output information is applied to one or more output devices in a known manner.

[0142] Each such computer program is preferably stored on or downloaded to a storage medium or device (e.g., solid-state memory or media, or magnetic or optical media) readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by a computer system to perform the processes described herein. The system of the present invention may also be considered to be implemented as a computer-readable storage medium configured with a computer program, wherein the storage medium so configured causes the computer system to operate in a specific and predefined manner to perform the functions described herein. (Software itself and intangible or ephemeral signals are excluded as being unlicensable subject matter.)

[0143] The above description illustrates various embodiments of the present disclosure and examples of how various aspects of the present disclosure can be implemented. The above examples and embodiments should not be considered to be the only embodiments, and they are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and the following claims, other arrangements, embodiments, implementations and equivalents will be apparent to those skilled in the art and may be adopted without departing from the spirit and scope of the disclosure as defined by the claims.

Claims

1. A method for processing audio for machine perception, the method comprising: receiving an audio signal, wherein the audio signal corresponds to audio that has been captured by a device; preconditioning the audio signal according to a machine perception parameter to generate a preconditioned audio signal, wherein the machine perception parameter is different from a human perception parameter, the machine perception parameter being a second echo cancellation parameter having a convergence of less than 50 ms, the human perception parameter being a first echo cancellation parameter having a convergence of 100 to 200 ms; as well as Machine perception is performed on the preconditioned audio signal to generate a machine perception output, the machine perception comprising automatic speech recognition.

2. The method of claim 1, further comprising: outputting a loudspeaker output based on the loudspeaker signal via the loudspeaker; capturing a captured audio signal by a microphone proximate to the loudspeaker, the captured audio signal comprising an echo component and a captured component, wherein the echo component corresponds to the loudspeaker output that has been output by the loudspeaker, and wherein the captured component corresponds to an audio source other than the loudspeaker; as well as Echo cancellation is performed on the captured audio signal using the microphone signal to generate the audio signal.

3. The method of claim 1 or 2, wherein the audio signal comprises speech.

4. The method of claim 1 or 2, wherein the first echo cancellation parameter corresponds to an echo amount of less than -60 dB, and wherein the second echo cancellation parameter corresponds to an echo amount of -40 to -20 dB.

5. A method of processing audio for human perception and for machine perception, the method comprising: receiving an audio signal, wherein the audio signal corresponds to audio that has been captured by a device; preprocessing the audio signal for human perception by reducing echo according to first echo cancellation parameters to generate a preprocessed audio signal; preconditioning the audio signal for machine perception by reducing echo according to second echo cancellation parameters to generate a preconditioned audio signal, wherein an amount of echo in the preconditioned audio signal is higher than an amount of echo in the preprocessed audio signal; and performing machine perception on the preconditioned audio signal to generate a machine perception output, the machine perception comprising automatic speech recognition, The first echo cancellation parameter has a convergence of 100 to 200 ms, and the second echo cancellation parameter has a convergence of less than 50 ms.

6. The method of claim 5, wherein the first echo cancellation parameter corresponds to a first echo amount less than -60 dB, and wherein the second echo cancellation parameter corresponds to a second echo amount of -40 to -20 dB.

7. A non-transitory computer readable medium storing a computer program, which, when executed by a processor, controls an apparatus to perform a process including the method of any one of claims 1 to 6.

8. An apparatus for processing audio for machine perception, the apparatus comprising: processor; as well as Memory, wherein the processor is configured to control the apparatus to receive an audio signal, wherein the audio signal corresponds to audio that has been captured by a device; wherein the processor is configured to control the apparatus to pre-condition the audio signal according to a machine perception parameter to generate a pre-conditioned audio signal, wherein the machine perception parameter is different from a human perception parameter, the machine perception parameter being a second echo cancellation parameter having a convergence of less than 50 ms, and the human perception parameter being a first echo cancellation parameter having a convergence of 100 to 200 ms; and Wherein the processor is configured to control the apparatus to perform machine perception on the preconditioned audio signal to generate a machine perception output, the machine perception comprising automatic speech recognition.

9. The apparatus of claim 8, further comprising: a microphone configured to output a microphone output based on the microphone signal; as well as a microphone proximate to the loudspeaker, configured to capture a captured audio signal, the captured audio signal comprising an echo component and a capture component, wherein the echo component corresponds to the loudspeaker output that has been output by the loudspeaker, and wherein the capture component corresponds to an audio source other than the loudspeaker, The processor is configured to control the device to perform echo cancellation on the captured audio signal using the microphone signal to generate the audio signal.

10. An apparatus for processing audio for human perception and for machine perception, the apparatus comprising: processor; as well as Memory, wherein the processor is configured to control the apparatus to receive an audio signal, wherein the audio signal corresponds to audio that has been captured by a device; wherein the processor is configured to control the apparatus to pre-process the audio signal for human perception by reducing echo according to first echo cancellation parameters to generate a pre-processed audio signal; wherein the processor is configured to control the apparatus to precondition the audio signal for machine perception by reducing echo according to a second echo cancellation parameter to generate a preconditioned audio signal, wherein an amount of echo in the preconditioned audio signal is higher than an amount of echo in the preprocessed audio signal; as well as and wherein the processor is configured to control the apparatus to perform machine perception on the preconditioned audio signal to generate a machine perception output, the machine perception comprising automatic speech recognition, The first echo cancellation parameter has a convergence of 100 to 200 ms, and the second echo cancellation parameter has a convergence of less than 50 ms.

11. The apparatus of claim 10, wherein the first echo cancellation parameter corresponds to a first echo amount less than -60 dB, and wherein the second echo cancellation parameter corresponds to a second echo amount of -40 to -20 dB.

Images