Background noise estimation using gap confidence

Abstract

To provide background noise estimation using gap confidence.SOLUTION: A noise estimation method includes steps of generating a gap confidence value in response to microphone output and a reproduction signal and generating an estimation value of background noise in a reproduction environment by using the gap confidence value. Each gap confidence value is indicative of confidence of presence of a gap at corresponding time in the reproduction signal and may be a combination of candidate noise estimation values weighted by the gap confidence values. Generation of the candidate noise estimation value may but need not include performance of echo cancellation. Optionally, noise compensation is performed on an audio input signal by using the generated background noise estimation value. Other aspects are systems configured to perform any embodiment of the noise estimation method.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Provisional Application No. 62 / 663,302, filed April 27, 2018, and European Patent Application No. 18177822.6, filed June 14, 2018, each of which is incorporated by reference in its entirety.

[0002] Technical Field The present invention relates to a system and method for estimating background noise in an audio signal reproduction environment and processing the audio signal for reproduction (e.g., performing noise compensation on the audio signal) using the noise estimates. In some embodiments, the noise estimation includes determining gap confidence values, each of which indicates a confidence that there is a gap (at a corresponding time) in the reproduced signal, and using the gap confidence values ​​to determine a sequence of background noise estimates. [Background technology]

[0003] The ubiquity of portable electronic devices means that people engage with audio on a daily basis in many different environments: listening to music, watching entertainment content, hearing audible notifications and announcements, participating in voice calls, etc. The listening environments in which these activities occur are often inherently noisy and involve constantly changing background noise conditions that detract from the enjoyment and comprehensibility of the listening experience. Placing a user in the loop to manually adjust playback levels in response to changing noise conditions distracts the user from the listening task and increases the cognitive load required to engage in audio listening tasks.

[0004] Noise compensated media playback (NCMP) alleviates this problem by adjusting the volume of the played media to suit the noise conditions the media is being played in. The concept of NCMP is well known, and many publications claim to have solved the problem of how to implement it effectively.

[0005] While a related field called Active Noise Cancellation attempts to physically cancel interfering noise through the reproduction of acoustic waves, NCMP adjusts the level of the reproduced audio so that the adjusted audio is audible and intelligible in the playback environment in the presence of background noise.

[0006] A major challenge in real-world implementations of NCMP is to automatically determine the current background noise level experienced by a listener, especially in situations where media content is played over loudspeakers and the background noise and media content are highly acoustically coupled. Solutions involving microphones face the problem that the media content and the noise conditions are observed (detected by the microphone) simultaneously.

[0007] A typical audio playback system implementing NCMP is shown in Figure 1. The system includes a content source 1 that outputs and provides an audio signal indicative of audio content (sometimes referred to herein as media content or playback content) to a noise compensation subsystem 2. The audio signal is intended to undergo playback to generate a sound (in the environment) indicative of the audio content. The audio signal may be a speaker feed (and the noise compensation subsystem 2 may be coupled and configured to apply noise compensation to it by adjusting the playback gain of the speaker feed), or another element of the system may generate a speaker feed in response to the audio signal (e.g., the noise compensation subsystem 2 may be coupled and configured to generate a speaker feed in response to the audio signal and apply noise compensation to the speaker feed by adjusting the playback gain of the speaker feed).

[0008] The system of FIG. 1 also includes a noise estimation system 5, at least one speaker 3 (coupled and configured to emit a sound indicative of the media content) responsive to the audio signal (or a noise-compensated version of the audio signal generated by subsystem 2), and a microphone 4 coupled as shown. In operation, the microphone 4 and speaker 3 are in a playback environment (e.g., a room), and the microphone 4 generates a microphone output signal indicative of both background (ambient) noise in the environment and echoes of the media content. The noise estimation subsystem 5 (sometimes referred to herein as a noise estimator) is coupled to the microphone 4 and configured to use the microphone output signal to generate an estimate of the current background noise level in the environment (the "noise estimate" in FIG. 1). Noise compensation subsystem 2 (sometimes referred to herein as a noise compensator) is coupled and configured to apply noise compensation by adjusting the audio signal (e.g., adjusting its playback gain) in response to the noise estimate generated by subsystem 5 (or adjusting speaker feeds generated in response to the audio signal), thereby generating a noise-compensated audio signal indicative of the compensated media content (as shown in FIG. 1). Typically, subsystem 2 adjusts the playback gain of the audio signal such that sound radiated in response to the adjusted audio signal is audible and intelligible in the playback environment in the presence of background noise (estimated by noise estimation subsystem 5).

[0009] As will be described later, a background noise estimator (eg, noise estimator 5 of FIG. 1) for use in an audio reproduction system implementing noise compensation may be implemented in accordance with one class of embodiments of the present invention.

[0010] Many publications have been devoted to the issue of noise compensated media reproduction (NCMP), and audio systems that compensate for background noise can be achieved with varying degrees of success.

[0011] It has been proposed to perform NCMP without microphones, using other sensors instead (e.g., the speedometer in the case of a car). However, such methods are not as effective as microphone-based solutions that actually measure the level of interfering noise experienced by the listener. It has also been proposed to perform NCMP by relying on microphones located in an acoustic space that are separate from the sounds indicative of the reproduced content, but such methods are prohibitively restrictive for many applications.

[0012] The NCMP methods mentioned in the previous paragraph do not attempt to accurately measure the noise level using a microphone that also captures the playback content, due to the "echo problem" that arises when the playback signal captured by the microphone is mixed with the noise signal of interest to the noise estimator. Instead, these methods try to ignore the problem by constraining the compensation they apply so that an unstable feedback loop does not form, or by measuring something else that is somewhat predictive of the noise level experienced by the listener.

[0013] It has also been proposed to address the problem of estimating background noise from a microphone output signal (which represents both the background noise and the playback content) by attempting to correlate the playback content with the microphone output signal and subtracting from the microphone output an estimate of the playback content captured by the microphone (called "echo")

[0014] The content of the microphone output signal generated when the microphone captures the sound is indicative of the playback content X emanating from the speaker and background noise N, and can be expressed as WX+N, where W is a transfer function determined by the speaker(s) emitting the sound indicative of the playback content, the microphone, and the environment (e.g., the room) in which the sound propagates from the speaker to the microphone. For example, in an academically proposed method for estimating the noise N (described below with reference to FIG. 2), a linear filter W' is adapted to facilitate an estimate W'X of the echo (the playback content captured by the microphone) WX to be subtracted from the microphone output signal. Even if nonlinearity exists in the system, a nonlinear implementation of the filter W' is rarely implemented due to computational costs.

[0015] FIG. 2 is a diagram of a system for implementing the above-mentioned conventional method for estimating background noise (sometimes referred to as echo cancellation) in an environment in which a speaker emits a sound indicative of the playback content. A playback signal X is presented to a speaker system S (e.g., a single speaker) in an environment E. A microphone M is located in the same environment E. In response to the playback signal X, the speaker system S emits a sound that reaches the microphone M (along with any ambient noise N present in the environment E). The microphone output signal is Y=WX+N, where W represents a transfer function that is the combined response of the speaker system S, the playback environment E, and the microphone M. A typical method implemented by the system of FIG. 2 adaptively infers the transfer function W from Y and X using any of a variety of adaptive filter methods. As shown in FIG. 2, a linear filter W' is adaptively determined to be an approximation of the transfer function W'. The playback signal content ("echo") as indicated by microphone signal M is estimated as W'X, and W'X is subtracted from Y to generate an estimate of noise N, Y'=WX-W'X+N. An increase in Y' increases the level of X, which introduces an upward bias in the estimate of N (Y'), which in turn increases the level of X, and so on. This form of solution relies heavily on the ability of the adaptive filter W' to subtract W'X from Y in order to remove a significant amount of the echo WX from the microphone signal M.

[0016] Further filtering of the signal Y' is usually required to keep the system of Fig. 2 stable. Since most noise compensation implementations in the field show lackluster performance, most solutions typically bias the noise estimate downwards and introduce aggressive time smoothing to keep the system stable. This comes at the cost of reduced and very slow compensation action.

[0017] Conventional implementations of systems (of the type described with reference to FIG. 2) that purport to implement the above-mentioned academic method for noise estimation typically ignore problems associated with the implemented process, including some or all of the following: · Although academic simulations of the solution show echo reduction of more than 40 dB, real-world implementations are limited to about 20 dB due to nonlinearities, the presence of background noise, and the non-stationarity of the echo path W. This means that measurements of the background noise are biased by residual echo; Environmental noise and certain playback content may cause "leakage" in such systems (e.g., when the playback content excites nonlinear regions of the playback system due to buzz, rattle, and distortion). In these cases, the microphone output signal contains a significant amount of residual echo that is misinterpreted as background noise. In such cases, the adaptation of filter W' may also become unstable as the residual error signal grows. Also, the adaptation of filter W' may become unstable when the microphone signal is corrupted by high levels of noise. The amount of computation required to generate a noise estimate (Y') that is useful for implementing NCMP operating over a wide frequency range (e.g., one that covers typical music playback) is high. Summary of the Invention [Problem to be solved by the invention]

[0018] Noise compensation (e.g., automatic leveling of loudspeaker-played content) to compensate for environmental noise conditions is a well-known and desirable feature that has not yet been convincingly implemented. Using a microphone to measure the environmental noise conditions also measures the loudspeaker-played content, presenting a major challenge for the noise estimation (e.g., online noise estimation) required to implement noise compensation. Exemplary embodiments of the present invention are noise estimation methods and systems that generate noise estimates useful for performing noise compensation (e.g., for implementing many embodiments of noise-compensated media playback) in an improved manner. The noise estimation performed by exemplary implementations of such methods and systems has a simple formulation. [Means for solving the problem]

[0019] In one class of embodiments, a method of the present invention (e.g., a method of generating an estimate of the background noise in a playback environment) comprises the following steps: generating a microphone output signal using a microphone during emission of sound in a playback environment, where the sound is indicative of audio content of the playback signal and the microphone output signal is indicative of background noise and audio content in the playback environment; generating gap confidence values ​​(i.e., signals or data indicative of gap confidence values) in response to the microphone output signal (e.g., in response to the smoothed level of the microphone output signal) and the playback signal, where each gap confidence value is for a different time t (e.g., a different time interval that includes time t) and indicates a confidence that there is a gap in the playback signal at time t; The gap confidence value is used to generate an estimate of the background noise in the playback environment.

[0020] The reproduction environment may relate to the acoustic environment or acoustic space in which said sound is emitted. For example, the reproduction environment may be the acoustic environment in which the sound is emitted (e.g. by a speaker in response to a reproduction signal).

[0021] Typically, the estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, each of the noise estimates indicative of the background noise in the playback environment at a different time t, each of the noise estimates being a combination of candidate noise estimates weighted by gap confidence values ​​for different time intervals including the time t. Thus, generating an estimate of the background noise in the playback environment using the gap confidence values ​​comprises, for each noise estimate, weighting the candidate noise estimates for different time intervals including the time t by the gap confidence value and combining the weighted candidate noise estimates to obtain the respective noise estimate.

[0022] The candidate noise estimates may have different reliability (e.g., with respect to whether they faithfully represent the noise to be estimated). Their reliability may be indicated by respective gap confidence values. The method may consider the candidate noise estimates for a time interval (e.g., a sliding analysis window including time t) that includes time t (there is one candidate noise estimate for each time in the time interval) and weight each candidate noise estimate with its respective gap confidence value (e.g., a gap confidence value for each time in the interval). Thus, generating an estimate of the background noise in the playback environment using the gap confidence values ​​may include weighting the candidate noise estimates with their respective gap confidence values ​​and combining the weighted candidate noise estimates. In other words, for each time t, an interval (e.g., a sliding analysis window) that includes time t is considered. The interval may include a candidate noise estimate for each time in the interval. The actual noise estimate for time t may then be obtained by combining the candidate noise estimates for an interval including time t, in particular by combining the weighted candidate noise estimates, where each candidate noise estimate is weighted by the gap confidence value for the time of the respective candidate noise estimate.

[0023] For example, each of the candidate noise estimates may be a minimum echo-cancelled noise estimate, M resminand the noise estimate for each time interval may be a combination of the minimum echo-canceled noise estimate for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. The minimum echo-canceled noise estimate may relate to a minimum value of the sequence of echo-canceled noise estimates. For example, the minimum echo-canceled noise estimate may be obtained by performing a minimum following on the sequence of echo-canceled noise estimates. The minimum following may operate using an analysis window of a given length / size. The minimum echo-canceled noise estimate may then be the minimum of the echo-canceled noise estimates within the analysis window. The echo-canceled noise estimates are typically calibrated echo-canceled noise estimates, which have been calibrated to be in the same level region as the reconstructed signal. In another example, each of the candidate noise estimates may be a combination of the minimum calibrated microphone output signal value M of the sequence of microphone output signal values. min and the noise estimate for each time interval may be a combination of the minimum microphone output signal values ​​for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. The microphone output signal values ​​are typically calibrated microphone output signal values, which have been calibrated to be in the same level range as the playback signal.

[0024] In one class of embodiments, the candidate noise estimates are processed in a minimum follower (of gap confidence-weighted samples), in the sense that a minimum follower process is performed on the candidate noise estimates in each of a sequence of different time intervals. The minimum follower includes each candidate sample (each value of the candidate noise estimate for a time interval) in its analysis window only if the associated gap confidence is higher than a predefined threshold (e.g., the minimum follower assigns a weight of one to a candidate sample if the gap confidence for the sample is equal to or greater than the threshold, and the minimum follower assigns a weight of zero to a candidate sample if the gap confidence for the sample is lower than the threshold). In this class of embodiments, generating a noise estimate for each time interval includes the steps of (a) identifying each of the candidate noise estimates for that time interval for which a corresponding one of the gap confidence values ​​exceeds a predefined threshold, and (b) generating a noise estimate for that time interval to be the minimum of the candidate noise estimates identified in step (a).

[0025] In an exemplary embodiment, each gap confidence value (i.e., the gap confidence value for time t) is determined by dividing the gap confidence value by the minimum value in the playback signal level (S min ) is the smoothed level of the microphone output signal (at time t) (M smoothed ) to indicate how different it is from S min The level M at which the values ​​are smoothed smoothed The greater the confidence that there is a gap in the played content at time t, the greater the confidence that the candidate noise estimate for time t (e.g., the value M resmin or M min ) represents the background noise (at time t) in the playback environment.

[0026] Typically, the method comprises generating a sequence of gap confidence values ​​and generating a sequence of background noise estimates using the gap confidence values. Some embodiments of the method also comprise performing noise compensation on the audio input signal using the sequence of background noise estimates.

[0027] Some embodiments perform echo cancellation (in response to the microphone output signal and the playback signal) to generate the candidate noise estimates. Other embodiments generate the candidate noise estimates without performing echo cancellation.

[0028] Some embodiments of the invention include one or more of the following aspects.

[0029] One such aspect relates to determining gaps in the played-back content (using data indicating the confidence of the existence of each gap) and generating a background noise estimate (e.g., by implementing sampling gaps corresponding to the played-back content gaps in gap confidence weighted candidate noise estimates). Some embodiments generate candidate noise estimates, weight the candidate noise estimates with gap confidence data values ​​to generate gap confidence weighted candidate noise estimates, and generate a background noise estimate using the gap confidence weighted candidate noise estimates. In some embodiments, generating the candidate noise estimates includes performing echo cancellation. In other embodiments, generating the candidate noise estimates does not include performing echo cancellation.

[0030] Another such aspect relates to methods and systems that use a background noise estimate generated according to any embodiment of the invention to perform noise compensation on an input audio signal (eg, media playback with noise compensation).

[0031] Another such aspect relates to methods and systems for estimating background noise in a playback environment, thereby generating a background noise estimate useful for performing noise compensation on an input audio signal (e.g., media playback with noise compensation). In some such embodiments, the methods and / or systems also perform self-calibration (e.g., determining calibration gains for application to the playback signal, microphone output signal, and / or echo cancellation residual values ​​to implement the noise estimation) and / or automatic detection of system failures (e.g., hardware failures) if echo cancellation (AEC) is used in generating the background noise estimate.

[0032] Aspects of the invention further include systems configured (e.g., programmed) to perform embodiments of the inventive method or any of its steps, and tangible, non-transitory computer-readable media (e.g., disks or other tangible storage media) implementing non-transitory storage of data that stores code (e.g., executable code to perform) embodiments of the inventive method or any of its steps. For example, embodiments of the inventive system may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the inventive method or any of its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform embodiments of the inventive method (or any of its steps) in response to data presented thereto. [Brief description of the drawings]

[0033] [Figure 1] FIG. 1 is a block diagram of an audio playback system implementing noise compensated media playback (NCMP). [Diagram 2] 1 is a block diagram of a conventional system for generating a noise estimate by a conventional method known as echo cancellation from a microphone output signal, the microphone output signal being generated by capturing sound (indicative of the playback content) and noise in a playback environment. [Diagram 3] 1 is a block diagram of an embodiment of the inventive system for generating noise level estimates for each frequency band of a microphone output signal, the microphone output signal typically being generated by capturing sounds (representative of the playback content) and noise in a playback environment. [Figure 4] FIG. 5 is a block diagram of an implementation of the noise estimate generation subsystem 37 of the system of FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0034] Notation and Name Throughout this disclosure, including the claims, a "gap" in the playback signal refers to a time (or period of time) in the playback signal that is absent (or has a level below a predetermined threshold) of playback content.

[0035] Throughout this disclosure, including the claims, "speaker" and "loudspeaker" are used interchangeably to refer to any sound emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may also be implemented to include multiple transducers (e.g., a woofer and tweeter) all driven by a single common speaker feed (the speaker feeds may receive different processing in different circuit branches coupled to different transducers).

[0036] Throughout this disclosure, including the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal) is used broadly to refer to performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has been subjected to preliminary filtering or preprocessing before the operation is performed thereon).

[0037] Throughout this disclosure, including the claims, the term "system" is used broadly to refer to an apparatus, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may be referred to as a decoder system.

[0038] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or otherwise configurable to perform operations on data (e.g., audio, or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other audio data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0039] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean a direct or indirect connection. Thus, when a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0040] Detailed Description of the Preferred Embodiments Many embodiments of the present invention are technically possible. It will be clear to one skilled in the art from this disclosure how to implement them. Some embodiments of the systems and methods of the present invention are described herein with reference to Figures 3 and 4.

[0041] The system of Figure 4 is configured to generate an estimate of the background noise in the playback environment 28 and to use the noise estimate to perform noise compensation on the input audio signal. Figure 3 is a block diagram of an implementation of a noise estimation subsystem 37 of the system of Figure 4.

[0042] The noise estimation subsystem 37 of Fig. 4 is configured to generate background noise estimates (typically a sequence of noise estimates each corresponding to a different time interval) according to an embodiment of the noise estimation method of the present invention. The system of Fig. 4 also includes a noise compensation subsystem 24, which is coupled and configured to perform noise compensation on the input audio signal 23 using the noise estimates output from subsystem 37 (or a post-processed version of such noise estimates output from post-processing subsystem 39, if subsystem 39 is operative to modify the noise estimates output from subsystem 37) to generate a noise-compensated version of the input signal 23 (reproduced signal 25).

[0043] The system of FIG. 4 includes a content source 22 coupled and configured to output and provide an audio signal 23 to a noise compensation subsystem 24. The signal 23 represents at least one channel of audio content (sometimes referred to herein as media content or playback content) and is intended to be played back to generate a sound (in environment 28) representative of each channel of the audio content. The audio signal 23 may be a speaker feed (or two or more speaker feeds in the case of multi-channel playback content), and the noise compensation subsystem 24 may be coupled and configured to apply noise compensation to each such speaker feed by adjusting a playback gain of the speaker feed. Alternatively, another element of the system may generate the speaker feed (or multiple speaker feeds) in response to the audio signal 23 (e.g., the noise compensation subsystem 24 may be coupled and configured to generate at least one speaker feed in response to the audio signal 23 and apply noise compensation to each speaker feed by adjusting a playback gain of the speaker feed, such that the playback signal 25 consists of at least one noise-compensated speaker feed). In one mode of operation of the system of FIG. 4, subsystem 24 does not perform noise compensation and the audio content of reproduced signal 25 is the same as the audio content of signal 23.

[0044] Speaker system 29 (including at least one speaker) is coupled and configured to emit sound (in reproduction environment 28) in response to playback signal 25. Signal 25 may consist of a single playback channel or may consist of two or more playback channels. In typical operation, each speaker of speaker system 29 receives a speaker feed indicative of the playback content of a different channel of signal 25. In response, speaker system 29 emits sound (in reproduction environment 28) in response to the speaker feed. The sound is perceived by listener 31 (in environment 28) as a noise-compensated version of the playback content of input signal 23.

[0045] Other elements of the system of FIG. 4 are described below.

[0046] This disclosure refers to three types of background noise: Distracting noise (e.g., impulsive, low-frequency events (e.g., events having a duration of less than 0.5 seconds) such as a door slamming, a car horn, or driving over a road bump); Disrupting (for example, a brief event that interferes with the playing content, such as an airplane passing overhead, driving through a short tunnel, or driving through a section of new road surface) Pervasive (continuous / constant noises that may have beginning and endings but generally remain stationary, e.g. air conditioning, fans, urban ambient noise, rain, kitchen appliances).

[0047] Based on the inventors' experiments, in order of importance, characteristics of successful noise compensation include: Stability (the noise estimate should not be corrupted by the playback content measured by the microphone. The noise estimate, and therefore the compensation gain, should not fluctuate significantly due to changes in the playback content. The noise estimate should not track faster than "disruptive" sources of noise. The noise estimate should ignore "distracting" impulsive events); fast reaction time (a good noise estimator will only track "pervasive" sources of noise; however, a great noise estimator can reliably track "disruptive" sources of noise. Reacting quickly to changing noise conditions is crucial to the user experience); and A comfortable amount of compensation (noise compensation should ensure intelligibility and preservation of timbre in the presence of noise. Too little or too much compensation will result in an unsatisfactory user experience. Compensation is performed more faithfully in a multiband sense than bulk volume adjustments).

[0048] Noise estimation using minimum-following filters to track stationary noise is an established technique. To perform such estimation, a minimum-follower filter accumulates input samples in a sliding, fixed-size buffer called the analysis window and outputs the minimum sample value in that buffer. Minimum following removes impulsive and distracting noise sources for both short and long analysis windows. Long analysis windows (with durations on the order of 10 seconds) are effective at locating stationary noise floors (pervasive noise) because the minimum follower retains minima that appear between user utterances in the vicinity of the microphone during gaps in the playback content. The longer the analysis window, the more likely it is that a gap will be found. However, this approach follows a minimum regardless of whether the minimum is actually a gap in the playback content. Furthermore, a long analysis window requires a long time for the system to track and rise to an increase in background noise, which is a notable drawback for noise compensation. Long analysis windows typically eventually track pervasive noise sources, but miss tracking disruptive noise sources.

[0049] An important aspect of exemplary embodiments of the present invention is to use knowledge of the playback signal to determine when conditions are most favorable for measuring a noise estimate from the microphone output (and optionally from an echo-canceled noise estimate generated by performing echo cancellation on the microphone output). A realistic playback signal viewed in the time-frequency domain typically contains points where the signal energy is low, which implies that those points in time and frequency are good opportunities for measuring ambient noise conditions. An important aspect of exemplary embodiments of the present invention is how to quantify how good these opportunities are (e.g., by assigning each of them a value called a "gap confidence" value or "gap confidence"). Approaching the problem in this way allows noise compensation (or noise estimation) for many types of content without the need for an echo canceller (to generate an echo-canceled noise estimate), lowering the requirements for the performance of the echo canceller (if an echo canceller is used).

[0050] Next, with reference to Figures 3 and 4, an embodiment of a method and system of the present invention for calculating a sequence of estimates of background noise levels for each of several different frequency bands of playback content will be described. Figure 4 is a block diagram of the system, and Figure 3 is a block diagram of an implementation of subsystem 37 of the system of Figure 4. It should be understood that the elements of Figure 4 (except for playback environment 28, speaker system 29, microphone 30, and listener 31) can be implemented in or as a processor. Those performing the signal (or data) processing operations of such elements (including those referred to herein as subsystems) are implemented in software, firmware, or hardware.

[0051] A microphone output signal (e.g., signal "Mic" in FIG. 4) is generated using a microphone (e.g., microphone 30 in FIG. 4) that occupies the same acoustic space (environment 28 in FIG. 4) as a listener (e.g., listener 31 in FIG. 4). Two or more microphones can be used (e.g., combining their individual outputs) to generate a microphone output signal, and thus the term "microphone" is used broadly herein to refer to either a single microphone or two or more microphones operated to generate a single microphone output signal. The microphone output signal, which represents both the sound reproduction signal (the sound reproduction content radiated from speaker system 29 in FIG. 4) and competing background noise, is converted to a frequency domain representation (e.g., by time-frequency transform element 32 in FIG. 4), thereby generating frequency domain microphone output data. The frequency domain microphone output data is banded (e.g., by element 33 in FIG. 4) into the power domain to provide microphone output values ​​(e.g., values ​​M' in FIGS. 3 and 4). For each frequency band, the level of a corresponding one of those values ​​(one of the values ​​M') is adjusted using a calibration gain G (e.g., applied by gain stage 11 in FIG. 3) to produce an adjusted value M (e.g., one of the values ​​M in FIG. 3). Application of the calibration gain G is required to correct for level differences in the digital playback signal (value S) and the digitized microphone output signal level (value M'). Methods for determining G (for each frequency band) automatically and through measurement are discussed below.

[0052] Each channel of the playback content (e.g., each channel of the noise-compensated signal 25 of FIG. 4), which is typically a multi-channel playback content, is frequency transformed (e.g., by the time-to-frequency transform element 26 of FIG. 4, preferably using the same transform as that performed by the transform element 32), thereby generating frequency-domain playback content data. The frequency-domain playback content data (for all channels) is downmixed (if the signal 25 includes more than one channel), and the resulting single stream of frequency-domain playback content data is banded (e.g., by the element 27 of FIG. 4, preferably using the same banding operation as that performed by the element 33 to generate the value M') to obtain the playback content value S (e.g., the value S of FIGS. 3 and 4). The value S should also be delayed (e.g., before being processed according to an embodiment of the present invention by the element 13 of FIG. 3) to account for any latency in the hardware (e.g., latency due to A / D and D / A conversion). This adjustment can be considered a coarse adjustment.

[0053] The system of FIG. 4 includes an echo canceller 34 coupled and configured to generate an echo-canceled noise estimate by performing echo cancellation on the frequency domain values ​​output from elements 26 and 32, and a banding subsystem 35 coupled and configured to perform frequency banding on the echo-canceled noise estimates (residual values) output from the echo canceller 34 to generate banded echo-canceled noise estimates (residual values) M'res (which includes a value M'res for each frequency band).

[0054] If signal 25 is a multi-channel signal (including Z playback channels), a typical implementation of echo canceller 34 would receive (from element 26) multiple streams of frequency-domain playback content values ​​(one stream for each channel) and apply a filter W' for each playback channel. i (corresponding to filter W' in FIG. 2). In this case, the frequency domain representation of the microphone output signal Y is W1X+W2X+...+WZ X+N, where each W i is the transfer function for a distinct ("i") one of the Z speakers. An implementation of such an echo canceller 34 may be implemented by computing each W' from a frequency domain representation of the microphone output signal Y. i The X estimates (one for each channel) are subtracted to produce a single stream of echo-canceled noise estimate (or "residual") values, which correspond to the echo-canceled noise estimate Y' of FIG.

[0055] In general, an echo-canceled noise estimate is obtained by applying echo cancellation to the microphone output signal (the echo resulting from or related to the sound / audio content of the reproduced signal). Thus, it can be said that the echo-canceled noise estimate is obtained by cancelling the echo resulting from or related to the sound from the microphone output signal (or, in other words, resulting from or related to the audio content of the reproduced signal). This can be done in the frequency domain.

[0056] The filter coefficients of each adaptive filter used by echo canceller 34 to generate the echo-canceled noise estimate (i.e., each adaptive filter implemented by echo canceller 34 that corresponds to filter W' of FIG. 2) are banded in banding element 36. The banded filter coefficients are provided from element 36 to subsystem 43 for use by subsystem 43 to generate a gain value G that is used by subsystem 37.

[0057] Optionally, echo canceller 34 is omitted (or inoperative), and thus no adaptive filter values ​​are provided to banding element 36, and no banded adaptive filter values ​​are provided from 36 to subsystem 43. In this case, subsystem 43 generates gain value G in one of several ways (described below) that do not use banded adaptive filter values.

[0058] If an echo canceller is used (i.e., the system includes and uses elements 34 and 35 as shown in FIG. 4), the residual values ​​output from echo canceller 34 are banded (e.g., in subsystem 35 of FIG. 4) to generate a banded noise estimate M'res. A calibration gain G (generated by subsystem 43) is applied (e.g., by gain stage 12 of FIG. 3) to the values ​​M'res (i.e., gain G includes a set of band-specific gains, one for each band, and each band-specific gain is applied to the values ​​M'res in the corresponding band) to bring the signal (represented by values ​​M'res) into the same level range as the reconstructed signal (represented by values ​​S). For each frequency band, a corresponding one of the values ​​M'res is level adjusted with the calibration gain G (applied by gain stage 12 of FIG. 3) to generate an adjusted value Mres (i.e., one of the values ​​Mres of FIG. 3).

[0059] If an echo canceller is not used (i.e., echo canceller 34 is omitted or inoperative), then the value M'res (in the present description of Figures 3 and 4) is replaced by the value M'. In this case, the banded value M' (from element 33) is presented not only to the input of gain stage 11 but also to the input of gain stage 12 (instead of the value M'res shown in Figure 3). A gain G is applied to the value M' (by gain stage 12 in Figure 3) to generate an adjusted value M, and the adjusted value M (rather than the adjusted value Mres as shown in Figure 3) is processed by subsystem 20 in the same way as (and instead of) the adjusted value Mres to generate the noise estimate.

[0060] In typical implementations (including that shown in FIG. 3), the noise estimate generation subsystem 37 is configured to perform a minimum fitting on the reproduced content values ​​S to locate gaps in (i.e., determined by) the adjusted version (Mres) of the noise estimate M'res. Preferably, this is implemented as described with reference to FIG.

[0061] 3, the subsystem 37 includes a pair of minimum followers (13 and 14), both of which operate with analysis windows of the same size. The minimum follower 13 calculates a value S that indicates the minimum (within each analysis window) of the values ​​S. min The minimum follower 14 is configured to look through the values ​​S to generate a minimum value M resmin The inventors have realised that because the values ​​S, M and Mres are at least roughly time-aligned, in gaps in the played content (as indicated by a comparison of the played content value S with the microphone output value M), The minimum value of Mres (the echo canceller residual) can be confidently considered to represent an estimate of the noise in the playback environment; The minimum M (microphone output signal) value is confidently considered to represent an estimate of the noise in the playback environment.

[0062] The inventors have also realised that other than during gaps in the playback content, the minimum value of the value Mres (or the value M) may not provide an accurate estimate of the noise in the playback environment.

[0063] Microphone output signals (M) and S min In response to the values ​​of M, the sample aggregator subsystem 20 generates gap confidence values. resmin (or the value of M if echo cancellation is not performed) and is configured to use the gap confidence value (produced by subsystem 16) as an indication of the reliability of the candidate noise estimate.

[0064] More specifically, the sample aggregator subsystem 20 of FIG. 3 collects candidate noise estimates (M resmin) in a weighted manner by the gap confidence values ​​(generated in subsystem 16) to generate a final noise estimate for each analysis window (i.e., the analysis window of aggregator 20 having length τ2 as shown in FIG. 3). Weighted candidate noise estimates corresponding to gap confidence values ​​indicating low gap confidence are assigned a weight of zero or a weight smaller than weighted candidate noise estimates corresponding to gap confidence values ​​indicating high gap confidence. Thus, subsystem 20 uses the gap confidence values ​​to output a sequence of noise estimates (a collection of current noise estimates, including one noise estimate for each analysis window and for each frequency band).

[0065] A simple example of a subsystem 20 is a minimum follower (of gap confidence weighted samples), e.g., a system that selects a candidate sample (M) only if its associated gap confidence is higher than a given threshold. resmin (i.e., the subsystem 20 selects a sample M if the gap confidence for the sample is equal to or greater than the threshold value. resmin If the gap confidence of a sample is less than the threshold, the weight of the sample is assigned to M resmin Other implementations of subsystem 20 may assign a weight of 0 to the gap confidence weighted samples (M resmin An exemplary implementation of the gap confidence weighted sample aggregation subsystem 20 is (or includes) a linear interpolator / unipolar smoother with an update rate controlled by the gap confidence value.

[0066] Subsystem 20 processes the incoming samples (M resmin A strategy may be adopted that ignores the gap confidence when the gap confidence (value of ) is lower than the current noise estimate (determined by subsystem 20) in order to track deteriorating noise conditions even when gaps are not available.

[0067] Preferably, subsystem 20 is configured to effectively hold the noise estimate during a period of low gap confidence until a new sampling opportunity occurs as determined by the gap confidence. For example, in one preferred implementation of subsystem 20, if subsystem 20 determines a current noise estimate (within an analysis window) where the gap confidence value (generated by subsystem 16) indicates a low confidence that there is a gap in the playback content (e.g., the gap confidence value indicates a gap confidence lower than a predetermined threshold), subsystem 20 continues to output the current noise estimate until the gap confidence value (in a new analysis window) indicates a higher confidence that there is a gap in the playback content (e.g., the gap confidence value indicates a gap confidence higher than the threshold), at which point subsystem 20 generates (and outputs) an updated noise estimate. By using the gap confidence values ​​in this manner to generate noise estimates in accordance with a preferred embodiment of the present invention (including by holding the noise estimates during periods of low gap confidence until a new sampling opportunity occurs as determined by the gap confidence), rather than relying solely on the candidate noise estimates output from the minimum follower 14 as a sequence of noise estimates (without determining and using the gap confidence values) or generating noise estimates in other conventional ways, the length of all minimum follower analysis windows used (i.e., the analysis window length τ1 of each of the minimum followers 13 and 14, and the analysis window length τ2 of the aggregator 20 if the aggregator 20 is implemented as a minimum follower of gap confidence-weighted samples) can be reduced by about an order of magnitude compared to conventional approaches. This improves the speed with which the noise estimation system can track noise conditions when gaps occur. Exemplary default values ​​for the analysis window sizes are described below.

[0068] In one class of implementations, the sample aggregator 20 is configured to forward report (i.e., output) not only the current noise estimates, but also an indication of how up-to-date the noise estimates are in each frequency band, referred to herein as "gap health." In an exemplary implementation, gap health is a unitless measure, calculated (in one exemplary implementation) as follows:

number

[0069] In one class of implementations, the gap confidence subsystem 16 uses S min value (output from minimum follower 13) and a smoothed version of the M value (output from gain stage 11) (i.e., the smoothed value M smoothed ) is configured to process, for example, S min Value M smoothed Typically, subsystem 16 generates (and provides to aggregator 20) a set of gap confidence values ​​(one for each frequency band) for each analysis window of minimum followers 13, and the discussion here focuses on the S for a particular frequency band ( min and M. smoothed Regarding the generation of gap confidence values ​​(from values ​​of

[0070] Each gap confidence value (for one band, at one time point) is the corresponding M resmin value (i.e., M for the same band and time resmin Each minimum (Mres value) recognized (during a gap in the playback content) by the minimum follower 14 (which acts on the Mres value) indicates how well the noise conditions in the playback environment are represented. resmin ) can be considered with confidence to be indicative of the noise conditions in the playback environment. If there are no gaps in the playback content, the minimum (M resmin ) cannot be considered with confidence as an indication of the noise conditions in the playback environment. Instead, the minimum (S min ) may be indicated.

[0071] Subsystem 16 typically calculates each gap confidence value (GapConfidence for time t) as min is the smoothed (average) level detected by the microphone (M smoothed ) is generated to show how different it is from S min is the smoothed (average) level detected by the microphone (M smoothed ), the greater the confidence that there is a gap in the playback content at time t. resmin represents the noise conditions (at time t) in the playback environment.

[0072] The calculation of each gap confidence value for each band (i.e., the gap confidence value for each time, e.g., for each analysis window of the minimum follower 13) is calculated by the minimum followed playback content energy level S at time t. min and the smoothed microphone energy level M at the same time t smoothed In a preferred embodiment, each gap confidence value output from subsystem 16 is a unitless value proportional to:

number

[0073] The value of δ sets the desired distance between the observed minimum of the played content and the smoothed microphone level. This parameter trades off error and stability against the update rate of the system, and depends on how aggressive the noise compensation gain is.

[0074] M smoothed Using δ as a comparison point means that, given the current conditions, the current gap confidence value takes into account the severity of the noise error in the estimate. In general, if δ is chosen to be large enough, the operation of the noise estimator exploits the following scenario: S min For a fixed value of M smoothed The larger the value of , the higher the gap confidence should be. smoothed If M increases, it is possible to tolerate a larger error in the noise estimate due to residual echo because the error becomes small compared to the magnitude of the noise condition. smoothed When S increases, the noise compensator does not perform as much compensation, and the effect of errors in the noise estimate is also reduced. min For a fixed value of M smoothedThe smaller the value of , the smaller the gap confidence should be. In this situation, the error introduced through the residual echo of the microphone output signal will be large relative to the reproduced content, and will have a large impact on the compensation experience. Therefore, it is appropriate for the noise estimator to be more conservative when calculating the gap confidence under these conditions.

[0075] In applications that make heavy use of echo cancellation ("AEC"), the cost of introducing errors is low and δ can be relaxed (reduced) so that the noise estimate (output from subsystem 20) will exhibit more frequent gaps. In applications without AEC, δ can be increased so that the noise estimate (output from subsystem 20) will exhibit only gaps of higher quality.

[0076] The following table summarizes the tuning parameters of the implementation of the inventive noise estimator of FIG. 3 with and without echo cancellation (the two right-hand columns of the table give typical default values ​​for the tuning parameters (δ, C, the analysis window length τ1 of the minimum followers 13, 14, and the analysis window length τ2 of the sample aggregator 20 implemented as a minimum follower of gap confidence weighted samples): [Table 1] All tuning parameters affect the update rate of the system, which must be balanced against the accuracy of the system's noise estimation. In general, a faster responding system that allows for some error is better than a conservative, slower responding system that relies on a high quality gap, as long as stability is maintained.

[0077] The described approach to calculating the gap confidence (e.g., the output of subsystem 16 in FIG. 3) is different from attempting to calculate the current signal-to-noise ratio, i.e., the ratio of the echo level to the current noise level. Any gap confidence calculation that relies on a current noise estimate will generally break as soon as there is a change in the noise conditions, either because it samples too liberally or too conservatively. Knowing the current SNR may be the best way to determine the gap confidence (in an academic sense), but this requires knowing the noise conditions, i.e., the very thing the noise estimator is trying to determine. This is a cyclic dependency that does not work in practice.

[0078] Referring again to Fig. 4, additional elements of an implementation of a noise estimation system (shown in Fig. 4) according to an exemplary embodiment of the present invention will be described in more detail. As mentioned above, noise compensation is performed (by subsystem 24) on the playback content 23 using a noise estimate spectrum generated by a noise estimator subsystem 37 (implemented as in Fig. 3 above). The noise compensated playback content 25 is played through a speaker system 29 to a listener (e.g., listener 31) in a playback environment (environment 28). A microphone 30 in the same acoustic environment (environment 28) as the listener receives both the environmental (ambient) noise and the playback content (echo).

[0079] The noise compensated playback content 25 is transformed (in element 26), downmixed (in element 27) and frequency banded to produce a value S. The microphone output signal is transformed (in element 32) and banded (in element 33) to produce a value M'. If an echo canceller (34) is used, the residual signal from the echo canceller (the echo cancelled noise estimate) is banded (in element 35) to produce a value Mres'.

[0080] Subsystem 43 determines a calibrated gain G (for each frequency band) according to the microphone-to-digital mapping, which captures the level difference per frequency band between the playback content in the digital domain at the point where the playback content is derived and provided to the noise estimator (e.g., the output of time-to-frequency domain transform element 26) and the playback content received by the microphone. Each set of current values ​​of gain G is provided from subsystem 43 to noise estimator 37 (for application by gain stages 11 and 12 of the FIG. 3 implementation of noise estimator 37).

[0081] Subsystem 43 has access to at least one of the following three sources of data: factory pre-set gains (stored in memory 40); the state of the gain G generated (by subsystem 43) during the previous session (and stored in memory 41); If an AEC (e.g., echo canceller 34) is present and used, banded AEC filter coefficient energies (e.g., those that determine the adaptive filter implemented by the echo canceller, corresponding to filter W' in FIG. 2). These banded AEC filter coefficient energies (e.g., the energy provided from banding element 36 to subsystem 43 in the system of FIG. 4) serve as an online estimate of gain G.

[0082] If AEC is not used (eg, if a version of the system of FIG. 4 that does not include echo canceller 34 is used), subsystem 43 generates the calibration gain G from gain values ​​in memory 40 or 41 .

[0083] Thus, in some embodiments, subsystem 43 is configured to cause the system of FIG. 4 to perform self-calibration by determining (e.g., from the banded AEC filter coefficient energies provided from banding element 36) calibration gains for application by subsystem 37 to the playback signal, the microphone output signal, and the echo cancellation residual values ​​to perform noise estimation.

[0084] 4, the sequence of noise estimates produced by noise estimator 37 is optionally post-processed (in subsystem 39). Post-processing includes by performing one or more of the following operations thereon: Imputation of missing noise estimates from partially updated noise estimates; Constraining the shape of the current noise estimate to preserve timbre; and Constraints on the absolute value of the current noise estimate.

[0085] The microphone-to-digital mapping performed by subsystem 43 to determine the gain value G captures the level difference between the playback content in the digital domain (e.g., the output of time-to-frequency domain transform element 26) at the point where the playback content is extracted for provision to the noise estimator and the playback content received by the microphone. The mapping is determined primarily by the physical spacing and characteristics of the speaker system and microphones, as well as the electrical amplification gains used in the sound reproduction and microphone signal amplification.

[0086] In the most basic example, the microphone-to-digital mapping may be a pre-stored factory tuning, measured during production design on a sample of devices, and reused for all such devices produced.

[0087] If an AEC (e.g., echo canceller 34 in FIG. 4) is used, more sophisticated control over the microphone-to-digital mapping is possible. An online estimate of the gain G can be determined by taking the absolute values ​​of the adaptive filter coefficients (determined by the echo canceller) and bundling them together. For a sufficiently stable echo canceller design, this online estimate can be as good as an offline pre-arranged factory calibration, provided there is sufficient smoothing on the estimated gain (G'). This allows the estimated gain G' to be used instead of factory adjustments. Another advantage of calculating the estimated gain G' is that any device-to-device deviations from the factory default value can be measured and taken into account.

[0088] Although the estimated gain G' can be substituted for the factory determined gain, a robust approach that combines both the factory gain and the online estimated gain G' to determine the gain G for each band is as follows: G = max(min(G',F+L),FL) where F is the factory gain for the band, G' is the estimated gain for the band, and L is the maximum allowable deviation from the factory setting. All gains are expressed in dB. If the value G' exceeds the indicated range for an extended period of time, this may indicate a hardware failure, and the noise compensation system may decide to fall back to a safe behavior.

[0089] A post-processing step performed (e.g., by element 39 of the system of FIG. 4) on the sequence of noise estimates generated (e.g., by element 37 of the system of FIG. 4) in accordance with an embodiment of the invention can be used to maintain a higher quality noise compensation experience. For example, post-processing that forces the noise spectrum to fit a particular shape to remove peaks can help prevent compensation gains that distort the timbre of the played content in an unpleasant way.

[0090] An important aspect of some embodiments of the noise estimation method and system of the present invention is post-processing (e.g., performed by an implementation of element 39 of the system of FIG. 4), e.g., implementing an imputation strategy to update old noise estimates (for some frequency bands) that have not been updated because there are no gaps in the playback content, while noise estimates for other bands have been sufficiently updated.

[0091] In some such embodiments, the gap health reported by the noise estimator (e.g., gap health values ​​for each frequency band generated by subsystem 20 of the FIG. 3 implementation of the inventive noise estimator, as described above) determines which bands (of the current noise estimate) are "stale" or "latest." An exemplary method (performed by an implementation of element 39 of the system of FIG. 4) of using the gap health values ​​(generated by noise estimator 37 for each frequency band) to impute a noise estimate includes the following steps: Starting with the first band, the gap health for that band is determined by a predetermined threshold α Healthy locate bands that are sufficiently up-to-date (healthy bands) by checking whether Once a healthy band is found, the subsequent bands are then compared for low gap health as determined by a different threshold, again with the threshold α Healthy Inspect the latest band as determined by; If a second healthy band is found and all bands between it and the first healthy band are stale, a linear interpolation operation is performed between the two healthy bands to generate at least one interpolated noise estimate. The noise estimates (for all bands between the two healthy bands) are linearly interpolated in the logarithmic domain between the two healthy bands to give new values ​​for the stale bands; then Continue the above processes starting with the next zone (ie, repeat the processes from step 1).

[0092] Imputation of stale values ​​may not be necessary in embodiments where a sufficient number of gaps are always available and bands rarely become stale. The default thresholds for the simple imputation algorithm are given in the following table: [Table 2] Other methods of operating on the gap robustness and noise estimates are of course possible.

[0093] In some embodiments, element 39 of the system of FIG. 4 is implemented to perform automatic detection of system failures (e.g., hardware failures) using the gap health values ​​generated by noise estimator 37 for each frequency band, for example when automatic echo cancellation (AEC) is used to generate the background noise estimates.

[0094] The gap reliability determination (and using the determined gap reliability data to perform noise estimation) according to the exemplary embodiments of the present invention disclosed herein enables a promising noise compensation experience (using a noise estimate determined using the gap reliability value) without the need for an echo canceller across a range of audio types encountered in media playback scenarios. By including an echo canceller performing the gap reliability determination according to some embodiments of the present invention, the responsiveness of the noise compensation (using a noise estimate determined using the determined gap reliability data) can be improved and the dependency on the playback content characteristics can be removed. The exemplary implementation of the gap reliability determination and using the determined gap reliability data to perform noise estimation reduces the requirements placed on the echo canceller (also used to perform the noise estimation) and reduces the significant optimization and testing efforts involved.

[0095] Removing the echo canceller from the noise compensation system: Saves a lot of development time since echo cancellers require a lot of time and research to tune to ensure cancellation performance and stability; Save computation time, since large adaptive filter banks (for implementing echo cancellation) typically consume large resources and often require high-precision operations to perform; Eliminates the need for a shared clock domain and time alignment between the microphone signal and the played-back audio signal. Echo cancellation relies on both the played-back and recorded signals being synchronized with the same audio clock.

[0096] The noise estimator (implemented, for example, without echo cancellation, according to any of the exemplary embodiments of the present invention) can operate at increased block rates / smaller FFT sizes for further complexity savings. Echo cancellation performed in the frequency domain typically requires narrow frequency resolution.

[0097] When using echo cancellation (and gap confidence determination) to generate a noise estimate in accordance with exemplary embodiments of the present invention, the performance of the echo canceller can be degraded without compromising the user's experience (when the user listens to a noise-compensated playback content achieved using a noise estimate generated in accordance with exemplary embodiments of the present invention) because the echo canceller only needs to perform enough cancellation to reveal gaps in the playback content, and does not need to maintain a high ERLE for peaks in the playback content (where "ERLE" stands for echo return loss enhancement, a measure of how much echo, expressed in dB, is removed by the echo canceller).

[0098] An exemplary embodiment of the method of the present invention includes: [E1] generating a microphone output signal using a microphone during emission of sound in a playback environment, the sound indicative of audio content of the playback signal, the microphone output signal indicative of background noise and the audio content in the playback environment; generating (e.g., in element 16 of the system of FIG. 3 ) gap confidence values ​​in response to the microphone output signal and the playback signal, each gap confidence value being for a different time t and indicating a confidence that a gap exists in the playback signal at time t; and generating (e.g. in element 20 of the system of FIG. 3 ) an estimate of the background noise in the playback environment using said gap confidence value. method. [E2] The method of claim E1, wherein the estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, and each of the noise estimates (e.g., each noise estimate output from element 20 of the system of FIG. 3, which is an implementation of element 37 of FIG. 4) being a combination of candidate noise estimates weighted by the gap confidence value for different time intervals including time t. [E3] The sequence of noise estimates includes a noise estimate for each of the time intervals, and generating a noise estimate for each of the time intervals includes: (a) identifying (e.g., in element 20 of the system of FIG. 3 ) each of the candidate noise estimates for the time interval for which a corresponding one of the gap confidence values ​​exceeds a predetermined threshold; (b) generating the noise estimate for the time interval to be the minimum of the candidate noise estimates identified in step (a). The method described in E2. [E4] Each of the candidate noise estimates is the smallest echo-cancelled noise estimate in the sequence of echo-cancelled noise estimates (e.g., the value M resmin and the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of the smallest echo cancelled noise estimate for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [E5] Each of the candidate noise estimates may be the minimum microphone output signal value of a sequence of microphone output signal values ​​(e.g., the value M output from element 14 of the system of FIG. 3 in an implementation in which element 12 of the system receives microphone output value M' rather than value M'res). min ), and the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of a minimum microphone output signal value for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [E6] The step of generating a gap confidence value comprises: generating a gap confidence value for each time t, wherein generating a gap confidence value for each time t comprises: processing the regenerated signal (e.g., in element 13 of the system of FIG. 3 ) to determine a minimum in the regenerated signal level for time t; processing the microphone output signal (e.g., in elements 11 and 17 of the system of FIG. 3 ) to determine a smoothed level of the microphone output signal for time t; by determining (e.g. in element 18 of the system of FIG. 3 ) the gap confidence value for time t to indicate how much the minimum in playback signal level for time t differs from the smoothed level of the microphone output signal for time t; The method described in E1. [E7] The estimate of the background noise in the playback environment is or comprises a sequence of noise estimates: performing noise compensation on the audio input signal using said sequence of noise estimates (e.g., in element 24 of the system of FIG. 4); The method described in E1. [E8] The step of performing noise compensation on the audio input signal comprises generating the reproduced signal, the method comprising: driving at least one speaker using the playback signal to generate the sound; The method described in E7. [E9] performing a time domain to frequency domain transform on the microphone output signal, thereby generating frequency domain microphone output data; generating frequency domain playback content data in response to the playback signal, the gap confidence value being generated in response to the frequency domain microphone output data and the frequency domain playback content data. The method described in E1. An exemplary embodiment of the system of the present invention includes: [E10] a microphone (e.g., microphone 30 of the system of FIG. 4 ) configured to generate a microphone output signal during emission of a sound in a playback environment, the sound indicative of audio content of a playback signal, the microphone output signal indicative of background noise and the audio content in the playback environment; coupled to receive the microphone output signal and the playback signal; generating gap confidence values ​​in response to the microphone output signal and the playback signal, each gap confidence value being for a different time t and indicating a confidence that a gap exists in the playback signal at time t; and generating an estimate of the background noise in the playback environment using said gap confidence value. system. [E11] The system of E10, wherein the noise estimation system is configured to generate the estimate of the background noise in the playback environment such that the estimate of the background noise in the playback environment is or includes a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, and each of the noise estimates (e.g., each noise estimate output from element 20 of the implementation of FIG. 3 of element 37 of FIG. 4) is a combination of candidate noise estimates weighted by the gap confidence value for different time intervals including time t. [E12] The sequence of noise estimates includes a noise estimate for each of the time intervals, and the noise estimation system generates a noise estimate for each of the time intervals by: (a) identifying (e.g., at element 20 of FIG. 3 ) each of the candidate noise estimates for the time interval for which a corresponding one of the gap confidence values ​​exceeds a predetermined threshold; (b) generating the noise estimate for the time interval to be the minimum of the candidate noise estimates identified in step (a). The system described in E11. [E13] Each of the candidate noise estimates is the smallest echo-canceled noise estimate in the sequence of echo-canceled noise estimates (e.g., the value M resminand the sequence of noise estimates includes a noise estimate for each of the time intervals, the noise estimate for each of the time intervals being a combination of a minimum echo cancelled noise estimate for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [E14] Each of the candidate noise estimates is the minimum microphone output signal value of a sequence of microphone output signal values ​​(e.g., the value M output from element 14 of the system of FIG. 3 in an implementation in which element 12 of the system receives microphone output value M' rather than M'res). min ), and the sequence of noise estimates comprises a noise estimate for each of the time intervals, the noise estimate for each of the time intervals being a combination of a minimum microphone output signal value for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [E15] The gap confidence values ​​include a gap confidence value for each time t, and the noise estimation system generates the gap confidence values ​​for each time t by: processing the regenerated signal to determine a minimum in the regenerated signal level for time t (e.g., in element 13 of the FIG. 3 implementation of element 37 of the FIG. 4 system); processing the microphone output signal (e.g., in elements 11 and 17 of the FIG. 3 implementation of element 37 of the FIG. 4 system) to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback signal level for time t differs from the smoothed level of the microphone output signal for time t (e.g., in element 18 of the implementation of FIG. 3 of the system of FIG. 4); The system described in E10. [E16] The estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, the system comprising: a noise compensation subsystem (e.g., element 24 of the system of FIG. 4 ) coupled to receive the sequence of noise estimates and configured to perform noise compensation on the audio input signal using the sequence of noise estimates to generate the reproduced signal. The system described in E10. [E17] said noise estimation system comprising: performing a time domain to frequency domain transform on said microphone output signals (e.g., in elements 32 and 33 of the system of FIG. 4 ), thereby generating frequency domain microphone output data; generating frequency domain playback content data in response to the playback signal (e.g., in elements 26 and 27 of the system of FIG. 4); and configured to generate the gap confidence value in response to the frequency domain microphone output data and the frequency domain playback content data. The system described in E10.

[0099] Aspects of the invention include systems or devices configured (e.g., programmed) to perform any of the method embodiments of the invention, and tangible computer readable media (e.g., disks) storing code for implementing any of the method embodiments or steps thereof. For example, a system of the invention may be or include a programmable general purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including the method embodiments or steps thereof. Such a general purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform the method embodiments (or steps thereof) of the invention in response to data presented thereto.

[0100] Some embodiments of the system of the present invention (e.g., some implementations of the system of FIG. 3 or some implementations of elements 24, 26, 27, 34, 32, 33, 35, 36, 37, 39, and 43 of the system of FIG. 4) are implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform necessary processing on an audio signal, including performing embodiments of the method of the present invention. Alternatively, some embodiments of the system of the present invention (e.g., some implementations of the system of FIG. 3 or some implementations of elements 24, 26, 27, 34, 32, 33, 35, 36, 37, 39, and 43 of the system of FIG. 4) are implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and memory) that is programmed in software or firmware and / or otherwise configured to perform any of a variety of operations, including embodiments of the method of the present invention. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform embodiments of the inventive method, the system also including other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform embodiments of the inventive method is typically coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0101] Another aspect of the invention is a computer readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., executable code for executing) for performing an embodiment of any of the methods or steps thereof of the invention.

[0102] While specific embodiments of the invention and applications of the invention have been described herein, it will be apparent to those skilled in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the invention as described and claimed herein. While certain forms of the invention have been illustrated and described, it should be understood that the invention is not limited to the specific embodiments described and illustrated or to the specific methods described.

[0103] Various aspects of the present invention can be understood from the following enumerated example embodiments (EEE): [EEE1] generating a microphone output signal using a microphone during emission of sound in a playback environment, the sound indicative of audio content of the playback signal, the microphone output signal indicative of background noise and the audio content in the playback environment; generating gap confidence values ​​in response to the microphone output signal and the playback signal, each gap confidence value being for a different time t and indicating a confidence that a gap exists in the playback signal at time t; and generating an estimate of the background noise in the playback environment using the gap confidence value. method. [EEE2] The method according to EEE1, wherein the estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, each of the noise estimates being a combination of candidate noise estimates weighted by the gap confidence value for different time intervals including time t. [EEE3] The sequence of noise estimates includes a noise estimate for each of the time intervals, and generating a noise estimate for each of the time intervals includes: (a) identifying each of the candidate noise estimates for the time interval for which a corresponding one of the gap confidence values ​​exceeds a predetermined threshold; (b) generating the noise estimate for the time interval to be the minimum of the candidate noise estimates identified in step (a). The method described in EEE2. [EEE4] Each of the candidate noise estimates is a minimum echo-canceled noise estimate, M resmin and the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of the smallest echo cancelled noise estimate for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [EEE5] Each of the candidate noise estimates is a minimum microphone output signal value M min and the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of a minimum microphone output signal value for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [EEE6] The step of generating a gap confidence value comprises: generating a gap confidence value for each time t, wherein generating a gap confidence value for each time t comprises: processing the playback signal to determine a minimum in playback signal level for time t; processing the microphone output signal to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback signal level for time t differs from the smoothed level of the microphone output signal for time t. The method according to any one of EEE1 to 5. [EEE7] The estimate of the background noise in the playback environment is or comprises a sequence of noise estimates: performing noise compensation on an audio input signal using said sequence of noise estimates. The method according to any one of EEE1 to 6. [EEE8] The step of performing noise compensation on the audio input signal comprises generating the reproduced signal, the method comprising: driving at least one speaker using the playback signal to generate the sound; The method described in EEE7. [EEE9] performing a time domain to frequency domain transform on the microphone output signal, thereby generating frequency domain microphone output data; generating frequency domain playback content data in response to the playback signal, the gap confidence value being generated in response to the frequency domain microphone output data and the frequency domain playback content data. The method according to any one of EEE1 to 8. [EEE10] a microphone configured to generate a microphone output signal during emission of a sound in a playback environment, the sound indicative of an audio content of a playback signal, the microphone output signal indicative of background noise and the audio content in the playback environment; coupled to receive the microphone output signal and the playback signal; generating gap confidence values ​​in response to the microphone output signal and the playback signal, each gap confidence value being for a different time t and indicating a confidence that a gap exists in the playback signal at time t; and generating an estimate of the background noise in the playback environment using the gap confidence value. system. [EEE11] The system of EEE10, wherein the noise estimation system is configured to generate the estimate of background noise in the playback environment such that the estimate of background noise in the playback environment is or comprises a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, and each of the noise estimates being a combination of candidate noise estimates weighted by the gap confidence value for different time intervals including time t. [EEE12] The sequence of noise estimates includes a noise estimate for each of the time intervals, and the noise estimation system generates a noise estimate for each of the time intervals by: (a) identifying each of the candidate noise estimates for the time interval for which a corresponding one of the gap confidence values ​​exceeds a predetermined threshold; (b) generating the noise estimate for the time interval to be the minimum of the candidate noise estimates identified in step (a). The system described in EEE11. [EEE13] Each of the candidate noise estimates is a minimum echo-canceled noise estimate, M resminand the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of the smallest echo cancelled noise estimate for that time interval weighted by a corresponding one of the gap confidence values ​​for that time interval. [EEE14] Each of the candidate noise estimates is a minimum microphone output signal value M min and the sequence of noise estimates comprises a noise estimate for each said time interval, the noise estimate for each said time interval being a combination of a minimum microphone output signal value for that time interval weighted by a corresponding one of gap confidence values ​​for that time interval. [EEE15] The gap confidence values ​​include a gap confidence value for each time t, and the noise estimation system generates the gap confidence values ​​for each time t by: processing the playback signal to determine a minimum in playback signal level for time t; processing the microphone output signal to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback signal level for time t differs from the smoothed level of the microphone output signal for time t. 8. A system as described in any one of claims EEE10 to 14. [EEE16] The estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, the system comprising: a noise compensation subsystem coupled to receive the sequence of noise estimates and configured to perform noise compensation on an audio input signal using the sequence of noise estimates to generate the reproduced signal. 8. A system as described in any one of claims EEE10 to 16. [EEE17] said noise estimation system comprising: performing a time domain to frequency domain transform on the microphone output signal, thereby generating frequency domain microphone output data; generating frequency domain playback content data in response to the playback signal; and configured to generate the gap confidence value in response to the frequency domain microphone output data and the frequency domain playback content data. 8. A system as described in any one of claims EEE10 to 16.

Claims

1. 13. A method of audio processing comprising: receiving a microphone output signal from a microphone of a playback environment, the microphone output signal corresponding to playback content being played by one or more loudspeakers and detected by the microphone, the microphone output signal also corresponding to background noise in the playback environment detected by the microphone; receiving a playback content value corresponding to the playback content; generating gap confidence values ​​in response to the microphone output signal and the played content values, each gap confidence value being for a different time t, indicating a confidence that a gap exists in the played content at time t, a gap representing a time or time interval during which there is no played content or has a level below a predetermined threshold, generating the gap confidence values ​​comprising: generating a gap confidence value for each time t; determining a minimum value in said playback content value for time t; processing the microphone output signal to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback content values ​​for time t differs from the smoothed level of the microphone output signal for time t; and generating an estimate of the background noise in the playback environment using the gap confidence value. method.

2. 2. The method of claim 1, wherein the microphone output signal and the playback content values ​​are frequency banded, and the gaps represent times or time intervals during which playback content is absent or has a level below a predetermined threshold in one or more frequency bands.

3. The method of claim 2 , wherein the gap confidence value is determined for at least one of the one or more frequency bands.

4. determining a plurality of gap confidence values ​​for each frequency band of the one or more frequency bands; determining a gap health value based on the plurality of gap confidence values, the gap health value indicating how current a noise estimate is for each frequency band of the one or more frequency bands. The method according to claim 3.

5. The method of claim 4 , wherein the plurality of gap confidence values ​​comprises n most recent gap confidence values, n being an integer.

6. The method of claim 5 , wherein determining the gap health value comprises adding the n most recent gap confidence values ​​and dividing a resulting sum by n.

7. The method of claim 4 , further comprising identifying one or more frequency bands for which a noise estimate is updated based at least in part on the gap health value.

8. The method of claim 7 , wherein the identifying step includes identifying a first healthy frequency band having a gap health value greater than or equal to a gap health value threshold.

9. 10. The method of claim 8, further comprising evaluating one or more frequency bands, including at least one frequency band adjacent to the first healthy frequency band, to locate at least one unhealthy frequency band having a gap health value less than the gap health value threshold.

10. The method of claim 9 , further comprising identifying a second healthy frequency band having a gap health value greater than or equal to the gap health value threshold.

11. The method of claim 10 , further comprising: calculating a noise estimate for at least one unhealthy frequency band between the first healthy frequency band and the second healthy frequency band.

12. 12. The method of claim 11, wherein the step of calculating the noise estimate comprises performing a linear interpolation between the noise estimate for the first healthy frequency band and the noise estimate for the second healthy frequency band.

13. The method of claim 12 , wherein the linear interpolation is performed in the logarithmic domain.

14. 2. The method of claim 1, wherein the estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, and each of the noise estimates being a combination of candidate noise estimates weighted by the gap confidence value for different time intervals including time t.

15. 2. The method of claim 1 , wherein the estimates of the background noise in the playback environment are or comprise a sequence of noise estimates, each of the noise estimates being an estimate of the background noise in the playback environment at a different time t, and generating the estimates of the background noise in the playback environment using the gap confidence values ​​comprises, for each noise estimate, weighting candidate noise estimates for different time intervals including time t by the gap confidence value and combining the weighted candidate noise estimates to obtain a respective noise estimate.

16. 2. The method of claim 1, wherein the estimate of the background noise in the playback environment is or comprises a sequence of noise estimates, and also comprising performing noise compensation on an audio input signal using the sequence of noise estimates.

17. 17. The method of claim 16, wherein performing noise compensation on the audio input signal includes generating the playback content, the method further comprising driving at least one speaker with the playback content to generate sound.

18. One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform a method, the method comprising: receiving a microphone output signal from a microphone of a playback environment, the microphone output signal corresponding to playback content being played by one or more loudspeakers and detected by the microphone, the microphone output signal also corresponding to background noise in the playback environment detected by the microphone; receiving a playback content value corresponding to the playback content; generating gap confidence values ​​in response to the microphone output signal and the played content values, each gap confidence value being for a different time t, indicating a confidence that a gap exists in the played content at time t, a gap representing a time or time interval during which there is no played content or has a level below a predetermined threshold, generating the gap confidence values ​​comprising: generating a gap confidence value for each time t; determining a minimum value in said playback content value for time t; processing the microphone output signal to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback content values ​​for time t differs from the smoothed level of the microphone output signal for time t; and generating an estimate of the background noise in the playback environment using the gap confidence value. One or more non-transitory media.

19. 20. The one or more non-transitory media of claim 18, wherein the method also includes performing noise compensation on an audio input signal based at least in part on the estimate of background noise.

20. 1. An apparatus having an input system and a noise estimation subsystem, comprising: The input system comprises: receiving a microphone output signal from a microphone of a playback environment, the microphone output signal corresponding to playback content being played by one or more loudspeakers and detected by the microphone, the microphone output signal also corresponding to background noise in the playback environment detected by the microphone; receiving a playback content value corresponding to the playback content; The noise estimation subsystem: generating gap confidence values ​​in response to the microphone output signal and the played content values, each gap confidence value being for a different time t, indicating a confidence that a gap exists in the played content at time t, a gap representing a time or time interval during which there is no played content or has a level below a predetermined threshold, generating the gap confidence values ​​comprising: generating a gap confidence value for each time t; determining a minimum value in said playback content value for time t; processing the microphone output signal to determine a smoothed level of the microphone output signal for time t; by determining the gap confidence value for time t to indicate how much the minimum in playback content values ​​for time t differs from the smoothed level of the microphone output signal for time t; and generating an estimate of the background noise in the playback environment using the gap confidence value. Device.

Images