Multi-band limiter modes and noise compensation methods

Abstract

Some embodiments relate to: receiving a content stream including audio data; receiving at least one type of level adjustment indication related to playback of the audio data; and controlling the level of the input audio data based on the at least one type of level adjustment indication to produce level-adjusted audio data. Some examples relate to: determining a multi-band limiter configuration based at least in part on the type(s) of level adjustment indication; applying the multi-band limiter to the level-adjusted audio data to produce multi-band limited audio data; and providing the multi-band limited audio data to one or more audio reproduction transducers of an audio environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the following U.S. Provisional Application Nos.: 62 / 945,292 filed on December 9, 2019; 63 / 198,995 filed on November 30, 2020; 62 / 945,303 filed on December 9, 2019; 63 / 198,996 filed on November 30, 2020; 63 / 198,997 filed on November 30, 2020; 62 / 945,607 filed on December 9, 2019; 63 / 198,998 filed on November 30, 2020; and 63 / 198,999 filed on November 30, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present disclosure relates to systems and methods for noise compensation. Background Art

[0004] Audio and video devices (including but not limited to televisions and associated audio devices) are widely deployed. Some such devices are configured to implement noise compensation algorithms that attempt to compensate for noise within an environment. Although existing systems and methods for noise compensation provide benefits, improved systems and methods would still be desirable.

[0005] Symbols and terminology

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used in a broad sense to mean performing the operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0010] Throughout this disclosure, including in the claims, the terms "couples" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0011] As used herein, a "smart device" is an electronic device that can operate interactively and / or autonomously to some extent, and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc. Several notable types of smart devices are smart phones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablet computers, smart watches, smart bracelets, smart key chains, and smart audio devices. The term "smart device" may also refer to a device that exhibits some properties of ubiquitous computing such as artificial intelligence.

[0012] In this document, the expression "smart audio device" is used to represent a smart device, which is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of the virtual assistant functionality). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to implement a single purpose. For example, although a TV can generally play (and is considered to be able to play) audio from program material, in most cases, modern TVs run some kind of operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device with (multiple) speakers and (multiple) microphones is typically configured to run local applications and / or services to directly use (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined together to enable audio to be played on a certain area or user-configured area.

[0013] A common type of multi-purpose audio device is an audio device that implements at least some aspects of the virtual assistant function, although other aspects of the virtual assistant function can be implemented by one or more other devices such as one or more servers, wherein the multi-purpose audio device is configured to communicate with the one or more servers. Such a multi-purpose audio device may be referred to as a "virtual assistant" in this article. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant can provide the ability to use multiple devices (different from the virtual assistant) for applications that support the cloud in a sense or are otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant function (e.g., speech recognition function) can be (at least partially) implemented by one or more servers or other devices, and the virtual assistant can communicate with one or more servers or other devices via a network (such as the Internet). Virtual assistants can sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants can work together in the sense that one of them (e.g., the virtual assistant that is most confident that it has heard the wake-up word) responds to the wake-up word. In some implementations, the connected virtual assistants may form a constellation that may be managed by a host application, which may be (or implement) the virtual assistant.

[0014] In this document, "wake-up word" is used in a broad sense to refer to any sound (e.g., a human-speaking word or other sound) that the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some instances, the so-called "wake-up word" in this document can include more than one word, for example, a phrase.

[0015] In this document, the expression "wake-up word detector" refers to a device configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model (or refers to software including instructions for configuring the device to continuously search for alignment between real-time sound features and a trained model). Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, computationally enhanced recognizer.

[0016] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some instances a collection of video signals, at least a portion of which is intended to be heard together. Examples include music selections, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from intelligent assistants, and the like. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same conversation in more than one language. In such instances, only one version of the audio data or a portion thereof is intended to be reproduced at a time (e.g., a version corresponding to a single language). Summary of the invention

[0017] At least some aspects of the present disclosure may be implemented via one or more audio processing methods including, but not limited to, content stream processing methods. In some instances, the method(s) may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods involve receiving a content stream comprising input audio data by a control system and via an interface system. Some such methods involve receiving at least one type of level adjustment indication related to playback of audio data by a control system and via an interface system. Some such methods involve controlling the level of input audio data by a control system based on at least one type of level adjustment indication to produce level-adjusted audio data. Some such methods involve determining a multi-band limiter configuration by a control system at least in part based on at least one type of level adjustment indication. Some such methods involve configuring a multi-band limiter by a control system according to a multi-band limiter configuration. Some such methods involve applying a multi-band limiter to the level-adjusted audio data to produce multi-band limited audio data. Some such methods involve providing multi-band limited audio data to one or more audio reproduction transducers of an audio environment.

[0018] According to some embodiments, at least one type of level adjustment indication may include a user input level adjustment indication received via a user input and / or a noise compensation level adjustment indication received from a noise compensation module. In some examples, if receiving at least one type of level adjustment indication involves receiving a user input level adjustment indication, determining the multi-band limiter configuration may involve determining a timbre retention configuration. According to some examples, if receiving at least one type of level adjustment indication involves receiving a noise compensation level adjustment indication, determining the multi-band limiter configuration may involve changing a timbre retention function. In some instances, changing the timbre retention function may involve at least partially disabling the timbre retention function. According to some examples, the noise compensation level adjustment indication may correspond to an ambient noise level in an audio environment. In some such examples, changing the timbre retention function may involve changing the timbre retention function based at least in part on the ambient noise level. In some instances, changing the timbre retention function may be based at least in part on the ambient noise level. Some examples may involve reproducing multi-band limited audio data on one or more audio reproduction transducers of an audio environment to provide reproduced audio data. Some such examples may involve determining or estimating the masking effect of the ambient noise level on the reproduced audio data. In some instances, changing the timbre-preserving function may be based at least in part on a masking effect.In some examples, the timbre-preserving configuration may be frequency band dependent.

[0019] In some instances, receiving at least one type of level adjustment indication may involve receiving both a user input level adjustment indication and a noise compensation level adjustment indication. In some such examples, determining a multi-band limiter configuration may involve determining a timbre-preserving configuration based at least in part on a weighted average of the user input level adjustment indication and the noise compensation level adjustment indication.

[0020] Some examples may also involve causing a noise compensation module operation change when multi-band limited audio data causes one or more audio reproduction transducers of an audio environment to operate outside a linear range. In some instances, a control system may cause one or more audio reproduction transducers of an audio environment to operate outside a linear range based at least in part on a noise compensation level adjustment indication and / or a noise estimate. In some examples, the noise compensation module operation change may involve changing an echo canceller function of the noise compensation module. According to some examples, the noise compensation module operation change may involve causing the noise compensation module to use only a quiet playback interval as an input to a noise estimator of the noise compensation module. A quiet playback interval may be an instance of an audio signal at or below a threshold level in at least one frequency band and / or at least one time interval. In some instances, multi-band limited audio data that causes one or more audio reproduction transducers of an audio environment to operate outside a linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

[0021] According to some embodiments, the noise compensation module can be a subsystem of the control system. In some examples, the level adjuster module of the control system can be configured to control the level of the input audio data to produce the audio data after the level adjustment. According to some such examples, a method can also involve providing multi-band limiter feedback to the level adjuster module from the multi-band limiter. In some instances, the multi-band limiter feedback can indicate the amount of restriction applied by the multi-band limiter to each band in the multiple bands of the audio data after the level adjustment. Some examples can also involve controlling the level of one or more bands in the multiple bands by the level adjuster module at least in part based on the multi-band limiter feedback.

[0022] Some alternative aspects of the present disclosure may also be implemented via one or more audio processing methods including, but not limited to, content stream processing methods. In some instances, the method(s) may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods involve: receiving a content stream including input audio data by a control system and via an interface system; and applying a multi-band limiter to the audio data or a processed version of the audio data by the control system to produce multi-band limited audio data. Some such methods involve: determining whether the multi-band limited audio data will cause one or more audio reproduction transducers to operate outside a linear range when played back on one or more audio reproduction transducers of an audio environment; and controlling by the control system whether an acoustic echo canceller updates one or more filter coefficients or a noise estimator updates a noise estimate based at least in part on whether the multi-band limited audio data will cause one or more audio reproduction transducers of an audio environment to operate outside a linear range. Some such methods involve providing multi-band limited audio data to one or more audio reproduction transducers of an audio environment.

[0023] In some examples, controlling whether the acoustic echo canceller updates one or more filter coefficients may involve controlling the acoustic echo canceller not to update one or more filter coefficients if the multi-band limited audio data causes one or more audio reproduction transducers of the audio environment to operate outside a linear range. Some embodiments may involve: receiving at least one type of level adjustment indication related to playback of audio data by a control system; determining a multi-band limiter configuration by the control system and at least in part based on at least one type of level adjustment indication; and configuring the multi-band limiter by the control system according to the multi-band limiter configuration. Some such examples may also involve controlling the level of input audio data by the control system based on at least one type of level adjustment indication to produce level-adjusted audio data. Applying the multi-band limiter to the audio data or a processed version of the audio data may involve applying the multi-band limiter to the level-adjusted audio data. In some examples, at least one type of level adjustment indication may include a user input level adjustment indication received via a user input and / or a noise compensation level adjustment indication received from a noise compensation module including the acoustic echo canceller. According to some examples, if receiving at least one type of level adjustment indication involves receiving a user input level adjustment indication, determining the multi-band limiter configuration may involve determining a timbre preservation configuration.

[0024] According to some embodiments, if receiving at least one type of level adjustment indication involves receiving a noise compensation level adjustment indication, determining the multi-band limiter configuration may involve changing the timbre retention function. In some instances, changing the timbre retention function may involve at least partially disabling the timbre retention function. In some examples, the noise compensation level adjustment indication may correspond to the ambient noise level in the audio environment. In some such examples, changing the timbre retention function may involve changing the timbre retention function based at least in part on the ambient noise level. Some examples may also involve reproducing multi-band limited audio data on one or more audio reproduction transducers of the audio environment to provide reproduced audio data. Some such examples may also involve determining or estimating the masking effect of the ambient noise level on the reproduced audio data. Changing the timbre retention function may be based at least in part on the masking effect. In some examples, the timbre retention configuration may be related to the frequency band.

[0025] In some examples, receiving at least one type of level adjustment indication may involve receiving both a user input level adjustment indication and a noise compensation level adjustment indication. In some such examples, determining a multi-band limiter configuration may involve determining a timbre-preserving configuration based at least in part on a weighted average of the user input level adjustment indication and the noise compensation level adjustment indication.

[0026] According to some embodiments, the control system may cause one or more audio reproduction transducers of the audio environment to operate outside of a linear range based at least in part on a noise compensation level adjustment indication and / or a noise estimate. In some such examples, the control system may cause one or more audio reproduction transducers of the audio environment to operate outside of a linear range based at least in part on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

[0027] Some examples may involve causing one or more additional noise compensation module operational changes when multi-band limited audio data causes one or more audio reproduction transducers of an audio environment to operate outside a linear range.(Multiple) additional noise compensation module operational changes may involve causing the noise compensation module to use only quiet playback intervals as inputs to a noise estimator of the noise compensation module. The quiet playback intervals may be instances of audio signals at or below a threshold level in at least one frequency band and / or instances of audio signals at or below a threshold level during at least one time interval. In some embodiments, the noise compensation module may be a subsystem of a control system.

[0028] According to some examples, the level adjuster module of the control system can be configured to control the level of the input audio data to produce the audio data after level adjustment. Some such examples can also involve providing multi-band limiter feedback to the level adjuster module from the multi-band limiter. The multi-band limiter feedback can, for example, indicate the amount of restriction applied by the multi-band limiter to each of the multiple frequency bands of the audio data after level adjustment. Some such examples can also involve controlling the level of one or more frequency bands in the multiple frequency bands by the level adjuster module at least in part based on the multi-band limiter feedback.

[0029] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Accordingly, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0030] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices may be capable of at least partially performing the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gates or transistor logic, discrete hardware components, or a combination thereof.

[0031] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative sizes of the following drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1A , Figure 1B and Figure 1C An example of a loudspeaker operating in the linear range is shown.

[0033] Figure 1D , Figure 1E and Figure 1F An example of a loudspeaker operating in a non-linear range is shown.

[0034] Figure 1G , Figure 1H and Fig. 1I An example of a multi-band limiter operation is shown.

[0035] Figure 1J , Figure 1K and Figure 1L An example of the operation of a timbre-preserving multi-band limiter according to one implementation is shown.

[0036] Figure 1M An example of a noise compensation system is shown.

[0037] Figure 1N A portion of a noise compensation system is shown, the portion including an example of a level adjuster configured to modify level adjusted audio data based on a compression feedback signal from a multi-band limiter.

[0038] Fig.1O According to an example Figure 1N A more detailed version of .

[0039] Figure 2A An example of a graphical user interface (GUI) for setting thresholds and other parameters of a multi-band limiter is shown.

[0040] Figure 2B Another example of thresholds and other parameters for a multi-band limiter is shown.

[0041] Figure 2C is a graph showing an example of threshold values ​​for a range of frequencies.

[0042] Figure 2D is a graph showing another example of threshold values ​​for a range of frequencies.

[0043] Figure 2E is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0044] Figure 3 is a flow chart outlining an example of the disclosed method.

[0045] Figure 4A and Figure 4B An example of a timbre-preserving modifier module is shown.

[0046] Figure 4C An example of a frequency band isolation modifier is shown.

[0047] Figure 5 is a flow chart outlining an example of another disclosed method.

[0048] Fig. 6A is a graph showing an example of time intervals during which a loudspeaker is overdriven.

[0049] Figure 6B shows the data that can be sent to Fig. 6AAn example of a graph corresponding to a signal of an echo canceller.

[0050] Figure 7 An example of a system configured to control an automatic echo canceller (AEC) based at least in part on the amount of "overdrive" occurring in the system is shown.

[0051] Figure 8 An example of a system configured to determine an amount of overdrive is shown.

[0052] Fig. 9 is a graph of noise estimation and output level according to an example.

[0053] Fig.10 An example of a floor plan of an audio environment, which in this example is a living space, is shown.

[0054] Throughout the various drawings, like reference numbers and designations refer to like elements. DETAILED DESCRIPTION

[0055] The noise compensation algorithm is designed to compensate for the noise in the so-called "audio environment" herein. As used herein, the term "audio environment" is not limited to components of an audio system, such as an audio reproduction transducer, an amplifier, etc. Alternatively, the term "audio environment" generally refers to an environment in which such components may reside and / or an environment in which one or more listeners may listen to playback audio. In some examples, the audio environment may be a home audio environment. In such an instance, the audio environment may correspond to one or more rooms in a home. In other examples, the audio environment may be an office environment, an automobile environment, a train or bus environment, a street or sidewalk environment, a park or other outdoor environment, or other types of environments.

[0056] Noise compensation methods may, for example, be designed to compensate for noise within an audio environment by adjusting the level of an output signal through one or more audio reproduction transducers based at least in part on the amount of noise within the audio environment. One of the challenges in implementing noise compensation methods is that audio reproduction transducers have a limited range over which they operate linearly in the physical world.

[0057] Figure 1A , Figure 1B and Figure 1C An example of loudspeaker operation in the linear range is shown. Figure 1A is a graph providing an example of a signal 100 output by an amplifier and provided to a loudspeaker 101 . Figure 1BAn example of a cross section of a loudspeaker 101 is shown. In this example, the signal 100 is a sinusoidal input signal having a peak-to-peak range of 10 volts. A signal within this range, when provided to the loudspeaker 101, causes a range of voice coil displacements, wherein the voice coil 102 remains within a portion of the magnetic field strength generated by the magnet 103 where uniform or substantially uniform field strength is present. Thus, the displacements of the voice coil 102 and diaphragm 104 are within a linear range.

[0058] Figure 1C is a graph showing an example of the displacement of voice coil 102 and diaphragm 104 when loudspeaker 101 is driven by signal 100. In this example, the displacement is sinusoidal and proportional to the input voltage of signal 100. Therefore, signal 100 is not distorted and loudspeaker 101 operates in a linear range.

[0059] When an audio device, such as a reproduction transducer, operates outside of a linear range, the audio device may be distorted and may sound unpleasant. Figure 1D , Figure 1E and Figure 1F An example of a loudspeaker operating in a non-linear range is shown. Figure 1D is a graph providing another example of a signal 100 output by an amplifier and provided to a loudspeaker 101 . Figure 1E Another example of a cross section of a loudspeaker 101 is shown. In this example, signal 100 is a sinusoidal input signal having a 20 volt peak-to-peak range. Signals within this range, when applied to loudspeaker 101, cause a range of voice coil displacements where voice coil 102 is not always held within a portion of the magnetic field generated by magnet 103 where the field strength is uniform or substantially uniform. Thus, the displacements of voice coil 102 and diaphragm 104 are within a nonlinear range. Another source of nonlinearity in loudspeaker 101 is the elasticity of the suspension. The periphery of diaphragm 104 is connected to the frame by suspension 105. As diaphragm 104 moves forward and backward, suspension 105 stretches to accommodate this movement. However, there is a limit to the elasticity of this material, and as diaphragm 104 is driven more forcefully and the movement of diaphragm 104 approaches the extremes of this limit, the driving force causing the movement is less able to overcome the opposing force of the elastic suspension 105 in its attempt to return to its neutral position. As with magnetic nonlinearity, this condition also results in the output signal no longer being the same as the input.

[0060] Figure 1F is a graph showing an example of the displacement of the voice coil 102 and the diaphragm 104 when the loudspeaker 101 is driven by the signal 100. In this example, the displacement is non-sinusoidal and not proportional to the input voltage of the signal 100. Figure 1FThe output shown is not proportional to the input voltage, but corresponds to both the input voltage and the loudspeaker distortion effects.Thus, in this example, signal 100 is distorted and loudspeaker 101 is operating in a non-linear range.

[0061] Echo cancellers, typically implemented via adaptive linear filters or via machine learning (e.g., via trained neural networks), are essential components of many noise compensation systems. If the echo canceller adapts completely successfully, the echo canceller will perform worse when adapting to nonlinear systems (e.g., in response to a loudspeaker operating in a nonlinear range). Furthermore, extended loudspeaker operation in a nonlinear range is likely to cause damage to the loudspeaker.

[0062] Multi-band limiters allow frequency-dependent control of the dynamic range of a loudspeaker. Multi-band limiters are typically configured to increase the sound pressure that a loudspeaker can produce while ensuring that the loudspeaker does not introduce non-linear distortion.

[0063] Figure 1G , Figure 1H and Fig. 1I An example of a multi-band limiter operation is shown. Figure 1G is a graph providing an example of an audio signal 100 provided to a multi-band limiter. In this example, the audio signal 100 corresponds to a "white" input spectrum, where the level of each frequency band is the same.

[0064] Figure 1H An example of a multi-band limiter threshold for each of a plurality of frequency bands is shown. In this example, each threshold is at or below the level of the input audio signal 100. For example, the multi-band limiter thresholds may correspond to the capabilities (e.g., distortion curve) of a particular loudspeaker and may be implemented by or for the loudspeaker. In some such examples, the multi-band limiter thresholds may be pre-set at the factory where the loudspeaker is manufactured.

[0065] Fig. 1I is shown when provided Figure 1G The input audio signal 100 shown has Figure 1H 1. A graph of an example of the output of a multi-band limiter with thresholds shown. In this example, the output of each band corresponds to the multi-band limiter threshold for that band because each threshold is at or below the level of the input audio signal 100. Therefore, in this example, the frequency content or timbre of the input audio signal 100 is not preserved.

[0066] As shown in the previous examples, a multi-band limiter may significantly change the spectral content or timbre of the input audio signal. Allowing the multi-band limiter to operate unconstrained may adversely affect the timbre of the output audio signal, thereby potentially reducing the enjoyment of the musical content.

[0067] Some multi-band limiters developed by the present assignee can at least partially preserve the timbre of an input audio signal. Figure 1J , Figure 1K and Figure 1L An example of the operation of a timbre-preserving multi-band limiter according to one embodiment is shown. The term "timbre-preserving" can have various meanings as used herein. Broadly speaking, a "timbre-preserving" method is a method of at least partially preserving the frequency content or timbre of an input audio signal. Some timbre-preserving methods can completely or almost completely preserve the frequency content of the input audio signal. The timbre-preserving method can involve constraining the output signal levels of at least some frequency bands according to the output signal levels of at least some other frequency bands and / or the applied thresholds. In some examples, the "timbre-preserving" method can involve constraining the output signal levels of all non-isolated frequency bands at least to some extent. However, as described in more detail herein, in some examples, the frequency bands can be completely isolated, while in other examples, the frequency bands can be only partially isolated.

[0068] Figure 1J is a graph providing an example of an audio signal 100 provided to a multi-band limiter. In this example, Figure 1G As shown, the level of the audio signal 100 is the same for each frequency band.

[0069] Figure 1K An example of a multi-band limiter threshold for each of the plurality of frequency bands is shown. In this example, each threshold is at or below the level of the input audio signal 100. For example, the multi-band limiter threshold may correspond to the capabilities of a particular loudspeaker and may be implemented by or for that loudspeaker.

[0070] Figure 1L is shown when provided Figure 1J The input audio signal 100 shown has Figure 1K Graph of an example of the output of a timbre-preserving multi-band limiter with respect to the thresholds shown. In this example, the output of each band does not correspond to the multi-band limiter threshold for that band. Alternatively, the timbre of the audio signal 100 is preserved by constraining the output signal level of each band to the minimum multi-band limiter threshold. This example shows an extreme case of a multi-band limiter with 100% timbre preservation. In most embodiments, timbre preservation is not so extreme. For example, some timbre-preserving embodiments may preserve the timbre in bands of a higher frequency range while allowing at least partial isolation of some bass bands. Alternatively or additionally, the timbre-preserving method may involve constraining (at least to some extent) the output signal level of a band based on less than the output signal level of all other bands and / or the thresholds applied.

[0071] Figure 1M An example of a noise compensation system is shown. In this example, the noise compensation system 150 includes a level adjuster 152, a multi-band limiter 154, a loudspeaker 156, a microphone 157, and a noise estimator 159. According to this example (and other examples of noise compensation systems disclosed herein, including but not limited to Figure 1N and Fig.1O ), the level adjuster 152, the multi-band limiter 154 and the noise estimator 159 are implemented by the control system 110, which can be the following reference Figure 2E 10. According to some examples, microphone 157 may also be controlled by and / or comprise a portion of control system 110. As noted elsewhere herein, control system 110 may reside in a single device or in multiple devices, depending on the particular implementation. In some examples, level adjuster 152, multi-band limiter 154, and / or noise estimator 159 may be implemented (at least in part) via software, for example, according to instructions stored on one or more non-transitory media.

[0072] As with other figures in this disclosure, Figure 1M The types, quantities, and arrangements of the elements shown in the drawings are examples only. Other embodiments may include more, fewer, and / or different elements. For example, some embodiments may include multiple audio reproduction transducers. Some embodiments may include multiple microphones.

[0073] According to this embodiment, the noise compensation system 150 includes a microphone 157, which is configured to detect the sound in the audio environment including the noise compensation system 150 and provide a corresponding microphone signal 158 to the noise estimator 159. The sound may include the sound produced by the loudspeaker 156 and the surrounding noise in the audio environment (which may also be referred to as ambient noise or background noise in this article). As pointed out elsewhere in this document, the term "audio environment" is not intended to be limited to the components of the audio system, such as audio reproduction transducers, amplifiers, etc. Alternatively, the term "audio environment" generally refers to the environment in which such components may exist and / or the environment in which one or more listeners may listen to the playback audio. In some examples, the audio environment may be a home audio environment. In such an instance, the audio environment may correspond to one or more rooms in the home. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc.

[0074] In this example, the noise estimator 159 is configured to estimate the level of background noise. According to this example, the noise estimator 159 is configured to implement an echo canceller to reduce the possibility that the audio data reproduced by the microphone 156 is part of the background noise estimate. In this example, the noise estimator 159 is configured to receive the multi-band limited audio data 155 output by the multi-band limiter 154, which is also provided to the microphone 156. The multi-band limited audio data 155 is an example of what is referred to herein as a "speaker reference", "microphone reference" or "echo reference" for the echo canceller implemented by the noise estimator 159. Implementing the echo canceller in the noise estimator 159 can prevent a positive feedback loop based on the sound generated by the microphone 156. In this example, the noise estimator 159 is configured to calculate a noise estimate of the surrounding noise and provide a noise estimator output 160 to the level adjuster 152. In some examples, the noise estimator output 160 will include a spectral noise estimate. For example, the noise estimator output 160 may include a noise estimate for each of a plurality of frequency bands.

[0075] In this example, a level adjuster 152 is shown as receiving input audio data 151. In some instances, the input audio data 151 may correspond to a content stream including video data. Here, the level adjuster 152 is configured to control (e.g., increase, decrease, or maintain) the level of the input audio data 151. According to this example, the level adjuster 152 is configured to control the level of the input audio data 151 based at least in part on the noise level that has been measured using the microphone 157. According to some examples, for example, the level adjuster 152 is configured to control the level of the input audio data 151 based at least in part on the noise estimator output 160. Therefore, the noise estimator output 160 is an example of what is referred to herein as a "level adjustment indication". More particularly, the noise estimator output 160 is an example of what is referred to herein as a "noise compensation level adjustment indication".

[0076] In this example, the level adjuster 152 is shown as receiving a user input 163 corresponding to a level adjustment, which is another example of what is referred to herein as a level adjustment indication. More specifically, the user input 163 is an example of what is referred to herein as a "user input level adjustment indication". It will be understood that the level adjuster 152 will not normally continuously receive the user input 163, but will typically only intermittently receive the user input 163 during a user's attempt to adjust the audio playback level by, for example, providing input via a voice command (e.g., a voice command received by the control system 110 via microphone 157), via a manual remote control, etc. The level adjuster 152 (or another element of the control system 110) may, for example, store a value in a memory device corresponding to the most recent user input 163. In this example, the level adjuster 152 is configured to control the level of the input audio data 151 based on at least one type of level adjustment indication. Here, the level adjuster 152 is configured to provide the level-adjusted audio data 153 to the multi-band limiter 154.

[0077] According to some examples, the level adjuster 152 can be configured to determine the noise compensation method based at least in part on the state of the noise estimator 159 and / or the user input 163. Therefore, in some embodiments, the level adjuster 152 can be configured to determine the noise compensation method based at least in part on the noise estimator output 160 and / or the user input 163.

[0078] In some examples, noise estimator 159 may determine which noise compensation method should be implemented by level adjuster 152. In some such examples, noise estimator output 160 may indicate to level adjuster 152 (e.g., via noise estimator output 160 and / or via additional information) which noise compensation method should be implemented by level adjuster 152.

[0079] In some embodiments where the noise estimator 159 is a multi-band noise estimator, if the noise estimate has a set of unupdated frequency bands (e.g., in the higher frequency bands) that have not been updated for a threshold amount of time (e.g., on the order of seconds, such as 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc.), the noise estimator output 160 can indicate that the noise compensation method should switch to a timbre-preserving mode because the quality of the noise estimate in the unupdated frequency bands is low, even though the quality of the noise estimate in the updated frequency bands may still be high. Alternatively or additionally, in some embodiments, the noise estimator can be configured to provide a quality indicator or confidence score to the noise compensation block, which the noise compensation block can use to determine which mode to be in (or partially in). For example, if the quality indicator or confidence score indicates that the quality of the noise estimate is low, the noise compensation block can determine that the noise compensation method should be a timbre-preserving mode.

[0080] In some embodiments, the control system 110 (e.g., the noise estimator 159) can be configured to provide a multi-band noise estimator functionality as described in International Publication No. WO2019 / 209973, filed on April 24, 2019, entitled “Background Noise Estimation Using Gap Confidence,” and in particular, the gap confidence values ​​and the use of gap confidence values ​​are discussed on pages 16-18, which is incorporated herein by reference.

[0081] According to some embodiments, in the timbre-preserving frequency range of the noise compensator, the multi-band limiter, or both (e.g., described below), Figure 2B A frequency band within the frequency range of a non-isolated frequency band may be selected based on a quality indicator of the noise estimate. For example, the quality indicator of the noise estimate may correspond to the amount of time since the noise estimate for the frequency band was updated.

[0082] In some examples, the gain applied in one frequency band (e.g., by a noise compensator) may be unconstrained compared to the gain applied in another frequency band (e.g., relative to the gain applied in an adjacent frequency band). Thus, according to this type of noise compensation method, the spectral content of the input audio signal is generally not preserved. Thus, this type of noise compensation method may be referred to herein as an "unconstrained" noise compensation method or a non-timbre-preserving noise compensation method.

[0083] According to some examples, the multi-band limiter 154 can be configured to apply compression to the level-adjusted audio data 153 based at least in part on a previous calibration or "adjustment" of the multi-band limiter 154 to prevent distortion in the loudspeaker 156 (and in some instances, other audio reproduction transducers of the audio environment). In some such examples, the multi-band limited audio data 155 does not produce distortion in the loudspeaker 156, so that the loudspeaker 156 operates within a linear range. The adjustment can correspond to a multi-band limiter threshold for each of the multiple frequency bands. For example, the multi-band limiter threshold can correspond to the capabilities (e.g., distortion curve) of the loudspeaker 156 and can be implemented by or for the loudspeaker. In some examples, the multi-band limiter threshold can be pre-set at the factory where the loudspeaker 156 is manufactured.

[0084] However, in some examples, the multi-band limiter 154 can be configured to apply compression to the audio data 153 after level adjustment, which allows at least some distortion in the loudspeaker 156 and / or one or more other audio reproduction transducers of the audio environment. In such examples, one or more audio reproduction transducers of the audio environment may be allowed to operate outside the linear range at least temporarily. In some such examples, the control system can cause one or more audio reproduction transducers of the audio environment to operate outside the linear range based at least in part on the noise compensation level adjustment indication and / or noise estimation (e.g., the noise estimator output 160 from the noise estimator 159). Some such examples can also involve causing the noise compensation module operation to change when the multi-band limited audio data 155 causes one or more audio reproduction transducers of the audio environment to operate outside the linear range. Some detailed examples are disclosed elsewhere in this document.

[0085] According to some embodiments, the control system 110 (e.g., the level adjuster 152) can be configured to determine the multi-band limiter configuration based at least in part on one or more types of level adjustment instructions received. In some such examples, if a user input level adjustment instruction is received, the multi-band limiter configuration can be a timbre preservation configuration. Various examples of timbre preservation configurations are disclosed herein. In some such examples, determining the multi-band limiter configuration can involve changing the timbre preservation function of the multi-band limiter 154 if a noise compensation level adjustment instruction is received.

[0086] In some such examples, the control system 110 (e.g., the level adjuster 152 or the multi-band limiter 154 itself) can configure the multi-band limiter 154 according to the determined multi-band limiter configuration. In some embodiments, the control system 110 (e.g., the level adjuster 152) can be configured to control the multi-band limiter 154 and / or the noise estimator 159 based at least in part on the output level and / or capability of the loudspeaker 156.

[0087] according to Figure 1M In the example shown, the level adjuster 152 is configured to control the multi-band limiter 154 via the control signal 161. In some implementations, if the received level adjustment indication is a user input level adjustment indication, the timbre preservation setting of the multi-band limiter 154 can be maintained, for example, at the factory after the initial adjustment of the multi-band limiter 154. Such implementations can help ensure that the user has a pleasant experience when the user adjusts the volume control.

[0088] According to some examples, if the received level adjustment indication is a noise compensation level adjustment indication, the timbre preservation setting of the multi-band limiter 154 can be gradually closed, for example, in proportion to the noise compensation level adjustment indication. In some such examples, when noise is present, the played back audio content can still be understood under the noise while the loss of fidelity is masked by the noise source.

[0089] In some implementations, if the control system 110 (e.g., the level adjuster 152) implements the timbre-preserving noise compensation mode, the level adjuster 152 can notify the multi-band limiter 154 (e.g., via the control signal 161) so that the multi-band limiter 154 also operates in the timbre-preserving mode and does not allow the loudspeaker 156 to be overdriven. According to some examples, if the control system 110 implements the unconstrained noise compensation mode, the multi-band limiter 154 can also operate in a relatively less constrained mode (e.g., as described below with reference to Figure 2A and Figure 2B ), in order to maximize the volume (eg, overdrive the loudspeaker 156).

[0090] Conversely, if the multi-band limiter 154 is limiting, in some embodiments, the control system 110 may cause the noise compensation mode to be an unconstrained noise compensation mode (even if the noise compensation mode was previously a timbre-preserving noise compensation mode) so that the volume may be maximized.

[0091] exist Figure 1M In the example shown, the multi-band limiter 154 is configured to send an optional compression feedback signal 162 to the level adjuster 152. For example, the compression feedback signal 162 may indicate an amount of limiting applied by the multi-band limiter 154 to the level adjusted audio data 153. In some examples, the compression feedback signal 162 may indicate an amount of limiting applied by the multi-band limiter 154 to each of the plurality of frequency bands of the level adjusted audio data 153.

[0092] Figure 1N A portion of a noise compensation system is shown, which includes an example of a level adjuster configured to modify the level-adjusted audio data based on a compression feedback signal from a multi-band limiter. In this example, the level adjuster 152 includes a bass enhancement module 167 and a noise compensation level adjuster 169. Figure 1NAlso indicated is a multi-channel stream 168 that has been processed by a bass enhancement module 167 and, in this example, provided to a noise compensation level adjuster 169. According to some examples, the bass enhancement module 167 is configured to propagate bass from one loudspeaker to one or more other loudspeakers. In some examples, the bass enhancement module 167 is configured to determine how much input bass should be propagated to other speakers based at least in part on the compression feedback signal 162.

[0093] In some embodiments, the bass enhancement module 167 is configured to implement psychoacoustic bass enhancement (e.g., virtual bass that exploits missing fundamental phenomena, such as described below with reference to Fig.1O According to some examples, the bass enhancement module 167 can be configured to determine how much virtual bass should be implemented based at least in part on the compression feedback signal 162.

[0094] In some embodiments, the bass enhancement module 167 is configured to receive the noise estimate 160. The bass enhancement module 167 can, for example, use the noise estimate 160 to control the bass propagation and / or the aggressiveness of the virtual bass. In some such examples, if the noise level is high, the bass enhancement module 167 will be relatively more aggressive in propagating the audio than if the volume limiting was caused by user input, even to the point where it is no longer just propagating bass frequencies but may be propagating most or all of the human audible spectrum (e.g., the bass enhancement module 167 will propagate frequencies higher than the bass frequencies to all loudspeakers). Further examples of "aggressiveness" are provided below. According to some embodiments, the bass enhancement module 167 can also begin introducing virtual bass processing earlier than if the volume limiting was caused entirely by user volume control.

[0095] Fig.1O According to an example Figure 1N A more detailed version of . Like other figures provided in this disclosure, Fig.1O The types, quantities, and arrangements of the elements shown in the examples are examples only. Other embodiments may include more, fewer, and / or different elements. In some alternative examples, the process flow may be similar to Fig.1O The differences shown in , for example, are where noise compensation is used to control bass boost.

[0096] exist Fig.1OIn the example shown, the level adjuster 152 includes an example of a bass enhancement module 178 and a noise compensation level adjuster 169. As noted elsewhere herein, according to some examples, the level adjuster 152 can be configured to determine a noise compensation method based at least in part on a state of the noise estimator 159 and / or a user input 163. Thus, in some embodiments, the noise compensation level adjuster 169 can be configured to determine a noise compensation method based at least in part on a noise estimator output 160 and / or a user input 163. Some such examples are described above with reference to Figure 1M Described in Figure 7 Additional examples are disclosed in FIG. 17 and in the corresponding description of U.S. Patent No. 8,090,120 (column 18, line 29 to column 26, line 46, and column 34, line 41 to column 35, line 11), which corresponding descriptions are incorporated herein by reference.

[0097] according to Fig.1O In the example shown in , the input audio data 151 is shown as two input channels (eg, stereo channels) CH1 and CH2. Fig.1O In the noise estimation 160 (which can be obtained by Figure 1M The compressed feedback signal 162 is provided to the noise estimator 159 of the noise compensation level adjuster 169. In this example, the compressed feedback signal 162 provided to the level adjuster 152 indicates the amount of limiting performed in the multi-band limiter 154. According to some examples, the compressed feedback signal 162 is optionally provided to both the bass enhancement module 178 and the noise compensation level adjuster 169. According to some such examples, the compressed feedback signal 162 may be provided to the virtual bass (VB) block 171. If the compressed feedback signal 162 is provided to the noise compensation level adjuster 169, then in some examples, the compressed feedback signal 162 may be used to modify the overall predicted output level used within the noise compensation level adjuster 169.

[0098] In some examples, the noise compensation level adjuster 169 optionally implements a psychoacoustic volume control. According to some such examples, instead of applying a broadband gain that causes the gain on all frequency bands to change by the same amount, which may result in a change in the perceived spectrum, a specific loudness scaling factor can be associated with the volume control adjustment. In some such examples, the gain in each of the multiple frequency bands is changed by an amount that takes into account the human hearing model so that the perceived spectrum does not change under ideal conditions. Some related examples are disclosed in the "Time-Invariant and Frequency-Invariant Function Suitable for Volume Control" section of U.S. Patent No. 8,090,120 (column 26, line 48 to column 28, line 13), which is incorporated herein by reference.

[0099] Some psychoacoustic volume control implementations may involve mapping between the digital / electrical domain to the acoustic domain (e.g., between decibels relative to full scale (dBFS) and decibels of sound pressure level (dBSPL)) that takes into account the amount of gain, etc. within downstream processing components. In some such examples, the psychoacoustic volume control may be calibrated in areas of the system where the multi-band limiter 154 is not activated. This means that when the multi-band limiter 154 is activated, the mapping of digital to sound pressure level (SPL) will generally be incorrect (because the mapping is generally a fixed mapping). By indicating to the level adjuster 152 (e.g., to the noise compensation level adjuster 169) that the multi-band limiter 154 is limiting, the mapping of digital to SPL can be corrected and, therefore, the amount of noise compensation required is not underestimated.

[0100] Fig.1O The block arrangement and order of operations shown in result from the fact that according to some examples of implementing psychoacoustic volume control by the noise compensation level adjuster 169, the noise compensation level adjuster 169 may incorporate a loudness curve of a psychoacoustic bass enhancement to control the volume level relative to the noise estimate. Therefore, performing the noise compensation block audio processing after the bass enhancement block audio processing requires less conversion from the acoustic domain to the digital domain, wherein the latter is the domain in which the psychoacoustic system according to this example will operate.

[0101] In addition, in some systems where the noise compensation level adjuster 169 runs out of gain headroom, other blocks of the level adjuster 152 (such as blocks 167 and / or 171) can be instructed by the noise compensation level adjuster 169 to increase the amount of processing they perform to ensure that the noise compensation system 150 can reach its peak loudness. The noise compensation level adjuster 169 running out of headroom can be indicated by the multi-band limiter 154 limiting and providing the compression feedback signal 162 to the noise compensation level adjuster 169. In some cases, the noise compensation level adjuster 169 can run out of headroom by compensating in response to high levels of noise. In addition, in some embodiments where the noise compensation level adjuster 169 controls other blocks of the level adjuster 152, the noise compensation level adjuster 169 can send a signal 161 to the multi-band limiter 154 indicating that the multi-band limiter 154 should stop operating in the timbre preservation mode and / or should allow one or more loudspeakers to be overdriven.

[0102] According to this example, the bass enhancement module 178 includes a bass extraction module 177, mixers 172 and 179, and a virtual bass (VB) block 171. In this example, the VB block 171 provides an output 168 to the noise compensation level adjuster 169. In some examples, each bass extraction module 177 can be implemented as a set of dynamic crossover filters that can be controlled at runtime. In some such examples, when the multi-band limiter 154 limits in the low frequency band, the bass (e.g., the corresponding low frequency band) can be extracted.

[0103] According to some such embodiments, the bass extraction module 177 may be configured to extract (from the input channels CH1 and CH2) high frequency content (high pass filtered signal 175) in a high frequency range above a crossover frequency and low frequency content (bass extracted audio 173) in a low frequency range below the crossover frequency. In some examples, the bass extraction module 177 may be configured to control the crossover frequency based at least in part on the compression feedback signal 162. The crossover frequency may be controlled by the amount of limiting performed in the multi-band limiter 154 (as indicated by the compression feedback signal 162). In some examples, the limiting may be only in the low frequency range (e.g., up to 500 Hz), but in alternative examples, the limiting may be in a wider frequency range or the full frequency range. In some examples, the compression feedback signal 162 may indicate the amount of compression applied by the multi-band limiter 154 in each of at least two low frequency bands within the low frequency range (e.g., up to 500 Hz) (and the crossover frequency may be determined by the amount of compression). Alternatively, the compression feedback signal 162 may indicate the amount of compression applied by the multi-band limiter 154 in a frequency band within a wider frequency range or the full frequency range (and the crossover frequency may be determined by the amount of compression).

[0104] In this example, bass extracted audio 173 from both input channels CH1 and CH2 has been low-pass filtered and provided to a mixer 172, which downmixes the bass extracted audio 173 into a single channel. According to this example, the mixer 172 provides the downmixed bass 174 to be propagated to both channels (in other words, mixed back to both channels) to a mixer 179. In this example, the mixer 179 mixes the downmixed bass 174 and the high-pass filtered signal 175 without bass extraction, and outputs a modified channel 170. According to some examples, the modified channel 170 has the bass propagated through both channels based on the multi-band limiter behavior and the noise estimation 160 (and / or based on the ratio of the gain corresponding to the user input 163 compared to the gain based on the noise compensation control).

[0105] In this example, the noise estimate 160 is optionally provided to the bass extraction module 177 and the VB block 171. In this embodiment, the bass enhancement module 178 also takes into account the noise estimate 160 (and / or the ratio of the system volume that has been controlled by the noise compensation level adjuster 169 compared to the volume corresponding to the user control). According to some examples, if the noise estimate 160 is high, the frequencies to be extracted will generally consist of a larger portion of the spectrum than when the noise estimate is low. According to some such examples, the bass enhancement module 178 can be configured to adjust what is referred to herein as the "aggressiveness" of the frequency extraction based on the noise estimate 160. As used herein, the term "aggressiveness" is a parameter that refers to the degree of bass volume enhancement.

[0106] In some such examples, bass extraction module 177 may be configured to determine the crossover frequency (“targeted_crossover” in the following formula) according to the following formula:

[0107] targeted_crossover=total_gain_ratio*max_freq_limiting*aggressiveness (Equation 1)

[0108] In Equation 1, “aggressiveness” represents a parameter indicating the aggressiveness of the bass volume enhancement. In some examples, the “aggressiveness” parameter can be adjusted by ear, for example, by a user or by a provider of a noise compensation system to ensure that the system does not include too much or too little energy in the downmixed bass 174. According to some examples, linear interpolation of the “aggressiveness” parameter can be used to fade between two “aggressiveness” settings (e.g., one high volume setting due to noise and another high volume setting due to user input).

[0109] In Equation 1, "max_freq_limiting" represents the maximum frequency covered by the frequency band limited in the multi-band limiter 154. In some examples, "max_freq_limiting" may be determined by or directly derived from the highest frequency in the highest frequency band limited by the multi-band limiter 154. In some implementations, "max_freq_limiting" may be clipped to a range supported by the bass extraction module 177.

[0110] In some examples,

[0111] "total_gain_ratio"=total_gain / max_possible_gain (Equation 2)

[0112] In Equation 2, "max_possible_gain" represents the sum of the maximum gains of each frequency band limited by the multi-band limiter 154 (at this time) for all frequency bands that can be subjected to bass extraction by the bass extraction module 177 (or in some embodiments, all frequency bands that can be limited in the multi-band limiter 154). In some examples, "max_possible_gain" may be the sum of the maximum gains that can be applied by the multi-band limiter 154 for all frequency bands that can be subjected to bass extraction, and in this sense, "max_possible_gain" may be the maximum integral of all gains that can be applied by the multi-band limiter 154 for bins having a frequency not exceeding the maximum crossover frequency.

[0113] In Equation 2, “total_gain” represents the sum of all gains applied to all frequency bands that may be bass extracted (or in some embodiments, all frequency bands that may be limited) (eg, as indicated by the compression feedback signal 162 for each frequency band).

[0114] In Equations 1 and 2, "total_gain_ratio" represents an indicator of the total amount of limiting by the multi-band limiter 154 across all frequency bands that may be subjected to bass extraction by the bass extraction module 177. In Equation 2, the "total_gain_ratio" is normalized (via the "max_possible_gain" parameter) so that the "total_gain_ratio" better indicates the total amount of limiting performed for a variable number of frequency bands.

[0115] In some embodiments, when the multi-band limiter 154 applies more limiting (e.g., when the "total_gain_ratio" in Equations 1 and 2 increases), the crossover frequency ("targeted_crossover" in Equation 1) of each of the filters 205 and 206 can be increased to increase the amount of bass enhancement applied. In some embodiments, when the multi-band limiter 154 applies less limiting (e.g., when the "total_gain_ratio" in Equations 1 and 2 decreases), the crossover frequency can be lowered to reduce the amount of bass enhancement applied. In some examples, the crossover frequency (e.g., "targeted_crossover" in Equations 1 and 2) can be smoothed through attack and release to ensure that the user does not notice the sudden jump in the panning.

[0116] According to this example, the VB module 171 creates a bass perception based on the missing fundamental phenomenon. According to some examples, the VB module 171 can be configured to create an enhanced bass perception by injecting a signal at a harmonic of a bass frequency within the harmonics of the input signal. In some such examples, the number of injected harmonics and the amplitude of the harmonics can be determined by both the corresponding compression feedback signal 162 and the noise estimate 160 (or the ratio of the volume controlled by the noise compensation level adjuster 169 to the volume corresponding to the user input). In some examples, if the noise estimate 160 is high, the virtual bass amount (e.g., the number of harmonics and their amplitude) will be increased (e.g., by adjusting the aggressiveness in equations 1 and 2) compared to the case where the noise estimate is low. In some embodiments, the virtual bass amount can be determined as follows:

[0117] virtual_bass_gains=min_virtual_bass_gain+((1+0.01×A) -limiter_gain -1)(Equation 3)

[0118] In Equation 3, "limiter_gain" represents the multi-band limiter gain value for the lowest band that the multi-band limiter 154 can provide to one or both of the VB modules 171. In Equation 3, "A" represents a parameter indicating the aggressiveness of the virtual bass application (e.g., how much virtual bass is applied per multi-band limiter gain amount). In one example, A=-25, but in alternative examples, A can be higher or lower. In Equation 3, "min_virtual_bass_gain" represents the minimum virtual bass gain amount that can be applied. According to some examples, linear interpolation of the "aggressiveness" parameter can be used to fade between two "aggressiveness" settings (e.g., one high volume setting due to noise and another high volume setting due to user input).

[0119] return Figure 1M , in which the level adjuster 152 is configured to send an optional control signal 164 to the noise estimator 159. In some embodiments, the multi-band limiter 154 can be configured to send an optional control signal 167 to the noise estimator 159. As described in more detail elsewhere herein, in some instances, the level adjuster 152 can configure the multi-band limiter 154 to allow some distortion (e.g., to increase the playback volume in the presence of ambient noise), causing one or more audio reproduction transducers of the audio environment to operate outside of a linear range. In such an instance, the audio path 165 from the loudspeaker 156 to the microphone 157 may include at least some sounds corresponding to the nonlinear distortion of the loudspeaker 156. At least in part because echo cancellers are typically based on linear algorithms, the echo canceller will not be able to properly cancel sounds corresponding to the nonlinear distortion of the loudspeaker 156.

[0120] Thus, according to some such examples, if the level adjuster 152 configures the multi-band limiter 154 to cause one or more audio reproduction transducers of the audio environment to operate outside the linear range, the level adjuster 152 may also send a control signal 164 (or the multi-band limiter 154 may also send a control signal 167) to the noise estimator 159, which indicates that the noise estimator 159 should change the operating mode. In some such examples, the control signal 164 (or the control signal 167) may indicate that the echo canceller implemented by the noise estimator 159 should only use the quiet playback intervals of the multi-band limited audio data 155 as input. In some such examples, the quiet playback interval may be an instance of an audio signal at or below a threshold level in one or more frequency bands. Alternatively or additionally, in some examples, the quiet playback interval may be an instance of an audio signal at or below a threshold level during a certain time interval. The quiet playback interval may also be referred to as a "gap" herein.

[0121] In some embodiments, if the level adjuster 152 configures the multi-band limiter 154 to cause one or more audio reproduction transducers of the audio environment to operate outside the linear range, the control signal 164 (or the control signal 167) can indicate that one or more functions of the echo canceller should be disabled or suspended. For example, the echo canceller can operate normally by updating the adaptive filter coefficients for each of the multiple frequency bands. In some such embodiments, if the multi-band limited audio data 155 will cause the loudspeaker 156 (or one or more other audio reproduction transducers of the audio environment) to operate outside the linear range, the control signal 164 (or the control signal 167) can control the acoustic echo canceller to not update the filter coefficients.

[0122] Figure 2A An example of a graphical user interface (GUI) for setting the thresholds and other parameters of a multi-band limiter is shown. The GUI 200 can be presented on a display device, for example, according to software executed by a control system. A user can interact with the GUI 200, for example, by providing user input via a touch screen on a display presenting the GUI 200, via a mouse, via a keyboard, via voice commands, etc. As with other figures presented herein, the types and quantities of the indicated elements and specific values ​​are provided only as examples.

[0123] In this embodiment, the y-axis indicates decibels in the range from 0dB to -60dB, and the x-axis indicates frequency in Hz. In this example, the GUI 200 shows a set of example threshold values ​​201 for each of the plurality of frequency bands 217. According to this example, each threshold value 201 is illustrated by a dot in a vertical line representing the corresponding frequency band 217. The center frequency of each frequency band in the frequency band 217 is indicated near the vertical line representing the frequency band 217. In some embodiments, the threshold value 201 is a level that a signal is not allowed to exceed in the corresponding frequency band. If the input level exceeds the threshold value 201, a negative gain that limits the level to the threshold value 201 can be applied.

[0124] The level of threshold 201 may be related to the maximum input value that still has a linear output response in the corresponding frequency band when reproduced by the audio reproduction transducer. Figure 2A The specific threshold 201 shown in can be preset according to the capabilities of the specific audio reproduction transducer. In this example, the threshold 201 is generally lower in the lowest frequency band. This indicates that the distortion level of the specific audio reproduction transducer at low frequencies is lower than that at high frequencies. In this example, frequencies above 844 Hz are not limited in terms of the maximum volume of this specific device.

[0125] According to this example, element 213 indicates an isolation setting for a particular frequency band. If a frequency band is set to be isolated, only the audio in that frequency band will have an effect on the limiting gain applied. Figure 2A In the example shown, no frequency bands are isolated.

[0126] Figure 2B Another example of thresholds and other parameters for a multi-band limiter is shown. Figure 2B An example of a GUI 200 is shown in which four elements 213 indicate that four corresponding bass bands are isolated. In some cases, it may be desirable to allow certain frequency bands, such as the bass bands, to operate in complete isolation without contributing to timbre preservation. For example, due to small speaker sizes, the fixed threshold for bass frequencies in some audio systems may be very low. If these bass bands were allowed to affect timbre preservation calculations, a dramatic reduction in the overall playback level would result. In this case, it may be desirable to allow these bass bands to operate independently (e.g., Figure 2B ), and applies a timbre-preserving method to the remaining frequency bands.

[0127] In some alternative embodiments, the frequency bands may be partially isolated (e.g., 25% isolated, 50% isolated, 75% isolated, etc.) rather than completely isolated or not isolated at all. In some such examples, the degree of isolation of the frequency bands may be selected via an alternative version of GUI 200 that includes a slider or other virtual user input device corresponding to one or more (e.g., each) of elements 213. In other examples, the degree of isolation of the frequency bands may be automatically changed due to changing conditions, such as a change in the ambient noise level in the audio environment. Figure 4C Some examples are described.

[0128] exist Figure 2A and Figure 2B In the example shown, dots 218 in some frequency bands indicate example adjustments for what is referred to herein as "overdrive," or operation outside the linear range of the audio reproduction transducer. Various examples are disclosed herein. Between dots 201 and 218, the audio reproduction transducer will not operate within the linear range. According to some examples, dots 218 represent hard limits beyond which the audio reproduction transducer will not be driven. In some alternative embodiments, dots 218 represent soft limits beyond which the audio reproduction transducer may be driven in some cases. In some embodiments, such explicit overdrive adjustments may be optional, and in some alternative examples, such explicit overdrive adjustments may not exist. In some embodiments, the explicit overdrive adjustments may be fixed values ​​across all frequency bands (e.g., 3dB added to threshold 201).

[0129] exist Figure 2A and Figure 2B In the example shown, element 212 allows a user (e.g., an employee of a device manufacturer) to select a timbre preservation setting. In these examples, the user can interact with element 212 by moving slider portion 212a and / or by entering a value in window portion 212b. In some implementations, the timbre preservation setting corresponds to the amount by which a signal in one frequency band can affect the gain applied to other frequency bands (e.g., one or more adjacent frequency bands, if the frequency bands are not isolated).

[0130] In some implementations, a timbre preservation setting of 1.00 may correspond to a time-varying threshold T b [n], the time-varying threshold is calculated for all frequency band signals x b [n] and all fixed thresholds L on all non-isolated frequency bands b Function of , b=1...B:

[0131] T b [n] = TPF({x i [n],L i|i=1...B})

[0132] For example, a fixed threshold L b It can correspond to the threshold 201. Then, the gain g of each frequency band is b [n] can be calculated as g b [n] = CF(x b [n],T b [n]).

[0133] For timbre preservation settings less than 1.00, each threshold T b [n] can be calculated as multiple but less than all frequency band signals x b [n] and / or multiple but less than all fixed thresholds L of non-isolated frequency bands b For example, in the timbre preservation setting of 0.50, each threshold T b [n] can be calculated as half of the frequency band signal x b [n] and fixed threshold L for non-isolated bands b function.

[0134] In some examples, the time-varying threshold of a frequency band may be calculated based on its nearest neighboring non-isolated frequency band or a range of neighboring non-isolated frequency bands.

[0135] In some examples, if a non-isolated frequency band receives a significant gain reduction due to being above its fixed threshold, the time-varying thresholds of other non-isolated frequency bands may also be lowered, thereby receiving some gain reduction. As the time-varying thresholds of the frequency bands are lowered below their corresponding fixed thresholds, the multi-band limiter 154 still reduces distortion while mitigating or otherwise preventing changes to the timbre.

[0136] In some examples, the control system (eg, multi-band limiter 154) may be configured to calculate the average difference of the audio input signal in each frequency band and its corresponding fixed threshold L b The time-varying threshold in each frequency band may then be the audio input signal level in that frequency band minus the average difference.

[0137] Alternatively or additionally, the time-varying threshold may be smoothed over time, at least as smooth as the gain g b [n] is smoother. That is, the level of the audio input signal used to calculate the threshold can be higher than that used to calculate the gain g b [n] signal (e.g., b [n]) is smoother. In one such example, a single-pole smoother with a longer time constant can be used to calculate a smoother energy signal s b [n]:

[0138]

[0139] In this case, 10 times the attack and release times of a conventional multi-band limiter can be used. The smoothed energy signal can then be expressed in dB as follows:

[0140] S b [n] = 10log 10 (s b [n])

[0141] The smoothed energy signal in each frequency band is compared with a fixed threshold L in each frequency band. b The difference (also expressed in dB) can be calculated as:

[0142] D b [n] = S b [n]-L b

[0143] And the minimum of these distances over all frequency bands can be found:

[0144]

[0145] The weighted average of these differences over all frequency bands can then be calculated as follows, where β represents the weighting factor:

[0146]

[0147] When β=1, the true mean of the differences is calculated, and when β>1, larger differences have a greater impact on the mean. In other words, the energy is much higher than the threshold L b In some examples, it has been found that β = 8 produces appropriate weighting. Finally, the threshold T b [n] can be calculated as the smoothed signal energy in the frequency band minus the mean difference - if this threshold is less than a fixed threshold L b Otherwise, according to some embodiments, the time-varying threshold may be kept equal to the fixed threshold, for example, as follows:

[0148]

[0149] In some alternative embodiments, the distance D b The threshold is calculated by taking the maximum value of [n] instead of the weighted average:

[0150]

[0151] Each threshold can then be calculated as the smoothed signal energy in the band minus the maximum distance plus some tolerance value D tol - If the threshold is less than the fixed threshold:

[0152]

[0153] Tolerance value D tol It may, for example, be designed to allow the amount of compression applied to each frequency band to vary slightly. In one specific embodiment, it has been found that D tol A practical value of = 12dB allows for sufficient variation.

[0154] exist Figure 2B In the example shown, threshold 201a is the lowest threshold for non-isolated frequency bands. In some such examples, if 100% timbre preservation is selected via element 212, no other non-isolated frequency bands can exceed the threshold.

[0155] Figure 2C is a graph showing an example of a threshold value for a certain range of frequencies. Figure 2C In the graph 200C of FIG. 1 , the vertical axis represents amplitude and the horizontal axis represents frequency. The thresholds for each frequency shown in the graph 200C are represented by line segments 203C and 205C. Line segment 203C represents the thresholds for the isolated frequency band in the bass range. Line segment 203C is intended to correspond approximately to the thresholds for Figure 2B Thresholds for four isolated frequency bands in the bass range shown in .

[0156] Line segment 205C indicates Figure 2B 204C indicates the boundary between the independent or isolated frequency bands and the timbre-preserving frequency bands. In some examples, there may be more than one boundary between the independent or isolated frequency bands and the timbre-preserving frequency bands.

[0157] In this example, it can be seen that although Figure 2B The thresholds for the higher frequency bands are different, but the output of the timbre-preserving method has limited the excitation of all frequencies represented by line segment 205C to Figure 2B The lowest threshold of the higher frequency band is Figure 2B Threshold 201a.

[0158] In the example of graph 200C, a control system (eg, implementing Figure 1M The control system of the level adjuster 152 of the multi-band limiter has determined the multi-band limiter configuration in response to a user input level adjustment indication received via a user input. For example, the user may have used a remote control device to adjust the volume. According to this embodiment, determining the multi-band limiter configuration involves determining a timbre preservation configuration because at least one type of level adjustment indication received is a user input level adjustment indication. In this example, Figure 1G and Figure 1JIn the example shown in , the input audio corresponds to white noise at a level that would cause limiting in the multi-band limiter. Figure 2C and Figure 2D In the example shown, the level adjuster has adjusted the input audio to this level based on one or more level adjustment instructions that may have been received via user input or via input from a noise estimator. It can be seen that for the frequency band corresponding to line segment 205C, the timbre is fully preserved in the output excitation.

[0159] Figure 2D is a graph showing another example of a threshold value for a range of frequencies. Figure 2D In the graph 200D of FIG. 1 , the vertical axis represents amplitude and the horizontal axis represents frequency. The thresholds for each frequency shown in the graph 200D are represented by line segments 203D and 205D. Line segment 203D represents the thresholds for the isolated frequency band in the bass range. Line segment 203D is intended to correspond approximately to the thresholds for Figure 2B In this example, line segment 203D is intended to be aligned with the thresholds of the four isolated frequency bands in the bass range shown in FIG. Figure 2C The same as line segment 203C.

[0160] In this example, line segment 205D represents Figure 2B The excitation of the higher frequency bands (those frequency bands higher than the four isolated frequency bands in the bass range) to which the timbre preservation method is normally applied. Here, the dotted line 204D indicates the boundary between the independent frequency bands and the frequency bands to which the timbre preservation method is normally applied. In some examples, there may be more than one boundary between the independent frequency bands and the frequency bands to which the timbre preservation method is normally applied.

[0161] In the example of graph 200D, a control system (eg, implementing Figure 1M The control system of the level adjuster 152 has responded to the noise compensation module (e.g., Figure 1M The multi-band limiter configuration is determined based on the noise compensation level adjustment indication received by the noise estimator 159 of the noise estimator 159. According to this example, determining the multi-band limiter configuration involves changing the timbre preservation function because at least one type of level adjustment indication received is a noise compensation level adjustment indication. In this embodiment, changing the timbre preservation function involves at least partially disabling the timbre preservation function.

[0162] According to this example, the noise compensation level adjustment indication corresponds to the ambient noise level in the audio environment, and changing the timbre preservation function involves changing the timbre preservation function based at least in part on the ambient noise level. In this example, the noise compensation level adjustment indication may indicate and / or be responsive to high ambient noise levels. As can be seen in graph 200D, in this example, the previously adjusted non-independent and timbre-preserving Figure 2BThe frequency band (corresponding to the frequency band of line segment 205D) becomes completely independent. In this example, the input audio corresponds to white noise at a level that would cause limiting in the multi-band limiter.

[0163] In some instances, one or more previously received user input level adjustment indications may have previously pushed the input level of one or more frequency bands into the restricted region of the multi-band limiter configuration. According to some such examples, the resulting multi-band limiter configuration may be a linear combination of 205D and 205C (such as a cross-fade between the two). In some other embodiments, the multi-band limiter configuration responsive to the noise compensation level adjustment indication may override the timbre preservation response to the user input level adjustment indication. Some examples are described below.

[0164] Figure 2E is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure. As with other figures provided herein, Figure 2E The types and quantities of the elements shown in are provided as examples only. Other embodiments may include more, less and / or different types and / or quantities of elements. According to some examples, device 240 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 240 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker or another type of device. In some embodiments, device 240 may be or may include a television control module. Depending on the specific embodiment, the television control module may be integrated into the television or may not be integrated into the television. In some embodiments, the television control module may be a device separated from the television, and in some instances may be sold separately from the television or sold as an additional or optional device that the purchased television may include. In some embodiments, the television control module may be available from a content provider (such as a provider of television programs, movies, etc.).

[0165] According to some alternative embodiments, device 240 may be or may include a server. In some such examples, device 240 may be or may include an encoder. Thus, in some instances, device 240 may be a device configured for use within an audio environment, such as a home audio environment, whereas in other instances, device 240 may be or may include a device configured for use in the “cloud”, e.g., a server.

[0166] In this example, the device 240 includes an interface system 207 and a control system 210. In some embodiments, the interface system 207 can be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 207 can be configured to exchange control information and associated data with the audio devices of the audio environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 240.

[0167] In some embodiments, the interface system 207 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. The audio data may include, but may not be limited to, an audio signal. In some instances, the audio data may include spatial data such as channel data and / or spatial metadata.

[0168] The interface system 207 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 207 may include one or more wireless interfaces. The interface system 207 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 207 may include a control system 210 and a memory system (e.g., Figure 2E 207). However, in some instances, the control system 210 may include a memory system. In some implementations, the interface system 207 may be configured to receive input from one or more microphones in the environment.

[0169] For example, the control system 210 may include a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0170] In some embodiments, the control system 210 may be present in more than one device. For example, in some embodiments, a portion of the control system 210 may be present in a device within the audio environment, and another portion of the control system 210 may be present in a device such as a server, a mobile device (e.g., a smart phone or a tablet computer) outside the audio environment. In other examples, a portion of the control system 210 may be present in a device within one of the environments depicted herein, and another portion of the control system 210 may be present in one or more other devices of the audio environment. For example, the control system function may be distributed across multiple smart audio devices of the audio environment, or may be shared by an orchestration device (such as a device that may be referred to as a smart home hub herein) and one or more other devices of the audio environment. In other examples, a portion of the control system 210 may be present in a device (such as a server) that implements a cloud-based service, and another portion of the control system 210 may be present in another device (such as another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 207 may also be present in more than one device.

[0171] In some implementations, the control system 210 may be configured to at least partially perform the methods disclosed herein.According to some examples, the control system 210 may be configured to implement a method of content stream processing.

[0172] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may, for example, reside in Figure 2E 215 and / or control system 210 shown in the optional memory system 215 and / or control system 210. Thus, various innovative aspects of the subject matter described in the present disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to process a content stream, encode a content stream, decode a content stream, etc. For example, the software may be provided by, for example, Figure 2E The control system 210 and other control system components are executed.

[0173] In some examples, device 240 may include Figure 2EOptional microphone system 220 shown in . Optional microphone system 220 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, device 240 may not include microphone system 220. However, in some such embodiments, device 240 may still be configured to receive microphone data of one or more microphones in the audio environment via interface system 210. In some such embodiments, a cloud-based implementation of device 240 may be configured to receive microphone data or noise indicators corresponding at least in part to microphone data from one or more microphones in the audio environment, or from one or more devices in the audio environment including at least one microphone, via interface system 210.

[0174] According to some embodiments, the device 240 may include Figure 2E Optional loudspeaker system 225 is shown in FIG. Optional loudspeaker system 225 may include one or more loudspeakers, which may also be referred to herein as "speakers," or more generally as "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 240 may not include loudspeaker system 225. However, in some such examples, one or more other devices of the audio environment may implement loudspeaker system 225.

[0175] In some embodiments, the device 240 may include Figure 2E Optional sensor system 230 shown in . Optional sensor system 230 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, optional sensor system 230 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of optional sensor system 230 may be present in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 230 may be present in a television, a mobile phone, or a smart speaker. In some examples, device 240 may not include sensor system 230. However, in some such embodiments, device 240 may still be configured to receive sensor data of one or more sensors in the audio environment via interface system 210.

[0176] In some embodiments, the device 240 may include Figure 2EOptional display system 235 shown in . Optional display system 235 may include one or more displays, such as one or more light emitting diode (LED) displays. In some instances, optional display system 235 may include one or more organic light emitting diode (OLED) displays. In some examples, optional display system 235 may include one or more displays of a television. In other examples, optional display system 235 may include a laptop display, a mobile device display, or another type of display. In some examples in which device 240 includes display system 235, sensor system 230 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 235. According to some such embodiments, control system 210 may be configured to control display system 235 to present one or more graphical user interfaces (GUIs).

[0177] According to some such examples, the device 240 may be or include a smart audio device. In some such implementations, the device 240 may be or include a wake-up word detector. For example, the device 240 may be or include a virtual assistant.

[0178] Figure 3 300 is a flowchart outlining an example of the disclosed method. As with other methods described herein, the blocks of method 300 need not be performed in the order indicated. In addition, such a method may include more or fewer blocks than those shown and / or described.

[0179] Method 300 may be performed by, for example Figure 2E 240 shown in and described above. In some examples, the blocks of method 300 may be performed by one or more devices within the audio environment, for example, an audio system controller or another component of the audio system, such as a smart speaker, a television, a television control module, a mobile device, etc. In some embodiments, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some blocks of method 300 may be performed by a device (such as a server) that implements a cloud-based service.

[0180] In this example, block 301 involves a control system and via an interface system (e.g., Figure 2E In some embodiments, the control system and the interface system may be a control system 210 and receive a content stream including input audio data via the interface system 207 in the figure. In some examples, the content stream may include video data corresponding to the audio data. In some embodiments, the control system and the interface system may be Figure 2EThe control system 210 and interface system 207 shown in FIG. 1 and described above. In some embodiments, the control system may be Figure 1M In some examples, block 301 may involve the control system 110 shown in FIG. Figure 1M The level adjuster 152 receives input audio data 151. According to some embodiments, block 301 may involve receiving an encoded content stream. In such embodiments, block 301 may involve decoding the encoded content stream. For example, the content stream may correspond to a movie, a television program, music, a music video, a podcast, etc.

[0181] In this embodiment, block 305 involves receiving, by the control system and via the interface system, at least one type of level adjustment indication related to playback of audio data. In some examples, the at least one type of level adjustment indication may include a user input level adjustment indication received via user input (e.g., via a remote control device, via a voice command, etc.). According to some embodiments, block 305 may involve Figure 1M The level adjuster 152 receives user input 163.

[0182] Alternatively or additionally, at least one type of level adjustment indication may include a noise compensation level adjustment indication received from a noise compensation module. Figure 1M The level adjuster 152 receives the noise estimator output 160. The noise compensation level adjustment indication may be, for example, in response to the ambient noise level detected by the noise compensation module or detected by one or more microphones from which the noise compensation module is configured to receive microphone signals. According to some embodiments, block 305 may involve Figure 1M The level adjuster 152 receives the noise estimator output 160 .

[0183] In this example, block 310 involves controlling, by a control system, a level of input audio data based on at least one type of level adjustment indication to produce level-adjusted audio data. According to some implementations, in block 310, Figure 1M The level adjuster 152 may generate level adjusted audio data 153 based on the user input 163 and / or the noise estimator output 160 .

[0184] According to this example, block 315 involves determining, by the control system and based at least in part on at least one type of level adjustment indication, a multi-band limiter configuration. According to some examples, Figure 1M The control system 110 shown in FIG. 1 may determine the multi-band limiter configuration based at least in part on the user input 163 and / or the noise estimator output 160. In some such examples, Figure 1MThe level adjuster 152 may determine the multi-band limiter configuration based at least in part on the user input 163 and / or the noise estimator output 160 .

[0185] According to some embodiments, determining the multi-band limiter configuration can involve determining a timbre-preserving configuration if the level adjustment indication (e.g., if the only level adjustment indication) is a user input level adjustment indication. In some instances, the timbre-preserving configuration can be related to frequency bands. For example, some frequency bands may be partially isolated or completely isolated. According to some examples, the levels of completely isolated frequency bands can be independently controlled without reference to the levels and / or thresholds of other frequency bands.

[0186] In some examples, if at least one type of level adjustment indication is a noise compensation level adjustment indication, determining the multi-band limiter configuration can involve changing the timbre preservation function. In some such examples, changing the timbre preservation function can involve at least partially disabling the timbre preservation function. In some implementations, the noise compensation level adjustment indication can correspond to an ambient noise level in the audio environment. In some such examples, the timbre preservation function can be changed based at least in part on the ambient noise level.

[0187] According to some embodiments, both a user input level adjustment indication and a noise compensation level adjustment indication may be received. In some such embodiments, determining the multi-band limiter configuration may involve determining a partial timbre preservation configuration based at least in part on an average (e.g., a weighted average) of the multi-band limiter configuration corresponding to the user input level adjustment indication and the multi-band limiter configuration corresponding to the noise compensation level adjustment indication.

[0188] In this example, block 320 involves configuring, by the control system, the multi-band limiter according to the multi-band limiter configuration.

[0189] In some of these examples, Figure 1M The level adjuster 152 may send a control signal 161 indicating a multi-band limiter configuration for application by the multi-band limiter 154. In some implementations, if the received level adjustment indication is a user input level adjustment indication, the timbre preservation setting of the multi-band limiter 154 may be maintained, for example, as the multi-band limiter 154 was originally adjusted at the factory. Such implementations may help ensure that the user has a pleasant experience when the user adjusts the volume control.

[0190] According to some examples, if the received level adjustment indication is a noise compensation level adjustment indication, the timbre preservation setting of the multi-band limiter 154 can be gradually closed, for example, in proportion to the noise compensation level adjustment indication. In some such examples, when noise is present, the played back audio content can still be understood under the noise while the loss of fidelity is masked by the noise source.

[0191] According to this embodiment, block 325 involves applying a multi-band limiter to the level-adjusted audio data to produce multi-band limited audio data. In some such examples, in block 325, Figure 1M The multi-band limiter 154 may generate multi-band limited audio data 155. In some embodiments, the method 300 may involve reproducing the multi-band limited audio data on one or more audio reproduction transducers of the audio environment to provide reproduced audio data. For example, the method 300 may involve reproducing the multi-band limited audio data via Figure 1M The loudspeaker 156 reproduces the multi-band limited audio data 155.

[0192] Figure 4A and Figure 4B An example of a timbre-preserving modifier module is shown. For example, the timbre-preserving modifier module can be implemented via Figure 1M The control system 110 or Figure 2E In some examples, the timbre-preserving modifier modules 403A and 403B can be implemented via the control system 210. Figure 1M In some alternative embodiments, the timbre-preserving modifier modules 403A and 403B may be implemented via Figure 1M The multi-band limiter 154 or via the control system 110 Figure 1M It is implemented by another element not shown in the figure.

[0193] exist Figure 4A In the example shown, the timbre preservation modifier module 403A is configured to control the amount of timbre preservation used at any point in time. In this example, the timbre preservation modifier module 403A is configured to control the amount of timbre preservation based at least in part on the noise estimate 402A. In this example, element 401A represents the original timbre preservation setting of the multi-band limiter. For example, element 401A may represent the original timbre preservation setting of the multi-band limiter. Figure 2A and Figure 2B The element 212 in the example shown indicates a tone preservation setting, for example, Figure 2B The value in the window portion 212b of .

[0194] Noise estimate 402A is an example of what is referred to herein as a noise compensation level adjustment indication. In some examples, noise estimate 402A may be an average, median, or maximum value of noise levels over a range of audible frequencies or a subset of audible frequency bands. In some examples, noise estimate 402A may be a spectral noise estimate of ambient noise provided to level adjuster 152 determined by noise estimator 159. In some instances, level adjuster 152 (or another component of the control system) may be configured to determine noise estimate 402A based on noise estimator output 160.

[0195] According to this example, when the noise estimate 402A indicates that the noise level is high, the timbre-preserving modifier module 403A is configured to control the timbre-preserving amount in the modified timbre-preserving amount 404A to be low. Conversely, when the noise estimate 402A indicates that the noise level is low, the timbre-preserving modifier module 403A is configured to control the timbre-preserving amount in the modified timbre-preserving amount 404A to be high (or not modified).

[0196] exist Figure 4B In the example shown, timbre preservation modifier module 403B is configured to control the amount of timbre preservation based at least in part on noise estimate 402A and user volume setting 463. User volume setting 463 is an example of what is referred to herein as a user input level adjustment indication.

[0197] According to this example, the timbre preservation modifier module 403B is configured to determine the modified timbre preservation amount 404B according to the following expression:

[0198] Tone retention = A*gain user +B*gain noisecomp (Equation 4)

[0199] In equation 4, A represents the amount of original timbre preservation, which corresponds to element 401A in this example. For example, A may represent Figure 2A and Figure 2B The element 212 in the example shown indicates a tone preservation setting, for example, Figure 2B Here, gain user represents the gain of the user application, i.e. Figure 4B The user volume setting in the example shown is 463. In Equation 4, gain noisecomp Indicates the noise compensation level adjustment indication, i.e. Figure 4B Noise estimation 402A in the example shown. In the case of a broadband noise compensation system, gain noisecomp can be a broadband gain. Alternatively, gain noisecompThe weighted average gain applied by the multi-band noise compensation system may be. In equation 4, B represents the timbre preservation value. The timbre preservation value B may, for example, indicate the minimum amount of timbre preservation allowed. The timbre preservation value B may indicate, for example, how much timbre preservation the system will have when the volume has been adjusted primarily by the noise compensator. The timbre preservation value B may, for example, be set by the manufacturer during a device tuning operation.

[0200] According to some embodiments, the "gain" term of Equation 4 is not intended to represent the unmodified gain, but rather is intended to represent a proportion of the gain applied by the control system. For example, if 75% of the gain applied by the control system is based on the user volume setting 463, then gain user will be 0.75. Accordingly, a gain of 25% will be based on the noise estimate 402A, so gain noisecomp will be 0.25.

[0201] According to this example, if the noise estimate 402A is low, the amount of timbre preservation will be close to the original adjusted timbre preservation value (represented by the A gain term). In this example, if the user volume setting 463 is high and the noise estimate 402A is also high, the timbre will be partially preserved in proportion to the relative values ​​of the user volume setting 463 and the noise estimate 402A. According to this example, if the noise estimate 402A is high and the user volume setting 463 is low, the B gain term of equation 4 will dominate and the timbre will not be preserved.

[0202] Figure 4C An example of a frequency band isolation modifier is shown. The frequency band isolation modifier 405 may be implemented, for example, via Figure 2E In some embodiments, the frequency band isolation modifier 405 can be implemented via the control system 210. Figure 1M 154). In some embodiments, the band isolation modifier 405 can be implemented as an alternative to embodiments that include the timbre preservation modifier module 403A or the timbre preservation modifier module 403B. However, some embodiments may include the band isolation modifier 405 and the timbre preservation modifier module 403A or the timbre preservation modifier module 403B, for example, to allow for the low-frequency band (such as Figure 2B The 4 bass bands are shown as completely isolated) and the band isolation modification is performed.

[0203] In this example, the frequency band isolation modifier 405 is shown as receiving an isolation setting 413 for each frequency band in the plurality of frequency bands. In some instances, the plurality of frequency bands may include all frequency bands. In other examples, the plurality of frequency bands may include frequency bands to which the timbre of the application is otherwise retained, e.g. Figure 2B non-isolated frequency band.

[0204] According to this example, the band isolation modifier 405 is also shown as receiving an optional user volume setting 463 and a noise estimate 402A. In some examples, the noise estimate 402A may be for a specific frequency band. In an alternative example, the noise estimate 402A may be for a subset of frequency bands, for example, a subset of frequency bands whose isolation values ​​may be modified by the band isolation modifier 405. In some examples, the noise estimate 402A may be for all frequency bands.

[0205] In this example, the band isolation modifier 405 is configured to determine whether to modify the isolation value of the frequency band, and if so, to generate a modified band isolation value 407 for the frequency band. The band isolation value modification can be a binary modification or a non-binary modification, depending on the specific implementation. In the case of a binary modification, in some examples, if the surrounding noise level is high, the band isolation value can be converted from non-isolated to isolated. In some such examples, if the surrounding noise level is low or the band is already isolated, the band isolation value can be unchanged. In some examples, the surrounding noise level can be based on the full spectrum. In other examples, the surrounding noise level can be specific to the frequency band frequency that the isolation may be modified.

[0206] In some alternative implementations, the frequency bands may be partially isolated (e.g., 25% isolated, 50% isolated, 75% isolated, etc.) rather than completely isolated or not isolated at all. In some such examples, the degree of isolation of the frequency bands may correspond to the ambient noise level in the audio environment. For example, the degree of isolation of the frequency bands may correspond to a weighting value that is used to weight the selected frequency bands more lightly than non-isolated frequency bands in the timbre preservation method.

[0207] In some such examples, the degree to which the effect of a frequency band on the timbre-preserving algorithm is weighted may, for example, correspond to (1-I), where I represents the degree of isolation of the frequency band. In one such example, if the frequency band is 75% isolated, then I equals 0.75, and the degree to which the effect of that frequency band on the timbre-preserving algorithm is weighted would be 0.25. In another such example, if the frequency band is 100% isolated, then I equals 1.0, and the degree to which the effect of that frequency band on the timbre-preserving algorithm is weighted would be 0.0: in other words, the threshold corresponding to that frequency band would not be used in the timbre-preserving calculation.

[0208] Return to Figure 3 In some examples, the method 300 may involve the control system causing a change in the operation of the noise compensation module, for example, in response to a predetermined input, indicator, and / or condition. Figure 1MAs described, in some embodiments, the level adjuster 152 can be configured to send a control signal 164 to the noise estimator 159. In some embodiments, the multi-band limiter 154 can be configured to send a control signal 167 to the noise estimator 159. In some examples, the control signal 164 and / or the control signal 167 can cause a change in the operation of the noise compensation module (e.g., causing a change in the functionality of the noise estimator 159). The noise compensation module can be a subsystem or module of the control system, such as Figure 1M shown.

[0209] Some such examples may involve causing a change in noise compensation module operation when multi-band restricted audio data (e.g., multi-band restricted audio data 155 output by multi-band limiter 154) causes one or more audio reproduction transducers of the audio environment (e.g., loudspeaker 156) to operate outside of a linear range. In some such instances, the control system may cause one or more audio reproduction transducers of the audio environment to operate outside of a linear range based at least in part on a noise compensation level adjustment indication and / or an ambient noise estimate. For example, multi-band restricted audio data that causes one or more audio reproduction transducers of the audio environment to operate outside of a linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

[0210] According to some examples, the noise compensation module operation change may involve changing the echo canceller function of the noise compensation module. For example, the noise compensation module operation change may involve causing the noise compensation module to use only "quiet" playback intervals as inputs to the noise estimator of the noise compensation module. A "quiet" playback interval may be an instance of an audio signal at or below a threshold level (e.g., a predetermined threshold level) in at least one of a frequency band or a time interval. In some embodiments, a "quiet" playback interval may be an instance of an audio reproduction transducer of an audio environment functioning within its linear range.

[0211] As noted elsewhere herein, in some embodiments, a level adjuster module (e.g., Figure 1M The level adjuster 152) can be configured to control the level of the input audio data to produce level-adjusted audio data (e.g., level-adjusted audio data 153). In some such embodiments, the method 300 may also involve, for example, Figure 1M The control signal 162 illustrated in FIG. 1 provides multi-band limiter feedback from the multi-band limiter to the level adjuster module. According to some such examples, the multi-band limiter feedback may indicate an amount of limiting applied by the multi-band limiter to each of the plurality of frequency bands of the level adjusted audio data.

[0212] In some such embodiments, the method 300 may also involve controlling, by the level adjuster module, the level of one or more of the plurality of frequency bands based at least in part on the multi-band limiter feedback. In some such examples, the method 300 may involve reducing the level of the one or more frequency bands based at least in part on the multi-band limiter feedback indicating that the level of the one or more frequency bands or other frequency bands in the plurality of frequency bands is limited. In some examples, the level adjuster may be configured to modify the level adjusted audio data based on a compression feedback signal from the multi-band limiter, as described above with reference to Figure 1N Descriptive.

[0213] Figure 5 500 is a flowchart outlining an example of another disclosed method. As with other methods described herein, the blocks of method 500 need not be executed in the order indicated. In addition, such a method may include more or fewer blocks than those shown and / or described.

[0214] Method 500 may be performed by, for example Figure 2E 240 shown in and described above. In some examples, the blocks of method 500 may be performed by one or more devices within the audio environment, for example, an audio system controller or another component of the audio system, such as a smart speaker, a television, a television control module, a smart speaker, a mobile device, etc. In some embodiments, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some blocks of method 500 may be performed by a device (such as a server) that implements a cloud-based service.

[0215] In this example, block 505 involves the control system and via the interface system (e.g., by Figure 2E In some embodiments, the control system and the interface system may be a control system 210 and receive a content stream including input audio data via the interface system 207 in the figure. In some examples, the content stream may include video data corresponding to the audio data. In some embodiments, the control system and the interface system may be Figure 2E The control system 210 and interface system 207 shown in FIG. 1 and described above. In some embodiments, the control system and interface system may be Figure 1M In some examples, block 505 may involve: Figure 1MThe level adjuster 152 receives the input audio data 151. According to some embodiments, block 505 may involve receiving an encoded content stream. In such embodiments, block 505 may involve decoding the encoded content stream. For example, the content stream may correspond to a movie, a television program, music, a music video, a podcast, etc.

[0216] According to this example, block 510 involves applying, by the control system, a multi-band limiter to the audio data or a processed version of the audio data to produce multi-band limited audio data. In some such examples, in block 510, Figure 1M The multi-band limiter 154 can generate multi-band limited audio data 155.

[0217] In this example, block 515 involves determining whether the multi-band restricted audio data, when played back on one or more audio reproduction transducers of the audio environment, would cause the one or more audio reproduction transducers to operate outside of a linear range. In some such examples, the control system may cause the one or more audio reproduction transducers of the audio environment to operate outside of a linear range based at least in part on at least one of a noise compensation level adjustment indication or a noise estimate. For example, the multi-band restricted audio data that causes the one or more audio reproduction transducers of the audio environment to operate outside of a linear range may be based on a noise compensation level adjustment corresponding to a high ambient noise level in the audio environment.

[0218] In some implementations, block 515 may involve referencing a data structure indicating a maximum level of operation within a linear range for each of a plurality of frequency bands. In some such examples, these maximum linear range levels may correspond to Figure 2A and Figure 2B For example, the control system may retrieve the maximum linear range level from memory. In some examples, block 515 may involve referencing a data structure indicating the maximum level of operation within the nonlinear range for each of the plurality of frequency bands. In some such examples, these maximum nonlinear range levels may correspond to Figure 2A and Figure 2B Point 218.

[0219] According to this example, block 520 involves controlling, by the control system, whether to update one or more filter coefficients of the acoustic echo canceller based on whether the multi-band restricted audio data causes one or more audio reproduction transducers of the audio environment to operate outside a linear range. According to some such examples, controlling whether the acoustic echo canceller updates one or more filter coefficients may involve controlling the acoustic echo canceller not to update one or more filter coefficients if the multi-band restricted audio data causes one or more audio reproduction transducers of the audio environment to operate outside a linear range. In some examples, block 520 may involve controlling, by the control system, whether to update a noise estimate of the noise based on whether the multi-band restricted audio data causes one or more audio reproduction transducers of the audio environment to operate outside a linear range. According to some such examples, the noise estimator may not be configured to implement the acoustic echo canceller.

[0220] For example, an acoustic echo canceller may generally operate by updating (e.g., periodically updating) adaptive filter coefficients for each of a plurality of frequency bands. In some examples, an acoustic echo canceller may be configured to: Figure 1M In some such embodiments, if the multi-band restricted audio data 155 will cause the loudspeaker 156 (or one or more other audio reproduction transducers of the audio environment) to operate outside the linear range, the control signal 164 (or the control signal 167) may control the acoustic echo canceller of the noise estimator 159 to not update the filter coefficients. According to some embodiments, the noise compensation system 150 may include one or more devices configured as a wake-up word detector. In some such embodiments, an echo-cancelled version of the microphone signal 158 may be provided to the wake-up word detector. In some instances, the acoustic echo canceller may be implemented by a device configured as a wake-up word detector. Not updating the filter coefficients when the multi-band restricted audio data 155 will cause one or more other audio reproduction transducers of the audio environment to operate outside the linear range may improve the performance of the wake-up word detector, at least in part because the echo-cancelled version of the microphone signal 158 provided to the wake-up word detector may more accurately correspond to the voice command provided to the wake-up word detector. Here, block 525 involves providing the multi-band restricted audio data to one or more audio reproduction transducers of the audio environment.

[0221] In some examples, method 500 (eg, block 510) may involve applying a multi-band limiter to the level-adjusted audio data, for example, as described above with reference to Figure 3as described. In such examples, the level-adjusted audio data is an example of the “processed version of the audio data” mentioned in box 510. Such examples may involve controlling the level of the input audio data by the control system based on at least one type of level adjustment indication to produce the level-adjusted audio data. Some such examples may involve: receiving at least one type of level adjustment indication related to the playback of the audio data by the control system; determining the multi-band limiter configuration by the control system and at least in part based on at least one type of level adjustment indication; and configuring the multi-band limiter by the control system according to the multi-band limiter configuration. In some examples, at least one type of level adjustment indication may include at least one of a user input level adjustment indication received via a user input or a noise compensation level adjustment indication received from a noise compensation module including an acoustic echo canceller.

[0222] According to some embodiments, determining the multi-band limiter configuration may involve determining a timbre-preserving configuration if the level adjustment indication (e.g., if the only level adjustment indication) is a user input level adjustment indication. In some instances, the timbre-preserving configuration may depend on the frequency bands. For example, some frequency bands may be partially isolated or completely isolated. The levels of completely isolated frequency bands may be independently controlled without reference to the levels and / or thresholds of other frequency bands.

[0223] In some examples, if the level adjustment indication received at least one type is a noise compensation level adjustment indication, determining the multi-band limiter configuration can involve changing the timbre preservation function. In some such examples, changing the timbre preservation function can involve at least partially disabling the timbre preservation function. In some implementations, the noise compensation level adjustment indication can correspond to an ambient noise level in the audio environment. In some such examples, the timbre preservation function can be changed based at least in part on the ambient noise level.

[0224] According to some embodiments, both a user input level adjustment indication and a noise compensation level adjustment indication may be received. In some such embodiments, determining a multi-band limiter configuration may involve determining a timbre-preserving configuration based at least in part on an average (e.g., a weighted average) of a multi-band limiter configuration corresponding to the user input level adjustment indication and a multi-band limiter configuration corresponding to the noise compensation level adjustment indication.

[0225] In some examples, method 500 may involve reproducing multi-band limited audio data on one or more audio reproduction transducers of an audio environment to provide reproduced audio data.

[0226] According to some examples, the noise compensation module operation change may involve changing an alternative or additional echo canceller functionality of the noise compensation module. For example, the noise compensation module operation change may involve causing the noise compensation module to use only "quiet" playback intervals as inputs to a noise estimator of the noise compensation module. A "quiet" playback interval may be an instance of an audio signal at or below a threshold level (e.g., a predetermined threshold level) in at least one of a frequency band or a time interval.

[0227] In some such embodiments, method 500 may also involve, for example, Figure 1M The control signal 162 illustrated in FIG. 1 provides multi-band limiter feedback from the multi-band limiter to the level adjuster module. According to some such examples, the multi-band limiter feedback may indicate an amount of limiting applied by the multi-band limiter to each of the plurality of frequency bands of the level adjusted audio data.

[0228] In some such embodiments, the method 500 may also involve controlling, by the level adjuster module, the level of one or more of the plurality of frequency bands based at least in part on the multi-band limiter feedback. In some such examples, the method 500 may involve reducing the level of the one or more frequency bands based at least in part on the multi-band limiter feedback indicating that the level of the one or more frequency bands or other frequency bands of the plurality of frequency bands is limited.

[0229] Fig. 6A is a graph showing an example of a time interval during which a loudspeaker is overdriven. Fig. 6A The elements are as follows:

[0230] 601: Amplitude axis of an echo reference, which may also be referred to herein as the "speaker reference" or "microphone reference" of an echo canceller. An example of an echo reference is Figure 1M Multi-band limited audio data 155;

[0231] 602: The time axis of the echo reference. The time axis 602 has Figure 6B The same scale as the graph of

[0232] 603: a portion of the content (audio data) played through the audio system of the audio environment. In this example, the audio data is represented as a sinusoid and the loudspeaker is not overdriven. Therefore, the acoustic echo canceller updates its filter coefficients during the corresponding time interval;

[0233] 604: start of the time interval for loudspeaker overdrive;

[0234] 605: A sine wave with a large amplitude, representing the audio data that the multi-band limiter will limit when the control system does not allow the loudspeaker to be overdriven;

[0235] 606: end of the time interval during which the speaker is overdriven;

[0236] 607: a sine wave of smaller amplitude, representing audio data during the time interval when the loudspeaker is again operating within its linear range;

[0237] 614: A limit indicating that the speaker-to-microphone path is no longer linear over that particular frequency range. Fig. 6A When the threshold 614 of Figure 1M ) may include at least some sound corresponding to nonlinear distortion of loudspeaker 156. In some examples, threshold 614 may correspond to a single threshold for a particular frequency band (e.g., Figure 2A Above this threshold 614, the echo canceller will have difficulty modeling the system because at least a portion of the reproduced sound that the echo canceller will model includes nonlinear distortion. However, since in many instances Figure 6B The fact that the corresponding overdrive setting of is for a short period of time, it may be acceptable for the echo canceller not to update the model during this period, because the coefficients most recently calculated before the overdrive time interval will be more accurate. Similarly, because the ambient noise included in the microphone signal provided to the noise estimator will include nonlinear distortion during the overdrive time interval, in some embodiments, the control system will cause the noise estimator to not provide an updated noise estimate during the overdrive time interval.

[0238] Figure 6B shows the data that can be sent to Fig. 6A An example of a graph corresponding to a signal of an echo canceller. In this example, Figure 6B The signal indicates to the control system when to implement the "overdrive" mode, during which the control system (e.g. Figure 1M The level adjuster 152) can configure the multi-band limiter to allow some distortion (e.g., to increase the playback volume in the presence of ambient noise) so that one or more audio reproduction transducers of the audio environment operate outside of a linear range. The "overdrive" mode can vary depending on the specific implementation. For example, the amount of overdrive allowed during the "overdrive" mode can be frequency-dependent and, in some examples, can be set during a tuning process (e.g., a factory tuning process). In some instances, the amount of overdrive allowed at lower frequencies can be greater than the amount of overdrive allowed at higher frequencies because humans are less sensitive to distortion at lower frequencies than at higher frequencies within the human audible frequency range. According to this example, Figure 6B The signal indicates whether the acoustic echo canceller should update its filter coefficients.

[0239] For example, Figure 6B The signal can be sent to the Figure 1M The noise estimator 159 implements the echo canceller. For example, Figure 6B The signal may be an example of the signal 167 sent by the multi-band limiter 154 to the noise estimator 159. In an alternative example, Figure 6B The signal can be limited by a broadband limiter or by Figure 1M The level adjuster 152 sends the signal to the echo canceller.

[0240] In this example, Figure 6B The elements are as follows:

[0241] 608: Overdrive signal axis. In this example, when the loudspeaker is overdriven, the overdrive signal is 1, and when the loudspeaker is not overdriven, the overdrive signal is 0;

[0242] 609: Timeline;

[0243] 610: The crossover time when the loudspeaker changes from being linearly driven to being overdriven;

[0244] 611: time interval during which the loudspeaker is overdriven as indicated by an overdrive signal value of 1;

[0245] 612: The crossover time for the loudspeaker to change from being overdriven to linear operation;

[0246] • 613: Time interval during which the loudspeaker is in linear operation as indicated by an overdrive signal value of 0.

[0247] Thus, in this example, the signal provided to the echo canceller includes an echo reference and an overdrive signal value indicating whether the speaker is operating outside its linear region. When the overdrive signal value is high (e.g., 1), in some examples, the noise estimate of the noise estimator will not be updated to ensure that the noise estimate remains accurate. According to some embodiments, when the overdrive signal value is high, the noise estimator can be stopped or paused from providing the noise estimate or providing the same unupdated noise estimate. In some embodiments, the overdrive signal value consists of only a single bit for the entire spectrum and works in the time domain. Such an embodiment has the advantage of not requiring a nonlinear model of the speaker. In some embodiments, the overdrive signal can be implemented by embedding an additional bit as part of the audio stream itself (e.g., as a least significant bit).

[0248] In some alternative embodiments, an overdrive signal may be provided on a per frequency band basis. In some such embodiments, it is assumed that the distortion products are harmonic distortion or intermodulation distortion (e.g., harmonic distortion is predicted using the harmonic series of the original content, and intermodulation distortion is predicted using the Volterra series), and therefore the frequency bands in which the distortion products appear should be predictable. According to some such examples, if the frequencies of the distortion products are known, the echo canceller will not update those frequencies when it is known that the loudspeaker is overdriven in the frequency bands that will generate those frequencies. In some examples, if the frequencies of the distortion products are known, the noise estimator will not provide updated noise estimates for those frequencies when it is known that the loudspeaker is overdriven in the frequency bands that will generate those frequencies.

[0249] Figure 7 A system is shown that is configured to control an acoustic echo canceller (AEC) based at least in part on the amount of "overdrive" occurring in the system. In some related embodiments, similar to Figure 7 The system shown in can control the noise estimator as a supplement or alternative to AEC. In some examples, Figure 7 The block consists of Figure 2E An example of a control system 210 or Figure 1M An example implementation of the control system 110 is shown in FIG. 1 . According to this embodiment, Figure 7 The elements are as follows:

[0250] 701A: Input signal of multi-band limiter;

[0251] 702A: A multi-band limiter having adjustments, in this example, including thresholds for each of a plurality of frequency bands above which the loudspeaker will operate outside of its linear region. In this example, the multi-band limiter 702A also implements the ability to overdrive above / override these thresholds to provide additional volume at the expense of increased loudspeaker distortion (e.g., during periods of high ambient noise levels);

[0252] 703A: The amount of overdrive occurring in the system. In some examples, element 703A may correspond to the above reference Fig. 6A and Figure 6B The overdrive signal described;

[0253] 704A: AEC controller. In this example, the AEC controller 704A is configured to instruct the AEC 707A whether it should update its coefficients based on the amount of distortion that will be produced by one or more loudspeakers;

[0254] 705A: A signal indicating whether the AEC 707A should update its coefficients;

[0255] 706A: Echo reference (or speaker feed). An example is Figure 1M Multi-band limited audio data 155;

[0256] 707A: AEC. In this example, AEC 707A is a linear echo canceller that can optionally update its coefficients based on signal 705A.

[0257] Figure 8 An example of a system configured to determine the amount of overdrive is shown. According to this embodiment, Figure 8 The elements are as follows:

[0258] 801A: Input signal to the multi-band limiter. In some examples, the input signal 801A may correspond to Figure 1M The level-adjusted audio data 153;

[0259] 802A: A loudspeaker model, which may include a limiter threshold for each of a plurality of frequency bands based on the capabilities of one or more loudspeakers in the audio environment. In some implementations, loudspeaker model 802A may correspond to the worst capable loudspeaker in the audio environment. In some instances, loudspeaker model 802A may be a more complex model, such as a harmonic distortion model or an intermodulation distortion model;

[0260] 803A: Overdrive determination module. In some examples, the overdrive determination module 803A may be composed of Figure 2E An example of a control system 210 or Figure 1M In some examples, the overdrive determination module 803A may be implemented by a multi-band limiter. In some examples, the overdrive determination module 803A is configured to determine the amount of overdrive that the system will allow to be provided in a speaker feed signal to one or more loudspeakers of an audio environment. In some examples, the overdrive determination module 803A may be configured to decide the amount of overdrive based on a noise estimate: if the noise estimate is high, then in some such examples, the system is allowed to overdrive under the assumption that any distortion caused by the overdrive will be masked. In some examples, the overdrive determination module 803A may be configured to incorporate a more complex model, such as a harmonic distortion model, to predict the amount of distortion that a speaker will generate.

[0261] 804A: Overdrive setting to be used in the multi-band limiter;

[0262] 805A: Indication of surrounding noise.

[0263] Fig. 9is a graph of noise estimation and output level according to an example. In this example, the limiting and the amount of timbre preservation or overdrive are determined based on the background noise level. According to this example, if the background noise level is low, the timbre preservation of the system is maintained at a maximum value. In this example, if the background noise level is high, the timbre preservation is turned off to maximize the volume of the system. In some implementations, the amount of overdrive can be determined based on whether the background noise level is high.

[0264] According to this example, Fig. 9 The elements are as follows:

[0265] 901: frequency axis in Hertz;

[0266] 902: The acoustic level of the system, sound pressure level (SPL) in dB;

[0267] 903: Output level of the audio reproduction transducer(s) playing the audio;

[0268] 904: Example of high SPL background noise estimation. Due to the high SPL background noise estimation, in some embodiments, the control system can determine or estimate that a relatively large amount of acoustic masking of the audio is possible;

[0269] 905: Example of medium SPL background noise estimation. Due to the medium SPL background noise estimation, in some embodiments, the control system can determine or estimate that some acoustic masking of the audio is possible;

[0270] 906: Example of low SPL background noise estimation. Due to the low SPL background noise estimation, in some embodiments, the control system can determine or estimate that acoustic masking of the audio is unlikely to occur;

[0271] 907: The difference between the high SPL background level and the acoustic level of the system's reproduced audio. In this example, the control system will operate the multi-band limiter in a manner that turns off timbre preservation in most frequency bands. According to some examples, the amount of overdrive may be increased to ensure that the difference between the noise level and the audio signal remains at or above a threshold (e.g., to ensure that the difference 907 is at least 2dB, 3dB, 4dB, 5dB, etc.). In some such embodiments, the noise estimator / AEC will be as described above with reference to Figure 7 In some such embodiments, the control system may allow the audio reproduction transducer(s) to be overdriven because the control system has determined or estimated that a relatively large amount of acoustic masking of the audio may occur, and therefore the introduced distortion will likely not be audible.

[0272] 908: The difference between the low SPL background level and the acoustic level of the audio played back by the system. In some such examples, the control system will operate the multi-band limiter at a highest timbre preservation level, or at least a higher timbre preservation level than during high background noise instances. According to some examples, the "highest" timbre preservation level can correspond to a baseline timbre preservation setting (e.g., a factory setting) for low noise or no noise conditions. In some embodiments, the control system will not overdrive the audio reproduction transducer(s) because the control system has determined or estimated that little or no acoustic masking of the audio will occur, and therefore the introduced distortion is likely to be audible.

[0273] 909: The difference between the medium SPL background level and the acoustic level of the audio played back by the system. In some implementations, the amount of timbre preservation will be between the amounts described above with reference to elements 908 and 907. In some such examples, overdriving the system will not be necessary or desirable because the control system has determined or estimated that only a moderate amount of acoustic masking of the audio will occur, and therefore the introduced distortion may be audible.

[0274] Fig.10 An example of a floor plan of an audio environment, which in this example is a living space, is shown. As with the other figures provided herein, Fig.10 The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0275] According to this example, the environment 1000 includes a living room 1010 at the upper left, a kitchen 1015 at the lower center, and a bedroom 1022 at the lower right. The boxes and circles distributed across the living space represent a group of loudspeakers 1005a to 1005h, at least some of which may be smart speakers in some embodiments, placed in a location convenient for the space, but not following any standard prescribed layout (arbitrarily placed). In some examples, the television 1030 may be configured to at least partially implement one or more disclosed embodiments. In this example, the environment 1000 includes cameras 1011a to 1011e distributed throughout the environment. In some embodiments, one or more smart audio devices in the environment 1000 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of optional sensor system 130 may be present in or on television 1030, in a mobile phone, or in a smart speaker (such as one or more of microphones 1005b, 1005d, 1005e, or 1005h). Although cameras 1011a-1011e are not shown in each depiction of environment 1000 presented in the present disclosure, in some embodiments, each environment 1000 may still include one or more cameras.

[0276] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed method, and a tangible computer-readable medium (e.g., disk) storing code for implementing one or more examples of the disclosed method or its steps. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations on data, including embodiments of the disclosed method or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed method (or its steps) in response to data asserted thereto.

[0277] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform required processing on (multiple) audio signals, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed and / or otherwise configured with software or firmware to perform any of a variety of operations, including one or more examples of the disclosed method. Alternatively, the elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more loudspeakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0278] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) storing code for executing one or more examples of the disclosed method or its steps (e.g., an encoder executable to execute one or more examples of the disclosed method or its steps).

[0279] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many changes may be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. An audio processing method, comprising: receiving, by the control system and via the interface system, a content stream comprising input audio data; receiving, by the control system and via the interface system, at least one type of level adjustment indication related to playback of the audio data; controlling, by the control system, a level of the input audio data based on the at least one type of level adjustment indication to produce level adjusted audio data; determining, by the control system and based at least in part on the at least one type of level adjustment indication, a multi-band limiter configuration; configuring, by the control system, a multi-band limiter according to the multi-band limiter configuration; applying the multi-band limiter to the level adjusted audio data to produce multi-band limited audio data; as well as providing the multi-band restricted audio data to one or more audio reproduction transducers of an audio environment, wherein receiving at least one type of level adjustment indication involves both receiving a user input level adjustment indication via user input and receiving a noise compensation level adjustment indication from a noise compensation module, and wherein determining the multi-band limiter configuration involves determining a timbre preservation configuration based at least in part on a weighted average of the user input level adjustment indication and the noise compensation level adjustment indication.

2. The method of claim 1, wherein: If receiving the at least one type of level adjustment indication involves receiving the noise compensation level adjustment indication, determining that the multi-band limiter configuration involves changing a timbre preservation function.

3. The method of claim 2, wherein: Changing the timbre preserving function involves at least partially disabling the timbre preserving function.

4. The method of claim 2, wherein: The noise compensation level adjustment indication corresponds to an ambient noise level in the audio environment, and wherein changing the timbre preservation function involves changing the timbre preservation function based at least in part on the ambient noise level.

5. The method of claim 4, further comprising reproducing the multi-band limited audio data on the one or more audio reproduction transducers of the audio environment to provide reproduced audio data, further comprising determining a masking effect of the ambient noise level on the reproduced audio data, wherein Changing the timbre-preserving function involves changing the timbre-preserving function based at least in part on the masking effect.

6. The method of claim 1, wherein: The timbre preservation configuration is frequency band dependent.

7. The method of any one of claims 1 to 6, further comprising causing a change in operation of a noise compensation module when the multi-band limited audio data causes one or more audio reproduction transducers of the audio environment to operate outside of a linear range.

8. The method of claim 7, wherein: The control system causes the one or more audio reproduction transducers of the audio environment to operate outside of the linear range based at least in part on at least one of a noise compensation level adjustment indication or a noise estimate.

9. The method of claim 7, wherein: The noise compensation module operational change involves changing an echo canceller function of the noise compensation module.

10. The method of claim 8, wherein: The noise compensation module operational change involves causing the noise compensation module to use only quiet playback intervals as input to a noise estimator of the noise compensation module, the quiet playback intervals being instances of audio signals at or below a threshold level in at least one of a frequency band or a time interval.

11. The method of claim 8, wherein: The multi-band limited audio data that causes one or more audio reproduction transducers of the audio environment to operate outside of a linear range is based on a noise compensation level adjustment that corresponds to a high ambient noise level in the audio environment.

12. The method according to any one of claims 1 to 6, wherein: The noise compensation module is a subsystem of the control system.

13. The method according to any one of claims 1 to 6, wherein: The level adjuster module of the control system is configured to control the level of the input audio data to generate the level adjusted audio data, further comprising providing multi-band limiter feedback from the multi-band limiter to the level adjuster module.

14. The method of claim 13, wherein: The multi-band limiter feedback indicates an amount of limiting applied by the multi-band limiter to each of a plurality of frequency bands of the level adjusted audio data.

15. The method of claim 14, further comprising controlling, by the level adjuster module, a level of one or more of the plurality of frequency bands based at least in part on the multi-band limiter feedback. 16 . An electronic device configured to implement the method according to claim 1 .

17. An electronic system configured to implement the method according to any one of claims 1 to 15.

18. One or more non-transitory media having software stored thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 15.

19. A computer program product comprising computer executable instructions for executing the method of any one of claims 1 to 15 by one or more processors.

Images