Content- and context-aware ambient noise compensation

Abstract

Some implementations involve receiving a content stream including audio data, determining a content type corresponding to the content stream, and determining a noise compensation method based at least in part on the content type. Some examples involve performing the noise compensation method on the audio data to produce noise-compensated audio data, rendering the noise-compensated audio data for reproduction via a set of audio reproduction transducers of an audio environment to produce rendered audio signals, and providing the rendered audio signals to at least some of the audio reproduction transducers of the audio environment.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefits of the following applications:

[0003] U.S. Provisional Patent Application No. 62 / 945,292, filed December 9, 2019;

[0004] U.S. Provisional Patent Application No. 62 / 945,303, filed December 9, 2019;

[0005] U.S. Provisional Patent Application No. 62 / 945,607, filed December 9, 2019;

[0006] U.S. Provisional Patent Application No. 63 / 198,995, filed on November 30, 2020;

[0007] U.S. Provisional Patent Application No. 63 / 198,996, filed on November 30, 2020;

[0008] U.S. Provisional Patent Application No. 63 / 198,997, filed on November 30, 2020;

[0009] U.S. Provisional Patent Application No. 63 / 198,998, filed on November 30, 2020; and

[0010] U.S. Provisional Patent Application No. 63 / 198,999, filed on November 30, 2020, which are incorporated herein by reference. Technical Field

[0011] The present disclosure relates to systems and methods for noise compensation. Background Art

[0012] Audio and video devices, including but not limited to televisions and associated audio equipment, are widely used. Although existing systems and methods for controlling audio and video devices provide benefits, improved systems and methods are still desirable.

[0013] Symbols and terminology

[0014] Throughout the disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-producing transducer (or collection of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. Speakers may be implemented to include multiple transducers (e.g., a woofer and a tweeter), which may be driven by a single common speaker feed or multiple speaker feeds. In some examples, one or more speaker feeds may undergo different processing in different circuit branches coupled to different transducers.

[0015] Throughout the disclosure, including the claims, statements about performing an operation on a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on it).

[0016] Throughout the disclosure, including the claims, the term "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and receives other XM inputs from external sources) may also be referred to as a decoder system.

[0017] Throughout the disclosure, including the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0018] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0019] As used herein, a "smart device" is an electronic device that is typically configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, near field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc., which can operate interactively and / or autonomously to some extent. Several notable types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart wristbands, smart key chains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some attributes of ubiquitous computing, such as artificial intelligence.

[0020] Here, we use the expression "smart audio device" to refer to a smart device that is a single-purpose audio device or a multi-purpose audio device (for example, an audio device that implements at least some aspects of the virtual assistant functionality). A dedicated audio device is a device (such as a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is designed to mostly or primarily achieve a single purpose. For example, while a television can typically play (and is considered capable of playing) audio from program material, in most cases modern televisions run some operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device with one or more speakers and one or more microphones is typically configured to run local applications and / or services to directly use the one or more speakers and one or more microphones. Some single-purpose audio devices can be configured to be combined together to enable audio to be played on a zone or user-configured zone.

[0021] A common type of multi-purpose audio device is an audio device that implements at least some aspects of the virtual assistant functionality, but other aspects of the virtual assistant functionality can be implemented by one or more other devices, such as one or more servers with which the multi-purpose audio device is configured to communicate. Such a multi-purpose audio device may be referred to as a "virtual assistant" here. A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (different from the virtual assistant) for applications that are cloud-enabled in a sense or are not all implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, such as speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate via a network such as the Internet. Virtual assistants can sometimes work together, such as in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together, with one virtual assistant (e.g., the virtual assistant that is most confident that it has heard the wake-up word) responding to the wake-up word. In some implementations, the connected virtual assistants may form a group, which may be managed by a master application, which may be (or implement) the virtual assistant.

[0022] Herein, "wake-up word" is used in a broad sense to refer to any sound (e.g., speech uttered by a person, or some other sound) to which the smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this case, "wake-up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some cases, what is referred to as a "wake-up word" herein may include more than one word, such as a phrase.

[0023] Here, the expression "wake-up word detector" refers to a device (or software including instructions for configuring the device) configured to continuously search for a match between real-time sound (e.g., speech) features and a trained model. Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting the wake-up word exceeds a predetermined threshold. For example, the threshold can be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. After the wake-up word event, the device can enter a state (which can be referred to as an "awakened" state or an "attention" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0024] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some cases a collection of video signals at least a portion of which is intended to be listened to together. Examples include a selection of music, a movie soundtrack, a movie, a television program, an audio portion of a television program, a podcast, a live voice call, a synthesized voice response from an intelligent assistant, etc. In some cases, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same dialogue in more than one language. In this case, it is intended that only one version of the audio data or portion thereof (e.g., a version corresponding to a single language) is reproduced at a time. Summary of the invention

[0025] At least some aspects of the present disclosure may be implemented by one or more audio processing methods, including but not limited to content stream processing methods. In some cases, the method may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods include receiving a content stream including audio data by a control system and via an interface system, and determining the content type corresponding to the content stream by the control system. Some such methods include determining a noise compensation method based at least in part on the content type by the control system. Some such methods include determining a noise estimate of an audio environment in which the content stream will be reproduced by the control system, and performing a noise compensation method on the audio data based at least in part on the noise estimate by the control system to generate noise-compensated audio data. Some such methods include rendering the noise-compensated audio data by the control system for reproduction via a set of audio reproduction transducers of the audio environment, generating a rendered audio signal, and providing the rendered audio signal to at least some of the audio reproduction transducers of a set of audio reproduction transducers of the audio environment via the interface system.

[0026] In some examples, determining the noise compensation method may be based at least in part on a dynamic range of at least one audio reproduction transducer of the audio environment. According to some examples, determining the noise compensation method may include selecting the noise compensation method from a plurality of noise compensation methods. Each noise compensation method may correspond to one or more content types in a plurality of content types. According to some examples, the plurality of content types may include at least one of a movie content type or a television program content type. In some examples, the plurality of content types may include at least one music content type.

[0027] In some examples, the noise compensation method corresponding to at least one type of music content may include a first method corresponding to a low frequency range and a second method corresponding to a high frequency range. According to some examples, the first method may include allowing a first gain applied to a first frequency band in the low frequency range to be independent of a second gain applied to a second frequency band in the low frequency range. In some cases, the second method may include constraining all gains in the frequency band applied to the high frequency range to be equal. However, in other examples, the second method may include constraining the difference between the first gain and the second gain to be less than or equal to a threshold amount. The first gain may be applied to a first frequency band in the high frequency range, and the second gain may be applied to a second frequency band in the high frequency range. According to some examples, the first frequency band may be adjacent to the second frequency band.

[0028] According to some examples, determining the content type can be based at least in part on determining a content provider of the content stream. In some examples, determining the content type can be based at least in part on user input regarding the content stream. In some such examples, the user input can be received via a voice command to a virtual assistant. In some examples, determining the content type can be based at least in part on determining an application that implements decoding of the content stream.

[0029] In some examples, the content stream may include content metadata.According to some examples, determining the content type may be based at least in part on the content metadata.

[0030] According to some examples, the control system can be configured to implement an audio classifier.In some examples, determining the content type can be based at least in part on an audio classification produced by the audio classifier.

[0031] In some implementations, determining the noise estimate may include receiving the noise estimate from a noise estimation module and / or calculating the noise estimate based on microphone signals of one or more microphones in the audio environment. In some cases, the noise estimate may indicate an estimated noise level for each of a plurality of frequency bands.

[0032] In some examples, determining the noise compensation method can be based at least in part on the time of day. According to some examples, the night noise compensation method can include controlling the playback level of at least some frequency bands of the reproduced audio data to be lower than the playback level of the corresponding frequency bands of the reproduced audio data corresponding to the daytime noise compensation method. The at least some frequency bands can, for example, correspond to bass bands.

[0033] According to some examples, determining a noise compensation method can be based at least in part on a type of ambient noise in the audio environment. In some examples, the type of ambient noise can correspond to a conversation. In some such examples, the noise compensation method can include controlling a playback level of the reproduced audio data to be lower than the ambient noise level.

[0034] In some embodiments, at least one noise compensation method may include a demo version and a default or "regular" version. For example, the demo version may include applying a higher gain in one or more frequency bands than the regular version.

[0035] Some implementations may include receiving metadata with the content stream. According to some examples, determining the noise compensation method may be based at least in part on the metadata. In some cases, the metadata may correspond to a dynamic range and / or a reference level of the audio data. According to some examples, the metadata may be "dialnorm" metadata for Dolby Digital, Dolby Digital Plus, or AC-4 audio technology. According to some examples, the metadata may be reference level metadata for a High-Efficiency Advanced Audio Coding audio coding format.

[0036] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure may be implemented by one or more non-transitory media having software stored thereon.

[0037] At least some aspects of the present disclosure can be implemented by an apparatus. For example, one or more devices can at least partially perform the method disclosed herein. In some implementations, the apparatus is or includes an audio processing system with an interface system and a control system. The control system can include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components or a combination thereof.

[0038] Details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative dimensions in the drawings may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 An example of a noise compensation system is shown.

[0040] Figure 2 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present invention.

[0041] Figure 3 is a flow chart outlining one example of the disclosed method.

[0042] Figure 4 An example of a system is shown in which a noise compensation method is based at least in part on user input.

[0043] Figure 5A An example of a system is shown in which a noise compensation method is based at least in part on application type information.

[0044] Figure 5B An example of a system is shown in which a noise compensation method is based at least in part on the state of a noise estimator.

[0045] Figure 6 An example of a system is shown in which a noise compensation method is based at least in part on a classification of ambient noise in an audio environment.

[0046] Figure 7 An example of a system is shown in which a noise compensation method is based at least in part on classification of an input content stream.

[0047] Figure 8 An example of a system is shown in which the noise compensation method is based at least in part on external control.

[0048] Fig. 9 An example of a graph corresponding to a noise compensation method is shown.

[0049] Fig.10 An example of a graph corresponding to another noise compensation method is shown.

[0050] Fig.11 An example of a graph corresponding to another noise compensation method is shown.

[0051] Fig.12 An example of a graph corresponding to another noise compensation method is shown.

[0052] Fig.13 An example of a graph corresponding to another noise compensation method is shown.

[0053] Fig.14 Examples of encoder and decoder blocks according to one implementation are shown.

[0054] Fig.15 An example of encoder and decoder blocks according to another implementation is shown.

[0055] Fig.16 An example of encoder and decoder blocks according to another implementation is shown.

[0056] Fig.17 An example of a floor plan of an audio environment, which in this example is a living space, is shown.

[0057] Fig.18An example is shown in which the noise compensation module is configured to base the noise compensation processing at least in part on data from an upstream linear processing module.

[0058] Fig.19 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dialogue enhancement process has occurred.

[0059] Fig. 20 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream virtualization process has occurred.

[0060] Fig.21 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream downmix or upmix process has occurred.

[0061] Fig. 22 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dynamic range compression process has occurred.

[0062] Fig.23 Another example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dynamic range compression process has occurred.

[0063] Similar reference numbers and designations in different drawings represent the same elements. DETAILED DESCRIPTION

[0064] The noise compensation system is configured to compensate for ambient noise in an audio environment, such as ambient noise. As used herein, the terms "ambient noise" and "surrounding noise" refer to noise generated by one or more noise sources external to the audio playback system and / or the noise compensation system. In some examples, the audio environment can be a home audio environment, such as one or more rooms in a home. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. Figure 1 An example of a noise compensation system is shown. In this example, the noise compensation system 100 is configured to adjust the level of an input audio signal 101 based on a noise estimate 108. According to this example, the noise compensation system 100 includes a speaker 104, a microphone 105, a noise estimator 107, and a noise compensator 102. In some examples, the noise estimator 107 and the noise compensator 102 can be implemented via a control system, for example, based on instructions stored on one or more non-transitory storage media. As described above, the terms "speaker", "loudspeaker" and "audio reproduction transducer" are used synonymously herein. As with other appendices provided herein, Figure 1 Sample, Figure 1 The types and quantities of the elements shown in the drawings are provided only as examples. Other implementations may include more, fewer and / or different types and quantities of elements, such as more speakers.

[0065] In this example, the noise compensator 102 is configured to receive an audio signal 101 from a file, a streaming service, etc. The noise compensator 102 may, for example, be configured to apply a gain adjustment algorithm, such as a frequency dependent gain adjustment algorithm or a wideband gain adjustment algorithm.

[0066] In this example, the noise compensator 102 is configured to send a noise compensated output signal 103 to a speaker 104. According to this example, the noise compensated output signal 103 is also provided to a noise estimator 107 and is a reference signal for the noise estimator 107. In this example, a microphone signal 106 is also sent from the microphone 105 to the noise estimator 107.

[0067] According to this example, the noise estimator 107 is a component configured to estimate the noise level in the environment including the system 100. The noise estimator 107 can be configured to receive the microphone signal 106 and calculate how much of the microphone signal 106 is composed of noise and how much is due to the playback of the speaker 104. In some examples, the noise estimator 107 may include an echo canceller. However, in some implementations, the noise estimator 107 may simply measure the noise when a signal corresponding to silence is sent to the speaker 104. In this example, the noise estimator 107 provides a noise estimate 108 to the noise compensator 102. Depending on the specific implementation, the noise estimate 108 can be a broadband estimate or a spectral estimate of the noise. In this example, the noise compensator 102 is configured to adjust the output level of the speaker 104 based on the noise estimate 108.

[0068] The speakers of some devices, such as mobile devices, often have quite limited capabilities. Therefore, the type of volume adjustment provided by system 100 is usually limited by the dynamic range of such speakers and / or speaker protection components (e.g., limiters and / or compressors).

[0069] A noise compensation system such as the noise compensation system 100 can apply a gain that is either frequency dependent or broadband. Some such noise compensation systems are configured to apply a frequency dependent gain based on a single model that increases the gain in the frequency band where ambient noise is present. This approach can be suitable for movie content, for example, when the noise compensation system 100 attempts to increase the intelligibility of dialogue over ambient noise. However, when such a noise compensation system attempts to compensate for ambient noise when music is being reproduced in an environment, this type of noise compensation can result in a noticeable timbre shift that some listeners may not like. Some listeners describe the effect of such noise compensation as making the music sound "tinny".

[0070] The present disclosure provides various methods that can overcome at least some of these potential shortcomings, as well as devices and systems for implementing the methods disclosed herein. Some disclosed implementations include determining a noise compensation method based at least in part on the type of content being reproduced. In some such examples, the noise compensation method for movie content types and / or television program content types may be different from the noise compensation method for music content types. However, other implementations do not involve determining the noise compensation method based on the content type. According to some implementations, the noise compensation method may be determined (at least in part) by one or more local devices of the audio environment. Alternatively or additionally, the noise compensation method may be determined (at least in part) by one or more remote devices (e.g., one or more devices implementing cloud-based services). In some disclosed implementations, the noise compensation method may be based at least in part on the dynamic range of one or more audio reproduction transducers of the audio environment and / or the dynamic range of the audio data being reproduced.

[0071] Figure 2 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present invention. Figure 1 Sample, Figure 2The types and quantities of the elements shown in are provided as examples only. Other implementations may include more, less and / or different types and quantities of elements. According to some examples, the device 200 may be configured to perform at least some of the methods disclosed herein. In some embodiments, the device 200 may be or may include one or more components of a television, an audio system, a mobile device (such as a cellular phone), a laptop computer, a tablet device, an intelligent speaker, or other types of equipment. In some implementations, the device 200 may be or may include a television control module. The television control module may be integrated into the television or may not be integrated into the television, depending on a specific implementation. In some implementations, the television control module may be a device separate from the television, and in some instances, may be sold separately from the television, or may be sold as an additional or optional device that may be included with the purchased television. In some embodiments, the television control module may be obtained from a content provider (such as a provider of television programs, movies, etc.).

[0072] According to some alternative embodiments, the apparatus 200 may be or may include a server. In some such examples, the apparatus 200 may be or may include an encoder. Thus, in some cases, the apparatus 200 may be a device configured for use in an audio environment such as a home audio environment, while in other cases, the apparatus 200 may be a device configured for use in the "cloud" such as a server.

[0073] In this example, the device 200 includes an interface system 205 and a control system 210. In some implementations, the interface system 205 can be configured to communicate with one or more other devices of the audio environment. In some examples, the audio environment can be a home audio environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 205 can be configured to exchange control information and associated data with the audio devices of the audio environment. In some examples, the control information and associated data can be related to one or more software applications being executed by the device 200.

[0074] In some embodiments, the interface system 205 may be configured to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some cases, the audio data may include spatial data, such as channel data and / or spatial metadata. According to some embodiments, the content stream may include metadata about the dynamic range of the audio data, and / or metadata about one or more noise compensation methods. The metadata about the dynamic range of the audio data and / or metadata about one or more noise compensation methods may, for example, have been provided by one or more devices (such as one or more servers) configured to implement cloud-based services. The metadata about the dynamic range of the audio data and / or metadata about one or more noise compensation methods may, for example, be provided by a device that may be referred to as an "encoder" here. In some such examples, in some examples, the content stream may include video data and audio data corresponding to the video data. Some examples of encoder and decoder operations are described below.

[0075] The interface system 205 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). According to some embodiments, the interface system 205 may include one or more wireless interfaces. The interface system 205 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 205 may include a control system 210 and a memory system (e.g., Figure 2 The interface system 205 may be configured to receive input from one or more microphones in the environment.

[0076] The control system 210 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0077] In some embodiments, the control system 210 may reside in more than one device. For example, in some embodiments, a portion of the control system 210 may reside in a device within one of the environments described herein, while another portion of the control system 210 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet), etc. In other examples, a portion of the control system 210 may reside in a device within one of the environments described herein, while another portion of the control system 210 may reside in one or more other devices of the environment. For example, the control system functionality may be distributed across multiple smart audio devices of the environment, or may be shared by a coordination device (e.g., a smart home hub may be referred to herein) and one or more other devices of the environment. In other examples, a portion of the control system 210 may reside in a device that implements a cloud-based service, such as a server, while another portion of the control system 210 may reside in another device that implements a cloud-based service, such as another server, a storage device, etc. In some examples, the interface system 205 may also reside in more than one device.

[0078] In some implementations, the control system 210 may be configured to at least partially perform the methods disclosed herein.According to some examples, the control system 210 may be configured to implement a method of content stream processing.

[0079] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include storage devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, on a computer. Figure 2 The optional memory system 215 shown and / or the control system 210. Thus, various innovative aspects of the subject matter described in this disclosure can be implemented in one or more non-transitory media having software stored thereon. The software can, for example, include instructions for controlling at least one device to process content streams, encode content streams, decode content streams, etc. The software can, for example, be provided by a processor such as Figure 2 The control system 210 is executed by one or more components of the control system.

[0080] In some examples, apparatus 200 may include Figure 2Optional microphone system 220 shown. Optional microphone system 220 may include one or more microphones. In some embodiments, one or more microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, device 200 may not include microphone system 220. However, in some such implementations, device 200 may still be configured to receive microphone data from one or more microphones in an audio environment via interface system 210. In some such implementations, a cloud-based implementation of device 200 may be configured to receive microphone data or noise metrics corresponding at least in part to microphone data from one or more microphones in an audio environment via interface system 210.

[0081] According to some embodiments, the apparatus 200 may include Figure 2 Optional speaker system 225 is shown. Optional speaker system 225 may include one or more speakers, which may also be referred to herein as "loudspeakers," or more generally, "audio reproduction transducers." In some examples (e.g., cloud-based implementations), device 200 may not include speaker system 225.

[0082] In some embodiments, the apparatus 200 may include Figure 2 Optional sensor system 230 shown. Optional sensor system 230 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, optional sensor system 230 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of optional sensor system 230 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 230 may be located in a television, a mobile phone, or a smart speaker. In some examples, device 200 may not include sensor system 230. However, in some such implementations, device 200 may still be configured to receive sensor data of one or more sensors in the audio environment via interface system 210.

[0083] In some embodiments, the apparatus 200 may include Figure 2Optional display system 235 shown. Optional display system 235 may include one or more displays, such as one or more light emitting diode (LED) displays. In some cases, optional display system 235 may include one or more organic light emitting diode (OLED) displays. In some examples, optional display system 235 may include one or more displays of a television. In other examples, optional display system 235 may include a laptop display, a mobile device display, or other types of displays. In some examples where device 200 includes display system 235, sensor system 230 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 235. According to some such embodiments, control system 210 may be configured to control display system 235 to present one or more graphical user interfaces (GUIs).

[0084] According to some such examples, the apparatus 200 may be or include a smart audio device. In some such implementations, the apparatus 200 may be or include a wake-up word detector. For example, the apparatus 200 may be or include a virtual assistant.

[0085] Figure 3 300 is a flowchart outlining an example of the disclosed method. As with other methods described herein, the blocks of method 300 are not necessarily executed in the order indicated. In addition, these methods may include more or fewer blocks than those shown and / or described.

[0086] Method 300 may be performed by, for example, Figure 2 2 and described above. In some examples, the blocks of method 300 may be performed by one or more devices in the audio environment, for example, an audio system controller or another component of the audio system, such as a smart speaker, a television, a television control module, a smart speaker, a mobile device, etc. In some embodiments, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in an alternative implementation, at least some blocks of method 300 may be performed by a device (e.g., a server) that implements a cloud-based service.

[0087] In this embodiment, block 305 includes receiving, by the control system and via the interface system, a content stream including audio data. In some examples, the content stream may include video data corresponding to the audio data. In some embodiments, the control system and the interface system may be Figure 2305. In some embodiments, block 305 may include receiving an encoded content stream. In such an implementation, block 305 may include decoding the encoded content stream. The content stream may, for example, correspond to a movie, a television program, music, a music video, a podcast, etc.

[0088] In this example, block 310 includes determining, by the control system, a content type corresponding to the content stream. According to some examples, determining the content type may include selecting the content type from a plurality of content types. The plurality of content types may, for example, include at least one movie content type, at least one television program content type, at least one music content type, and / or at least one other content type.

[0089] In some implementations, the control system can be configured to implement an audio classifier. In some such examples, determining the content type can be based at least in part on an audio classification generated by the audio classifier.

[0090] In some cases, determining the content type can be based at least in part on determining the content provider of the content stream. For example, if the control system determines that the content provider is a music streaming service, then in some embodiments, the control system can determine that the content type is a music content type. However, if the control system determines that the content provider is a movie and / or television program streaming service, then in some embodiments, the control system can determine that the content type is a movie or television program. In some examples, determining the content type can be based at least in part on determining an application that implements decoding of the content stream.

[0091] In some implementations, determining the content type can be based at least in part on user input, such as user input regarding a content stream. For example, the user input can be a selection of a particular content streaming service. In some examples, the user input can be a selection of music, a selection of a movie, a selection of a television program, etc. In some cases, the user input can be received via a voice command to a virtual assistant.

[0092] According to some implementations, the content stream may include content metadata. In some such examples, determining the content type may be based at least in part on the content metadata.

[0093] According to this example, block 315 includes determining, by the control system, a noise compensation method based at least in part on the content type. In some such examples, determining the noise compensation method may include selecting the noise compensation method from a plurality of noise compensation methods. Each noise compensation method may correspond to one or more of the plurality of content types. However, in some alternative examples, determining the noise compensation method may not be based on the content type.

[0094] In some examples, the noise compensation method for at least one type of music content may be a noise compensation method referred to herein as "timbre preservation". The term "timbre preservation" as used herein may have a variety of meanings. Broadly speaking, a "timbre preservation" noise compensation method is a method that at least partially preserves the frequency content or timbre of an input audio signal. Some timbre preservation noise compensation methods may completely or almost completely preserve the frequency content of an input audio signal. A timbre preservation noise compensation method may include constraining the output signal levels of at least some frequency bands based on the applied thresholds and / or output signal levels of at least some other frequency bands. In some examples, the "timbre preservation" noise compensation method may include constraining the output signal levels of all non-isolated frequency bands at least to some extent. (In some examples, if a frequency band is "isolated", only the audio in that frequency band has an effect on the applied limiting gain). Various examples of timbre preservation noise compensation methods are disclosed herein. In some examples, the noise compensation method for at least one type of music content may include a first method corresponding to a low frequency range and a second method corresponding to a high frequency range. For example, the first method may include allowing a first gain applied to a first frequency band in the low frequency range to be independent of a second gain applied to a second frequency band in the low frequency range.

[0095] In some embodiments, the second method may include constraining all gains applied in the frequency band of the high frequency range to be equal. According to some embodiments, the second method may include constraining the difference between the first gain and the second gain to be less than or equal to a threshold amount. In some such implementations, the first gain may be applied in the first frequency band of the high frequency range, and the second gain may be applied in the second frequency band of the high frequency range. For example, the first frequency band may be adjacent to the second frequency band.

[0096] According to some examples, determining the noise compensation method may be based at least in part on the time of day. In some such examples, determining the noise compensation method may not be based on the input content type. In some examples, the night noise compensation method may include controlling the playback level of one or more frequency bands of the reproduced audio data to be lower than the playback level of the same frequency band of the reproduced audio data corresponding to the daytime noise compensation method. According to some examples, the playback level of the night noise compensation method may be more compressed than the playback level of the daytime noise compensation method. In some embodiments, the night noise compensation method may include controlling the playback level of one or more frequency bands of the reproduced audio data to be lower than a threshold, even in the presence of a noisy noise source. For example, the frequency band may be a bass band.

[0097] Some disclosed noise compensation method implementations include controlling the playback level of the reproduced audio data to be higher than the ambient noise level. However, in some examples, determining the noise compensation method in block 315 may be based at least in part on the type of ambient noise in the audio environment. In some such examples, determining the noise compensation method may not be based on the input content type. In some cases, the type of ambient noise may correspond to a conversation. In some such examples, the noise compensation method may include controlling the playback level of the reproduced audio data to be lower than the ambient noise level.

[0098] According to some examples, method 300 may include receiving metadata and content streams. In some such examples, determining a noise compensation method may be based at least in part on metadata. In some cases, metadata may correspond to and / or may indicate a noise compensation method. For example, metadata may indicate a timbre-retaining noise compensation method, an unconstrained noise compensation method, or a noise compensation method that is a mixture of the two. In some examples, metadata may indicate a smoothing coefficient, a frequency smoothing constraint, a start time, and / or a release time. In some cases, metadata may be provided by an encoder. However, in other examples, metadata may be provided by another device of an audio environment such as a smart home hub. In some examples, metadata may correspond to the dynamic range of audio data and / or the amount of compression that has been applied to the audio data. For example, metadata may be "dialogue normalization (dialnorm)" metadata of Dolby Digital, Dolby Digital Plus, or AC-4 audio technology. In some cases, metadata may be "reference level" metadata of an efficient advanced audio coding audio coding format. Alternatively or additionally, determining a noise compensation method may be based at least in part on the dynamic range of at least one audio reproduction transducer of an audio environment.

[0099] In some examples, at least one noise compensation method may be a demonstration or "demo" version, while another noise compensation method may be a regular version. A regular version may be a version designed for normal use (e.g., use under normal circumstances in a home audio environment or another audio environment). The demonstration version may be suitable for use in, for example, a showroom environment, a retail environment, a sales environment, etc. For example, the demonstration version may include applying a higher gain in one or more frequency bands than the regular version.

[0100] In this example, block 320 includes determining, by the control system, a noise estimate of the audio environment in which the content stream is to be reproduced. Determining the noise estimate may also be referred to herein as determining a noise metric.

[0101] According to some examples, determining the noise metric may include receiving microphone data by the control system from one or more microphones of the audio environment in which the audio data is to be rendered, and determining the noise metric by the control system based at least in part on the microphone signal. Some such examples may include receiving microphone data from one or more microphones of the audio environment in which the control system is located. In some such implementations, the microphone signal may be received from a device including at least one audio reproduction transducer of a set of audio reproduction transducers of at least one microphone and the environment. For example, the device including at least one microphone and at least one audio reproduction transducer may be or may include a smart speaker. However, some alternative examples may include receiving microphone data or noise metrics from one or more devices of the audio environment that are not in the same location as the control system.

[0102] According to some examples, determining the noise metric may include identifying ambient noise in the received microphone signal and estimating a noise level corresponding to the ambient noise. In some such examples, determining the noise metric may include determining whether the noise level is above or below one or more thresholds. In some examples, the noise estimate may indicate an estimated noise level for each of a plurality of frequency bands.

[0103] In some examples, determining the noise metric may include determining one or more metrics corresponding to the reverberation of the environment, the frequency response of the environment, playback characteristics of one or more audio reproduction transducers of the environment, and the like.

[0104] In some cases, determining the noise estimate may include receiving the noise estimate from a noise estimation module or calculating the noise estimate based on microphone signals of one or more microphones in the audio environment.

[0105] According to the implementation, block 325 involves performing, by the control system and at least in part based on the noise estimate, a noise compensation method on the audio data to produce noise-compensated audio data. Here, block 330 includes rendering, by the control system, the noise-compensated audio data for reproduction via a set of audio reproduction transducers of the audio environment to produce a rendered audio signal. In this example, block 335 involves providing the rendered audio signal to at least some of the set of audio reproduction transducers of the audio environment via the interface system.

[0106] Figure 4 An example of a system in which a noise compensation method is based at least in part on user input is shown. Figure 1 Sample, Figure 4 The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0107] In this example, system 400 includes speaker 104, microphone 105, noise estimator 407, and noise compensator 402. In this example, noise estimator 407 and noise compensator 402 are implemented by a control system, which in this example is Figure 2 402. In some examples, the system 400 may include a display system, and the control system 210 may be configured to provide one or more graphical user interfaces for receiving the user input 402 on the display system. In some embodiments, the control system 210 may be configured to receive the user input 402 via speech received by the microphone 105. In some such implementations, the control system 210 may be configured to at least partially implement an automatic speech recognition (ASR) process. According to some such embodiments, the control system 210 may be configured to implement a cloud-based ASR process. In some examples, another device in the system 400, such as a virtual assistant, may be configured to receive voice commands and to provide a signal corresponding to the user input 402 to the control system 210.

[0108] In this example, the noise compensator 402 receives the audio signal 101 from a file, a streaming service, etc. According to this example, the noise compensator 402 is configured to determine a noise compensation method for the audio signal 101 based at least in part on the user input 402. In some cases, the user input 402 may be an explicit selection of a noise compensation method, such as a noise compensation method for music, a noise compensation method for movies, a noise compensation method for television programs, etc. In some examples, the user input 402 may correspond to a selection among two or more preset noise compensation methods.

[0109] However, in some cases, user input 402 may not involve a direct or explicit selection of a noise compensation method. For example, user input 402 may correspond to a content type of a content stream including audio signal 101. According to some such examples, noise compensator 402 may be configured to determine a noise compensation method based at least in part on the content type. Depending on the specific implementation, information for selecting a noise compensation method based on the content type may or may not be provided to the user.

[0110] In some cases, user input 402 may correspond to a content provider of a content stream. Control system 210 may be configured to determine the content type based at least in part on the content provider. For example, if the control system determines that the content provider is a music streaming service, in some embodiments, the control system may determine that the content type is a music content type. However, if the control system determines that the content provider is a movie and / or television program streaming service, in some embodiments, the control system may determine that the content type is a movie or television program. In some examples, determining the content type may be based at least in part on 402 corresponding to a user input selected by an application that implements content stream decoding.

[0111] In this example, the noise compensator 402 sends a noise compensated output signal 403 to the speaker 104. According to this example, the noise compensated output signal 403 is also provided to the noise estimator 407 and is a reference signal for the noise estimator 407. In this example, the microphone signal 106 is also sent from the microphone 105 to the noise estimator 407.

[0112] According to this example, the noise estimator 407 is a component configured to estimate the noise level in the environment including the system 400. The noise estimator 407 can be configured to receive the microphone signal 106 and calculate how much of the microphone signal 106 is composed of noise and how much is due to the playback of the speaker 104. In some examples, the noise estimator 407 may include an echo canceller. However, in some implementations, the noise estimator 407 may simply measure the noise when a signal corresponding to silence is sent to the speaker 104. In this example, the noise estimator 407 provides a noise estimate 408 to the noise compensator 402. Depending on the specific implementation, the noise estimate 408 can be a broadband estimate or a spectral estimate of the noise. In this example, the noise compensator 402 is configured to adjust the output level of the speaker 104 based on the noise estimate 408.

[0113] Figure 5A An example of a system in which a noise compensation method is based at least in part on application type information is shown. Figure 1 Sample, Figure 5A The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0114] In this example, system 500 includes speaker 104, microphone 105, noise estimator 407, and noise compensator 402. In this example, noise estimator 407 and noise compensator 402 are implemented by a control system, which in this example is Figure 2. According to this example, the control system 210 is executing a software application 502, which may be referred to herein as an "application."

[0115] In this example, the noise compensator 402 receives the audio signal 101 from a file, a streaming service, etc. According to this example, the noise compensator 402 is configured to determine the noise compensation method based at least in part on the application 502 currently being executed by the control system 210. In this example, the noise compensator 402 is configured to determine the noise compensation method based at least in part on the signal 501 indicating the application 502 running on the control system 210. For example, if the application 502 is a music-related application, the noise compensator 402 can be configured to determine a noise compensation method suitable for music. Some detailed examples are described below. For example, if the application 502 is a video application, the noise compensator 402 can be configured to determine a noise compensation method suitable for movies and / or television programs.

[0116] In some instances, application 502 may be an application that implements decoding of a content stream including audio, for which noise compensator 402 will determine a noise compensation method. Noise compensator 402 may be configured to determine the content type of the content stream, and may be configured to determine a noise compensation method corresponding to the content type. For example, the encoded video / audio may indicate the content type as part of the bitstream, for example, via metadata indicating that the content is a movie, a television program, or music. Application 502 may provide signal 501 indicating the content type to noise compensator 402. Application 502 does not need to "know" (e.g., be provided with information about the fact) that application 502 is providing information to and / or being integrated into the noise compensation system. For example, an operating system running on control system 210 may indicate to noise compensator 402 what application is playing audio at the time, and noise compensator 402 may be configured to change its processing accordingly, for example, based on a mapping between application name, content type, and corresponding noise compensation method.

[0117] In some cases, application 502 may be based directly or indirectly on user input, for example, as described above with reference to Figure 4In some cases, the user input may correspond to a selection of an application 502 corresponding to a content provider of the content stream. The control system 210 may be configured to determine the content type based at least in part on the content provider. For example, if the control system determines that the application 502 selected by the user is an application provided by a music streaming service, then in some embodiments, the control system may determine that the content type is a music content type. However, if the control system determines that the application 502 selected by the user is an application provided by a movie and / or television program streaming service, then in some embodiments, the control system may determine that the content type is a movie or television program.

[0118] Figure 5B An example of a system in which a noise compensation method is based at least in part on the state of a noise estimator is shown. Figure 1 Sample, Figure 5B The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0119] In this example, system 500 includes speaker 104, microphone 105, noise estimator 407, and noise compensator 402. In this example, noise estimator 407 and noise compensator 402 are implemented by a control system, which in this example is Figure 2 In this example, the noise compensator 402 receives the audio signal 101 from a file, a streaming service, or the like.

[0120] According to this example, the noise compensator 402 is configured to determine a noise compensation method based at least in part on a state of the noise estimator 407. In this example, the noise estimator 407 provides the noise estimator state information 501A to the noise compensator 402. Thus, in some implementations, the noise compensator 402 may be configured to determine a noise compensation method based at least in part on the noise estimator state information 501A.

[0121] In some examples, noise estimator 407 can determine which noise compensation method should be implemented by noise compensator 402. In some such examples, noise estimator state information 501A can indicate to noise compensator 402 (e.g., via noise estimator state information 501A and / or via additional information) which noise compensation method should be implemented by noise compensator 402.

[0122] In some embodiments where the noise estimator 407 is a multi-band noise estimator, if the noise estimates in a set of unupdated frequency bands (e.g., high frequency bands) have not been updated within a threshold amount of time (e.g., on the order of seconds, such as 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, etc.), the noise estimator state information 501A may indicate that the noise compensation method should switch to timbre preservation or "music mode" because the quality of the noise estimates in the unupdated frequency bands is lower, although the quality of the noise estimates in the updated frequency bands may still be higher. Alternatively or additionally, in some implementations, the noise estimator may be configured to provide a quality metric or confidence score to the noise compensation block, which the noise compensation block may use to determine which mode to be in (or partially in). For example, if the quality metric or confidence score indicates that the quality of the noise estimate is lower, the noise compensation block may determine that the noise compensation method should be timbre preservation or "music mode". In some examples, the quality metric can be directly related to the “staleness” metric described in International Publication No. WO 2019 / 209973, filed on April 24, 2019, entitled “Background Noise Estimation Using GapConfidence,” and in particular the discussion of the “staleness” metric on pages 24 and 25, which is incorporated herein by reference.

[0123] In some embodiments, the control system 210 (e.g., the noise estimator 407) can be configured to provide the multi-band noise estimator functionality described in International Publication No. WO 2019 / 209973, filed on April 24, 2019, entitled “Background Noise Estimation Using Gap Confidence,” and in particular the discussion of gap confidence values ​​and the use of gap confidence values ​​on pages 16-18, which is incorporated herein by reference.

[0124] According to some embodiments, the timbre-preserving frequency range may be selected based on a quality metric of the noise estimate (e.g., Fig.10 The quality measure of the noise estimate may, for example, correspond to the amount of time since the noise estimate for the frequency band was updated.

[0125] Figure 6 An example of a system in which a noise compensation method is based at least in part on a classification of ambient noise in an audio environment is shown. Figure 1 Sample, Figure 6The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0126] In this example, the system 600 includes a classifier 602, a speaker 104, a microphone 105, a noise estimator 407, and a noise compensator 402. In this example, the classifier 602, the noise estimator 407, and the noise compensator 402 are implemented by a control system, which in this example is Figure 2 An example of a control system 210.

[0127] According to this example, the noise compensator 402 is configured to determine a noise compensation method for the input audio signal 101 based at least in part on the noise estimate 408 from the noise estimator 407 and the signal 603 output by the classifier 602. In some examples, the signal 603 may indicate the type of noise compensation method that the noise compensator 402 should implement. Alternatively or additionally, the signal 603 may indicate a set of frequency gains that the noise estimator 407 should apply in addition to the noise compensation that the noise estimator 407 will apply based on the noise estimate 408.

[0128] In some examples, signal 603 may indicate the type of ambient noise. Noise compensator 402 may determine the type of noise compensation method to be implemented based at least in part on the type of ambient noise. For example, noise compensator 402 may be configured to determine one type of noise compensation method if classifier 602 determines that the ambient sound is a human voice, and to determine another type of noise compensation method if classifier 602 determines that the ambient sound is ambient noise, such as a vacuum cleaner, a blender, an HVAC (heating, ventilation and air conditioning) system, etc.

[0129] For example, if classifier 602 determines that the ambient sound corresponds to speech (e.g., speech corresponding to one or more ongoing conversations in the audio environment), then in some implementations, signal 603 output by classifier 602 may indicate that noise compensator 402 should avoid audio 101 (reduce its level) so that the reproduced audio remains below the speech level. In some such examples, determining the noise compensation method may not be based on the input content type. Fig.11 Describe a more detailed example.

[0130] In some examples, if classifier 602 determines that the ambient noise is music, signal 603 can indicate that noise compensator 402 should disable noise compensation. Disabling the noise compensation process can ensure that neither the noise compensation implemented by system 600 nor the noise compensation implemented by another system that is reproducing music attempts to increase the level of the reproduced music beyond the level reproduced by the other system.

[0131] According to some examples, if classifier 602 determines that the ambient noise is engine noise, blender noise, traffic noise, street noise, etc., signal 603 can indicate that noise compensator 402 should implement another type of noise compensation. In some such examples, signal 603 can indicate that noise compensator 402 should implement another type of noise compensation as described above with reference to Figure 4 The noise compensator 402 operates as described above.

[0132] In this example, the noise estimator 407 provides a noise estimate 408 to the noise compensator 402, and optionally to the classifier 602. The noise estimate 408 may be a wideband estimate or a spectral estimate of the noise, depending on the particular implementation.

[0133] According to this example, the classifier 602 is configured to receive the microphone signal 106 from the microphone 105. In some implementations, the classifier 602 is configured to receive the noise estimation 408 from the noise estimator 407. Based on one or both of these inputs, in this implementation, the classifier 602 is configured to determine the type of surrounding noise, which may also be referred to as ambient noise in this article. In some implementations, the classifier 602 may be configured to determine the type of surrounding noise by implementing one or more context-aware models based on audio. Some such implementations may use Mel frequency cepstral coefficients and their derivatives as features, and use a hidden Markov model (HMM) (e.g., continuous density HMM) as an acoustic model. In some implementations, the classifier 602 may be configured to determine the type of ambient noise by implementing a neural network that has been trained according to one or more context-aware models based on audio. Alternatively or additionally, the noise estimator 407 may be configured to determine the type of ambient noise by one or more of the aforementioned processes.

[0134] In some cases, the time (window size) at which the ambient noise is evaluated may affect the results obtained by the classifier 602 and / or the noise estimator 407. For example, a longer time window may be suitable for determining persistent ambient noise, such as HVAC noise. According to some embodiments, during the encoding process, the encoder identifies transient content when allocating bits, including changing the window size. In some such implementations, the window size and transient can be encoded and represented with metadata. Therefore, in some such examples, the metadata can be provided to a control system that is configured to implement noise compensation and / or ambient noise classification to inform related audio processing.

[0135] Figure 7 An example of a system in which a noise compensation method is based at least in part on the classification of an input content stream is shown. Figure 1 Sample, Figure 7The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0136] In this example, the system 700 includes a classifier 702, a speaker 104, a microphone 105, a noise estimator 407, and a noise compensator 402. In this example, the classifier 702, the noise estimator 407, and the noise compensator 402 are implemented by a control system, in which implementation, the control system is Figure 2 An example of a control system 210.

[0137] According to this example, the noise compensator 402 is configured to determine a noise compensation method for the input audio signal 101 based at least in part on the noise estimate 408 from the noise estimator 407 and the signal 703 output by the classifier 702. In some examples, the signal 703 may indicate the type of noise compensation method that the noise compensator 402 should implement. Alternatively or additionally, the signal 703 may indicate a set of frequency gains that the noise estimator 407 should apply in addition to the noise compensation that the noise estimator 407 will apply based on the noise estimate 408.

[0138] In some embodiments, the signal 703 may indicate a content type of at least a portion of the input content stream. In some examples, the noise compensator 402 may be configured to determine one type of noise compensation method if the classifier 702 determines that the input audio 101 corresponds to music, and to determine another type of noise compensation method if the classifier 702 determines that the input audio 101 corresponds to a movie or television program. Some detailed examples of each noise compensation method are described below.

[0139] In some examples, the classifier 702 can determine that the noise compensation method should be changed even if the same movie, television program, etc. is input and reproduced by the system 700. For example, a movie or television program may include a music introduction and / or a break during which music is reproduced. For these segments of the movie or television program, the classifier 702 can determine that the noise compensation method should correspond to the music.

[0140] In some embodiments, the classifier 702 may have predetermined that the content (e.g., the input audio 101) corresponds primarily to a movie or television program, but one or more portions of the content are, for example, a mixture of movie content and music content. For example, if the classifier 702 determines that a portion of the content is a combination of movie content and music content, then in some embodiments, the noise compensator 402 will implement a hybrid noise compensation method that combines aspects of a music noise compensation method and a movie noise compensation method. Although other examples may not explicitly mention a hybrid noise compensation method, the hybrid noise compensation method is widely applicable to many disclosed implementations. In some examples, the hybrid noise compensation method may be an average of a first set of frequency gains that the noise estimator 407 should apply to music and a second set of frequency gains that the noise estimator 407 should apply to movie content or television content. According to some such examples, if the classifier 702 were to determine that the portion of the content is 50% movie content and 50% music content, then in some implementations, the noise compensator 402 will implement a hybrid noise compensation method that averages the gain applied according to the music noise compensation method and the gain applied according to the movie noise compensation method. In some such implementations, the weighted average may be based on a determination by the classifier 702 whether the input audio 101 is music-dominant or movie-dominant, based, for example, on whether there is dialogue at a particular time, the level of dialogue compared to the level of music, etc. In some examples, if the classifier 702 determines that the portion of content is X% movie content and Y% music content, the noise compensator 402 may be configured to implement a hybrid noise compensation method that applies X% of the gain that would be applied according to the movie noise compensation method and Y% of the gain that would be applied according to the music noise compensation method.

[0141] In some implementations, the classifier 702 may be configured to receive metadata included with the input audio 101 and / or with other content received with the input audio 101. According to some such implementations, the signal 703 output by the classifier 702 may correspond to metadata. The metadata may, for example, indicate whether the content is a movie, a television program, etc. In some cases, the metadata may indicate whether a particular portion of the content is primarily music, primarily dialogue, etc. According to some examples, the metadata may correspond to the dynamic range of the audio data and / or the compression level that has been applied to the audio data. In some such examples, the metadata may be dialogue normalization metadata of Dolby Digital, Dolby Digital plus, or AC-4. In some cases, the metadata may be reference level metadata of an efficient advanced audio coding audio coding format.

[0142] In some embodiments, classifier 702 can be configured to compare the histogram of input audio 101, and be used for matching the spectral shape of the input histogram with the histogram of the audio of known type (such as movie histogram, noise histogram or music histogram).In some such examples, classifier 702 can be configured to implement the "media intelligence" classifier developed by Dolby. According to some such examples, the classifier 702 can be configured to: extract short-term features from short-term audio segments, each short-term audio segment comprising a sequence of audio frames; classify the sequence of short-term segments in the long-term audio segments into short-term audio types using the corresponding short-term features; calculate statistics about the results of the classification operation on the sequence of short-term segments in the long-term audio segments as long-term features; and classify the long-term audio segments into long-term audio types using the long-term features, for example, as described in U.S. patent application Ser. No. 14 / 779,322, entitled “Apparatus and Methods for Audio Classifying and Processing,” filed on March 25, 2014 (grant No. U.S. Pat. No. 9,842,605), which is incorporated herein by reference.

[0143] Figure 8 An example of a system in which the noise compensation method is at least partially based on external control is shown. Figure 1 Sample, Figure 8 The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0144] In this example, the system 800 includes an external control module 802, a speaker 104, a microphone 105, a noise estimator 407, and a noise compensator 402. In this example, the noise estimator 407 and the noise compensator 402 are implemented by a control system, which in this example is Figure 2 In some implementations, the external control module 802 may also be implemented by the control system 210.

[0145] According to this example, the noise compensator 402 is configured to determine a noise compensation method for the input audio signal 101 based at least in part on a noise estimate 408 from the noise estimator 407 and a signal 803 output by the external control module 802. According to some embodiments, the signal 803 may correspond to a time, for example, corresponding to a time of day. In some examples, the signal 803 may indicate the type of noise compensation method that the noise compensator 402 should implement. Alternatively or additionally, the signal 803 may indicate a set of frequency gains that the noise estimator 407 should apply in addition to the noise compensation that the noise estimator 407 will apply based on the noise estimate 408.

[0146] In some embodiments, if the external control module 802 determines that it is daytime, the signal 803 may indicate the type of noise compensation method and / or audio reproduction method suitable for daytime. According to some such embodiments, if the external control module 802 determines that it is nighttime, the signal 803 may indicate the type of noise compensation method and / or audio reproduction method suitable for nighttime. According to some examples, compared to the "daytime" type of noise compensation method and / or playback volume control (which may be referred to as "daytime mode" here), the "nighttime" type of noise compensation method and / or playback volume control (which may be referred to as "nighttime mode" here) may be calculated as less likely to keep others awake, disturb neighbors, etc. In some such examples, the nighttime mode may involve a lower level of audio reproduction in one or more frequency bands (e.g., a bass band) compared to the daytime mode. According to some examples, the nighttime mode may include applying relatively low limits to the playback level for noise compensation in some or all frequency bands compared to the limits of the playback level for the daytime mode. According to some examples, the nighttime mode may include enabling closed captions, subtitles, or subtitles, such as for movie content or television content.

[0147] According to some embodiments, night mode can be implemented during the day, for example, based on user input. For example, the night mode method can be implemented during children's nap time, study time, video conferencing time, meditation time, yoga time, or other times when relative quiet is required in or near the audio environment.

[0148] In some examples, the functionality of the external control module 802 can be combined with the functionality of one or more other modules and / or inputs, such as user input, input from the classifier 602, input from the classifier 702, etc. Some such combinations can be implemented to create a system in which noise compensation acts as an "automatic volume" so that the user does not need to adjust the volume control (e.g., on a remote control device) unless necessary. Alternatively or additionally, some "automatic volume" implementations can include controlling the playback volume to prevent or limit sudden increases or decreases in volume, such as increases in volume during commercials. In some such implementations, a control system (e.g., a control system for a television (TV)) can be configured to measure the audio system's own response to determine the appropriate volume for an audio environment for a particular content type at a particular time.

[0149] Alternatively or additionally, the control system may be configured to determine the appropriate volume for the audio environment based on user input received via a GUI, via voice commands, etc. For example, the user input may include a setup process in which the user indicates a cycle time for implementing the night mode, criteria that will trigger the night mode (e.g., initiation of a conference call, activation of a baby monitor, etc.). In some implementations, signal 803 may not be based on time, but may be based on one or more other inputs, such as initiation of a conference call, activation of a baby monitor, etc. However, in some such implementations, the noise compensation mode triggered by such input may correspond to the night mode of other implementations.

[0150] Alternatively or additionally, the control system can be configured to determine the appropriate volume for an audio environment at a particular time for a particular ambient noise level and / or ambient noise type. In some examples, the control system can be configured to implement a psychoacoustic model and / or measured responses of speakers used for reproduction to estimate whether the volume is at a desired level and make adjustments as needed.

[0151] According to some embodiments, via a combination of two or more disclosed systems, different noise compensation methods can be triggered based on various inputs. For example, night mode can be automatically triggered at a specific time at night and can be automatically ended at a specific time during the day. In some embodiments, the following reference Fig.11 The described background music mode can be automatically triggered via detection of persistent ambient noise in the human voice frequency range (which indicates a possible conversation). The background music mode can be combined with a day mode or a night mode, for example, depending on the time of day the conversation occurs. In some embodiments, the background music mode can be triggered by user input such as a voice command, an input to a GUI, etc.

[0152] Fig. 9An example of a graph corresponding to a noise compensation method is shown. Fig. 9 The noise compensation method represented by the curve diagram shown can be applicable to movie content types or television program content types. As described in more detail below, the noise compensation method that can be applied when the control system determines that the input content is movie content type or television program content type can also be referred to as an "unconstrained" noise compensation method or a non-timbre-preserving noise compensation method.

[0153] In this example, the vertical axis of the graph 900 indicates level and the horizontal axis indicates frequency. In some examples, the horizontal axis may indicate frequencies within the range that a typical human can hear, for example, within the range of 20-20,000 Hz. The graph 900 indicates an input audio signal 905, which in this example is a white noise input audio signal. The white noise input audio signal is used only to illustrate the type of gain that the noise compensation system will apply. The input audio signal 905 may have any type of frequency content. Fig.10 In the graph shown in FIG. 13 , a white noise input signal should also be assumed. In graph 900 , curve 902 corresponds to the spectrum of the detected background noise. For example, curve 902 may correspond to the spectrum estimate of the noise in noise estimate 408 provided by noise estimator 407 to noise compensator 402 .

[0154] According to this implementation, arrow 906 indicates that the noise compensator 402 generates a signal corresponding to Fig. 9 According to this example, arrow 906 represents an example of a gain that is applied by the noise compensator 402 to the audio data of the input content stream to produce a noise compensated output signal 403, which corresponds to the noise compensated output signal 901 in this example. In this example, when the control system determines that the input content is a movie content type or a television program content type, the noise compensation method can be applied.

[0155] exist Fig. 9 , difference 903 represents the difference between curve 901 and curve 902. Thus, difference 903 represents the difference between the background noise of the audio environment and the level of the noise compensated output signal 403. In some examples, the gain applied by the noise compensator 402 ensures that difference 903 is within Fig. 9903 is constant or approximately constant within the frequency range shown. According to some such examples, if the ambient noise level of the audio environment increases, the playback level of the noise compensated output signal 403 will increase by the amount required to keep the difference 903 constant or approximately constant. In this document, "approximately" may mean within a predetermined range, such as within 2%, within 5%, within 7%, within 10%, within 1 dB, within 2 dB, within 3 dB, within 4 dB, etc.

[0156] In some examples, such as Fig. 9 As shown, the gain applied in one frequency band may be unconstrained relative to the gain applied in another frequency band, for example, relative to the gain applied in an adjacent frequency band. Therefore, according to this noise compensation method, the spectral content of the input audio signal 905 is generally not retained. Therefore, the noise compensation method that can be applied when the control system determines that the input content is a movie content type or a television program content type may also be referred to herein as an "unconstrained" noise compensation method or a non-timbre-preserving noise compensation method. In some cases, the unconstrained noise compensation method may be suitable for other types of content and / or for situations where the noise compensation method is not selected based on the content type.

[0157] In some such examples, the gain applied by the noise compensator 402 ensures that the signal-to-noise ratio (SNR) is within Fig. 9 The gain applied in one frequency band may be unconstrained relative to the gain applied in another frequency band, such as relative to the gain applied in an adjacent frequency band.

[0158] However, according to some embodiments, Fig. 9 The difference 903 and / or the SNR may not be constant or approximately constant over the frequency range shown. In some such examples, the level of the noise compensated output signal 403 may be shaped to emphasize the dialogue region of the noise compensated output signal 403, e.g., increasing the level of the noise compensated output signal 403 at frequencies in the range of 300 Hz to 3000 Hz, in the range of 80 Hz to 260 Hz, in the range of 80 Hz to 3000 Hz, etc., relative to the level of the noise compensated output signal 403 in other frequency ranges.

[0159] In some examples, Fig. 9The noise compensation method can be based on a psychoacoustic model. In some such examples, the noise compensation method can use equal loudness curves that apply different gains to different frequency bands. According to some such examples, the noise compensation method can correspond to one of the Dolby Volume methods, such as one of those described in U.S. Patent No. 8,090,120, entitled "Calculating and Adjusting the Perceived Loudness and / or the Perceived Spectral Balance of an Audio Signal," filed on October 25, 2005, which is incorporated herein by reference.

[0160] Fig.10 An example of a graph corresponding to another noise compensation method is shown in FIG. Fig.10 The noise compensation method represented by the graph shown may be suitable for music content types. Fig.10 The noise compensation method represented by the graph shown is a timbre-preserving noise compensation method, at least for a higher frequency range (FR2). As described elsewhere herein, the timbre-preserving noise compensation method may include constraining the output signal levels of at least some frequency bands based on the output signal levels of at least some other frequency bands and / or an applied threshold. In some examples, the "timbre-preserving" noise compensation method may include constraining the output signal levels of all non-isolated frequency bands, at least to some extent. In some cases, the timbre-preserving noise compensation method may be applicable to other types of content and / or to situations where the noise compensation method is not selected based on the type of content.

[0161] In this example, the vertical axis of the graph 1000 represents level and the horizontal axis represents frequency. In some examples, the horizontal axis may indicate frequencies within a range that a typical human can hear (e.g., within a range of 20–20,000 Hz). In the graph 1000, the curve 1001 corresponds to the frequency spectrum of the detected background noise. For example, the curve 1001 may correspond to the frequency spectrum estimate of the noise in the noise estimate 408 provided by the noise estimator 407 to the noise compensator 402.

[0162] According to the implementation, curves 1002 and 1004 represent gains applied by the noise compensator 402 in order to reproduce corresponding to the noise compensation method. According to this example, curves 1002 and 1004 represent gains applied by the noise compensator 402 to the audio data of the input content stream in order to produce the noise compensated output signal 403. In this example, the noise compensation method can be applied when the control system determines that the input content is a music content type.

[0163] In this example, the noise compensation method includes corresponding to the low frequency range (frequency range 1, Fig.10 The first method corresponding to the high frequency range (frequency range 2, in Fig.10 Some related noise compensation methods may include three or more frequency ranges. In this example, the low frequency range corresponds to the "bass" frequency range, which in some examples may be between 20 and 250 Hz. In other examples, the low frequency range may be a different range, such as between 20 and 300 Hz, between 20 and 350 Hz, between 20 and 400 Hz, between 20 and 450 Hz, between 20 and 500 Hz, between 50 and 300 Hz, between 50 and 350 Hz, between 50 and 400 Hz, between 50 and 450 Hz, between 50 and 500 Hz, between 100 and 300 Hz, between 100 and 350 Hz, between 100 and 400 Hz, between 100 and 450 Hz, between 100 and 500 Hz, and so on. In some examples, the high frequency range may be the entire hearing range above the low frequency range. However, in some other examples, the high frequency range may be smaller than the entire audible range above the low frequency range.

[0164] According to the implementation, the first method includes allowing a first gain applied to a first frequency band in the low frequency range to be independent of a second gain applied to a second frequency band in the low frequency range. In some cases, the first frequency band may be adjacent to the second frequency band. Therefore, according to this noise compensation method, the spectral content of the input audio signal in the low frequency range is generally not retained. Therefore, the noise compensation method may be referred to as an "unconstrained" noise compensation method or a non-timbre-preserving noise compensation method in the low frequency range.

[0165] exist Fig.10 , difference 1003 represents the difference between curve 1001 and curve 1002. Thus, in one example of the first method, difference 1003 represents the difference between the background noise of the audio environment and the level of the noise-compensated output signal 403. In some examples, the gain applied by the noise compensator 402 ensures that difference 1003 is constant or approximately constant over a low frequency range. According to some such examples, if the ambient noise level of the audio environment increases, the playback level of the noise-compensated output signal 403 will increase by the amount required to keep difference 1003 constant or approximately constant. In this document, "approximately" may mean within a predetermined range, such as within 2%, within 5%, within 7%, within 10%, within 1dB, within 2dB, within 3dB, within 4dB, etc. In some examples of the first method, the gain applied by the noise compensator 402 ensures that the signal-to-noise ratio (SNR) is within Fig.10 It is shown to be constant or nearly constant over the low frequency range.

[0166] However, according to some embodiments, Fig.10 The difference 1003 and / or SNR may not be constant or approximately constant over the low frequency range shown. In some such examples, the level of the noise compensated output signal 403 may be shaped to highlight the dialog portion of the noise compensated output signal 403 relative to the level of the noise compensated output signal 403 in other parts of the low frequency range.

[0167] According to some embodiments, the second method involves preserving the timbre of the reproduced audio in the high frequency range. Fig.10 The second method includes constraining all gains applied in the frequency bands of the high frequency range to be equal. In this example, the gain applied to the lowest frequency band of the high frequency range is equal to the gain applied to the highest frequency band of the low frequency range, or is within a predetermined range of the gain.

[0168] In some examples, the gain applied in the high frequency range is the weighted average of the gain of the SNR on all frequency bands of the high frequency range. Below is a set of example weights that can be applied to the gain of the SNR in the frequency bands of the high frequency range: [1.0, 0.9, 0.8, 0.7, 0.6, 0.5, 0.4, 0.3, 0.2, 0.1]. According to some embodiments, this group of weights can be included together with the input audio data as metadata, for example. In this example, the gain applied to the lowest frequency band of the high frequency range is equal to the gain applied to the highest frequency band of the low frequency range. Such an example may have several benefits. One benefit is to prevent discontinuity between the gain applied to the low frequency range and the gain applied to the high frequency range. Another benefit is that this method highlights the spectral region where human hearing is most sensitive. In some alternative examples, the weighting of the gain corresponding to the frequency band of the high frequency range can be the inverse of the equal loudness curve, so that the frequency that human hearing is most sensitive to is weighted more than other frequency bands.

[0169] To ensure that listening to music is a low-fatigue experience, music is often low-pass filtered at mix time. By using a fixed gain over a wide frequency range, the intent of this low-pass filtering and the overall timbre of the music can be maintained. Another benefit is that the Gaussian noise of the microphones used in the music recording will not be overly prominent relative to the more important content, especially in the high frequencies. Therefore, the enjoyment of the music can be maintained.

[0170] However, in some alternative examples, the second method may include constraining the difference between the first gain applied in the first frequency band of the high frequency range and the second gain applied in the second frequency band of the high frequency range to be less than or equal to the threshold amount. In some such examples, the first frequency band may be adjacent to the second frequency band.

[0171] Fig.11 An example of a graph corresponding to another noise compensation method is shown. Fig.11 The noise compensation method represented by the graph shown may be referred to herein as a "background music" noise compensation method. In some such examples, determining the noise compensation method may not be based on the input content type.

[0172] In this example, the vertical axis of the graph 1100 represents level and the horizontal axis represents frequency. In some examples, the horizontal axis may indicate frequencies within the range that a typical human can hear (e.g., within the range of 20-20,000 Hz). In the graph 1100, the curve 1101 corresponds to the frequency spectrum of the detected background noise. For example, the curve 1101 may correspond to the frequency spectrum estimate of the noise in the noise estimate 408 provided by the noise estimator 407 to the noise compensator 402. In this example, the control system is configured to implement an environmental noise classifier, such as the one described above with reference to FIG. Figure 6 6. According to this example, the classifier has determined that the ambient noise corresponds to one or more conversations. The context may be, for example, a party, a family gathering, etc.

[0173] According to the implementation, curve 1102 represents the gain applied by the noise compensator 402 in order to reproduce the corresponding noise compensation method. According to the example, curve 1102 represents the gain applied by the noise compensator 402 to the audio data of the input content stream in order to produce the noise compensated output signal 403. In this example, the noise compensation method can be applied when the control system determines that the input content is a music content type, and when the classifier determines that the ambient noise corresponds to one or more dialogues. In some examples, the noise compensation method can be applied when the control system determines that the input content is a movie content type or a television program content type, and when the classifier determines that the ambient noise corresponds to one or more dialogues.

[0174] According to this example of a "background music" noise compensation method, the control system ensures that the level of reproduced audio, which may or may not be music, remains below the background noise level. Fig.11 , difference 1103 represents the difference between curve 1101 and curve 1102. Thus, difference 1103 represents the difference between the background noise of the audio environment and the level of the noise compensated output signal 403. In some examples, the gain applied by the noise compensator 402 ensures that difference 1103 is Fig.111103 is constant or approximately constant over the frequency range shown. According to some such examples, if the ambient noise level of the audio environment decreases, the playback level of the noise compensated output signal 403 will decrease by the amount required to keep the difference 1103 constant or approximately constant. In this context, "approximately" may mean within a predetermined range, such as within 2%, within 5%, within 7%, within 10%, within 1 dB, within 2 dB, within 3 dB, within 4 dB, etc.

[0175] In some such examples, the gain applied by the noise compensator 402 ensures that the signal-to-noise ratio (SNR) is within Fig.11 The gain applied in one frequency band may be unconstrained relative to the gain applied in another frequency band (eg, relative to the gain applied in an adjacent frequency band).

[0176] This approach can ensure that when the user of the system is speaking, the level of the reproduced audio remains below their speech level. Therefore, this approach can ensure that people in the audio environment have a reasonable opportunity to hear each other and have a conversation. In some examples, the "background music" noise compensation method can be started based on one or more other factors, such as user input, time of day, content type and / or location. For example, in the kitchen or dining area of ​​a home where multiple people live, the "background music" noise compensation method may be needed in many cases to talk while preparing food, eating, etc. In some examples, the "background music" noise compensation method can be implemented by "whole home" audio, where multiple speakers in different rooms are configured to play the same content. According to some such examples, at least some of the speakers (in some cases all speakers) of the audio environment can be smart speakers, which are independently configured to implement the "background music" noise compensation method. In some alternative examples, at least some of the speakers (in some cases all speakers) of the audio environment can be controlled by a central home control system, such as a smart home hub, which is configured to implement the "background music" noise compensation method.

[0177] Fig.12 An example of a graph corresponding to another noise compensation method is shown. Fig.12 The noise compensation method represented by the graph shown may be referred to herein as a "nighttime noise compensation method". However, this noise compensation method may also be applicable to times other than nighttime, such as nap time, study time, telephone conference time, or other times when it is desirable to avoid disturbing others with unwanted reproduced audio (particularly audio reproduced in the bass range). In some such examples, determining the noise compensation method may not be based on the input content type.

[0178] As in the previous example, the vertical axis of the graph 1200 represents level and the horizontal axis represents frequency, which in this case is the frequency within the range that a typical human can hear. In the graph 1200, the curve 1201 corresponds to the frequency spectrum of the detected background noise. For example, the curve 1201 may correspond to the frequency spectrum estimate of the noise in the noise estimate 408 provided by the noise estimator 407 to the noise compensator 402.

[0179] According to the implementation, curves 1202 and 1204 represent the gains applied by the noise compensator 402 in order to reproduce the noise compensation method corresponding to the night time. In some embodiments, curves 1202 and 1204 can be connected, such as smoothly connected. According to this example, curves 1202 and 1204 represent the gains applied by the noise compensator 402 to the audio data of the input content stream in order to produce the noise compensated output signal 403. In this example, when the control system determines that the time of day is night time, for example, after a predetermined time in the evening that can be configurable according to user input, the night time noise compensation method can be applied. However, in some examples, the user can choose to provide a user input indicating that the night time noise compensation method should be invoked, regardless of the time of day.

[0180] In this example, the noise compensation method includes corresponding to the low frequency range (frequency range 1, Fig.12 The first method corresponding to the high frequency range (frequency range 2, in Fig.12 In this example, the low frequency range corresponds to the "bass" frequency range, which in some examples may be between 20 and 250 Hz. In other examples, the low frequency range may be a different range, such as between 20 and 300 Hz, between 20 and 350 Hz, between 20 and 400 Hz, between 20 and 450 Hz, between 20 and 500 Hz, etc. In some examples, the high frequency range may be the entire audible range above the low frequency range. However, in some other examples, the high frequency range may be less than the entire audible range above the low frequency range.

[0181] In this implementation, the first method involves controlling the gain applied to the audio data in the low frequency range to remain at a constant level. In this example, the first method includes controlling the gain applied to the audio data in the low frequency range to remain at a level lower than the lowest playback level of any higher frequency audio data. According to this implementation, the first method includes controlling the gain applied to the audio data in the low frequency range to be lower than the ambient noise level in the low frequency range.

[0182] exist Fig.12, difference 1203 represents the difference between curve 1201 and curve 1204. Thus, in one example of the first method, difference 1203 represents the difference between the background noise of the audio environment and the level of the noise-compensated output signal 403. In some examples, the gain applied by the noise compensator 402 ensures that difference 1203 is constant or approximately constant over a high frequency range. According to some such examples, if the ambient noise level of the audio environment increases, the playback level of the noise-compensated output signal 403 will increase by the amount required to keep difference 1203 constant or approximately constant. In this document, "approximately" may mean within a predetermined range, such as within 2%, within 5%, within 7%, within 10%, within 1 dB, within 2 dB, within 3 dB, within 4 dB, etc. In some examples of the first method, the gain applied by the noise compensator 402 ensures that the signal-to-noise ratio (SNR) is within Fig.12 The high frequency range is shown to be constant or nearly constant.

[0183] However, according to some embodiments, Fig.12 In the high frequency range shown, the difference 1203 and / or the SNR may not be constant or approximately constant. In some such examples, the level of the noise compensated output signal 403 may be shaped to emphasize the dialog portion of the noise compensated output signal 403 relative to the level of the noise compensated output signal 403 in other parts of the high frequency range. In some nighttime noise compensation embodiments, a timbre-preserving noise compensation mode may be used for at least some frequency bands, such as for the frequency range FR2. In some such examples, the curve 1204 may have Fig.10 Characteristics of curve 1004.

[0184] Fig.13 An example of a graph corresponding to another noise compensation method is shown. As in the previous example, the vertical axis of the graph 1300 represents level and the horizontal axis represents frequency, which in this case is the frequency within the range that a typical human can hear. In the graph 1300, the curve 1301 corresponds to the frequency spectrum of the detected background noise. In this example, the curve 1302 corresponds to the "normal" or "conventional" noise compensation method. The curve 1302 may, for example, correspond to a noise compensation method suitable for a movie content type or a television program content type during daily use (e.g., in a person's home).

[0185] The noise compensation method represented by curve 1303 may be referred to herein as a "demonstration noise compensation method" because such noise compensation methods are suitable for use during demonstrations of the capabilities of an audio device and / or audio system, such as during times when potential purchasers are evaluating the audio device and / or system. The demonstration noise compensation method is designed to overemphasize the noise compensated amount to ensure that even non-audiophile listeners can hear the effects of the noise compensation. However, such noise compensation methods may also be suitable for use in other situations. Some listeners may prefer the demonstration noise compensation method over the "normal" or "conventional" noise compensation method used in everyday situations.

[0186] Some disclosed implementations may involve operations that will be referred to as "encoders" herein. Although the encoder can be shown by a single block, the encoder can be implemented via one or more devices. In some implementations, the encoder can be implemented by one or more devices of cloud-based services, such as one or more servers in a data center, data storage devices, etc. In some examples, the encoder can be configured to determine the compensation process to be performed in response to a noise metric. Some such implementations may involve interactions between an encoder and a downstream "decoder", such as where the decoder provides an ambient noise metric to the encoder. An implementation in which the encoder performs at least some of the disclosed methods (e.g., determining a compensation process, or determining a plurality of selectable compensation processes) may be potentially advantageous because the encoder will typically have a much larger processing power than the decoder.

[0187] Fig.14 An example of encoder and decoder blocks according to one implementation is shown. In this example, encoder 1401 is shown transmitting an encoded audio bitstream 1402 to decoder 1403. In some such examples, encoder 1401 may be configured to transmit the encoded audio bitstream to multiple decoders.

[0188] According to some embodiments, the encoder 1401 and the decoder 1403 may be implemented by separate instances of the control system 210, while in other examples, the encoder 1401 and the decoder 1403 may be considered to be part of a single instance of the control system 210, such as components of a single system. Fig.14 1401 and / or decoder 1403 may include more than one component, such as modules and / or sub-modules configured to perform various tasks.

[0189] In some implementations, the decoder 1403 may be implemented via one or more devices of an audio environment (e.g., a home audio environment). Some of the tasks that the decoder 1403 may perform are described above. In some such examples, the decoder 1403 may be implemented via a television of the audio environment, via a television control module of the audio environment, in multiple speakers of a "whole home" audio system, and the like. However, in some examples, at least some of the functions of the decoder 1403 may be implemented via one or more other devices of the audio environment, such as through hearing aids, personal sound amplification products, cochlear implants, headphones, laptops, mobile devices, smart speakers, smart home hubs configured to communicate with the decoder 1403 (e.g., via the Internet), and televisions of the audio environment, and the like.

[0190] In some implementations, the encoder 1401 may be implemented via one or more devices of a cloud-based service, such as one or more servers in a data center, a data storage device, etc. Fig.14 In the example shown, encoder 1401 has received or obtained an audio bitstream, has encoded the received audio bitstream, and is in the process of transmitting an encoded audio bitstream 1402 to decoder 1403. In some such examples, the encoded audio bitstream 1402 may be part of an encoded content stream that includes, for example, encoded video data corresponding to a television program, a movie, a musical performance, etc. The encoded audio bitstream 1402 may correspond to the encoded video data. For example, the encoded audio bitstream 1402 may include speech (e.g., dialogue) corresponding to the encoded video data. In some implementations, the encoded audio bitstream 1402 may include music and audio effects (M&E) corresponding to the encoded video data.

[0191] Fig.15 An example of encoder and decoder blocks according to another implementation is shown. In this example, an encoder 1401 is shown transmitting an encoded audio bitstream 1402 and one or more types of metadata 1504 to a decoder 1403 .

[0192] In some such examples, the encoder 1401 may be configured to provide content metadata indicating a type of content corresponding to at least a portion of the encoded audio bitstream 1402. For example, the metadata 1504 may include content metadata indicating whether at least a portion of the encoded audio bitstream 1402 corresponds to a music type. Alternatively or additionally, the metadata 1504 may include content metadata indicating whether at least a portion of the encoded audio bitstream 1402 corresponds to audio of a movie or television program. In some implementations, the content metadata may indicate whether at least a portion of the audio of the movie or television program in the encoded audio bitstream 1402 is a music-dominant portion and / or a music interlude, where the music is not merely background music but is intended to be clearly heard. In some such examples, the content metadata may indicate, for example, that a portion of the content is X% movie content and Y% music content.

[0193] According to some such implementations, decoder 1403 (or a control system configured to implement decoder 1403) may be configured to determine a content type of at least a portion of encoded audio bitstream 1402 based at least in part on content metadata of metadata 1504. Figure 3 The process of determining the content type in block 310 of the embodiment of the present invention may be based at least in part on the content metadata. In some such implementations, the decoder 1403 (or a control system configured to implement the decoder 1403) may be configured to determine the noise compensation method based at least in part on the content type, for example, as described above with reference to Figure 3 315 of . If the content metadata indicates, for example, that a portion of the content is X% movie content and Y% music content, then in some examples, the decoder 1403 (or a control system configured to implement the decoder 1403) can be configured to implement a hybrid noise compensation method that applies a gain of X% to be applied according to a movie noise compensation method and a gain of Y% to be applied according to a music noise compensation method. In some such examples, the decoder 1403 (or a control system configured to implement the decoder 1403) can be configured to perform the remaining blocks of method 300.

[0194] According to some examples, the metadata 1504 may indicate which noise compensation modes are allowed for, for example, some or all of the audio bitstream 1402. Alternatively or additionally, the metadata 1504 may indicate an amount of post-processing that may be performed on the audio bitstream 1402. Alternatively or additionally, the metadata 1504 may indicate a maximum amount of gain in a subsequent equalization process of the audio bitstream 1402.

[0195] Alternatively or additionally, metadata 1504 may include a timbre preservation indicator that indicates the degree to which the timbre of audio bitstream 1402 should be preserved. In some such examples, the noise compensation system of decoder 1403 may be configured to operate in a "music" mode when the timbre preservation indicator is a high value (e.g., greater than or equal to a threshold), and may be configured to operate in a "movie" mode when the timbre preservation indicator is a low value (e.g., below a threshold).

[0196] According to some examples, metadata 1504 may include metadata corresponding to the dynamic range of at least a portion of the audio data and / or the amount of compression that has been applied to at least a portion of the audio data. In some such examples, the metadata may be dialogue normalization metadata of Dolby Digital, Dolby Digital Plus, or AC-4. In some cases, the metadata may be reference level metadata of an efficient advanced audio coding audio coding format. In some examples, the metadata may include dynamic range control metadata and / or loudness metadata according to MPEG-D Part 4: Dynamic Range Control (ISO / IEC 23003-4:2015). In some such implementations, the decoder 1403 (or a control system configured to implement the decoder 1403) may be configured to determine the following noise compensation method, which is based at least in part on metadata corresponding to the dynamic range, loudness, and / or the amount of compression that has been applied. According to some such examples, for audio to which at least a threshold amount of compression has been applied, in some examples, a "music" noise compensation method may be applied, and for audio to which less than a threshold amount of compression has been applied, in some examples, a "movie" or "TV show" noise compensation method may be applied.

[0197] Alternatively or additionally, decoder 1403 (or a control system configured to implement decoder 1403) can be configured to control the noise estimation process based at least in part on metadata corresponding to the amount of compression that has been applied. In some such examples, for audio to which at least a threshold amount of compression has been applied, in some examples, a relatively low band count noise estimator can be applied. According to some such examples, for audio to which less than a threshold amount of compression has been applied, in some examples, a default multi-band noise estimator can be used.

[0198] In some implementations, encoder 1401 may be configured to determine a noise compensation process for one or more content types. In some examples, encoder 1401 may be configured to determine a noise compensation process for each of a plurality of content types. A plurality of content types may, for example, include at least one music content type, a movie content type, and / or a television program content type. According to some such examples, encoder 1401 may be configured to determine noise compensation metadata corresponding to each of a plurality of noise compensation processes, and to provide noise compensation metadata corresponding to a certain type of content currently being provided to decoder 1403 to decoder 1403.

[0199] In some implementations, encoder 1401 may be configured to determine a noise compensation process for one or more types of surrounding noise profiles. In some examples, each surrounding noise profile may correspond to the category of surrounding noise, such as traffic noise, train noise, rain, etc. In some such examples, encoder 1401 may be configured to determine a plurality of noise compensation processes for each type of surrounding noise. Each noise compensation process in a plurality of noise compensation processes may, for example, correspond to different surrounding noise levels. For example, a noise compensation process may correspond to a low surrounding noise level, another noise compensation process may correspond to a medium surrounding noise level, and another noise compensation process may correspond to a high surrounding noise level.

[0200] According to some such examples, the encoder 1401 may be configured to determine noise compensation metadata corresponding to a noise compensation process, and to provide the noise compensation metadata to the decoder 1403. In some such implementations, the encoder 1401 may be configured to determine noise compensation metadata corresponding to each noise compensation process in a plurality of noise compensation processes. In some such examples, the decoder 1403 (or another downstream device) may be configured to determine the category and / or level of ambient noise in the audio environment, and select the corresponding noise compensation process according to the noise compensation metadata received from the encoder 1401. Alternatively or additionally, the decoder 1403 may be configured to determine the audio environment location and select the corresponding noise compensation process according to the noise compensation metadata received from the encoder 1401.

[0201] Fig.16An example of an encoder and decoder block according to another implementation is shown. In this example, the encoder 1401 is shown as transmitting a coded audio bitstream 1402 and one or more types of metadata 1504 to a decoder 1403. According to this example, the encoder 1401 receives a noise metric 1602 from the decoder 1403. In this example, the noise metric 1602 indicates the level of ambient noise in the audio environment in which the decoder 1403 is located. In some implementations, the noise metric 1602 may indicate the type of ambient noise in the audio environment, such as whether the ambient noise corresponds to a conversation in the audio environment. The noise metric 1602 may have been received by the decoder 1403, by a control system configured to implement the decoder 1403 (e.g., by an implementation such as Figure 6 The encoder 1401 may be configured to determine the noise metric, for example, based on ambient noise information (e.g., a raw or processed microphone signal) provided by the decoder 1403, by a control system configured to implement the decoder 1403, or by another device in the same audio environment as the decoder 1403.

[0202] In this example, the encoder 1401 is configured to determine a noise compensation process to be performed in response to the noise metric 1602, such as disclosed elsewhere herein. The noise compensation process may, for example, correspond to the type of content provided via the audio bitstream 1402. According to this example, the metadata 1504 includes compensation metadata corresponding to the noise compensation process. In this implementation, the decoder 1403, a control system implementing the decoder 1403, or another device in the audio environment where the decoder 1403 is located is configured to determine the noise compensation process corresponding to the compensation metadata, and to apply the noise compensation process to the decoded audio bitstream 1402.

[0203] Fig.17 An example of a floor plan of an audio environment, in this example a living space, is shown. Figure 1 Sample, Fig.17 The types and quantities of the elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements.

[0204] According to this example, the environment 1700 includes a living room 1710 located at the upper left, a kitchen 1715 located at the lower middle, and a bedroom 1722 located at the lower right. The boxes and circles distributed in the living space represent a group of speakers 1705a-1705h, at least some of which may be smart speakers, placed in a convenient location in the space, but not following any standard prescribed layout (arbitrary placement). In some examples, the TV 1730 may be configured to at least partially implement one or more disclosed embodiments. In this example, the environment 1700 includes cameras 1711a-1711e, which are distributed throughout the environment. In some embodiments, one or more smart audio devices in the environment 1700 may also include one or more cameras. One or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may be located in or on the TV 1730, in a mobile phone, or in a smart speaker, such as one or more of speakers 1705b, 1705d, 1705e, or 1705h. Although cameras 1711a - 1711e are not shown in every depiction of environments 1700 presented in this disclosure, in some implementations, each environment 1700 may include one or more cameras.

[0205] Figure 18-23 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on data from one or more previous or "upstream" audio processes. In some cases, the one or more upstream audio processes may occur in the encoder. However, in some examples, the one or more upstream audio processes may have previously occurred at the decoder side, for example, after decoding but before the noise compensation module performs audio processing. Thus, Figure 18-23 Additional examples are shown in which the noise compensation module can be configured to determine the noise compensation process based at least in part on factors other than the input content type. Figure 18-23 The frame can be implemented, for example, by a control system, for example by Figure 2 The control system 210 is implemented.

[0206] Fig.18 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on data from an upstream linear processing module. In some examples, the processing module 1802 and the noise compensation module 1805 may be implemented by an instance of the control system 210 .

[0207] According to this example, audio data 1801 is input to processing module 1802. In some examples, audio data 1804 has been processed by processing module 1802. In some examples, processing module 1802 may be configured to perform a linear or nonlinear processing that would weaken or even destroy the audio enhancement caused by processing module 1802 if the noise compensation method implemented by noise compensation module 1805 is not constrained.

[0208] In this example, the processing module 1802 also provides processing data 1803 to the noise compensation module 1805. In some examples, the processing data 1803 may indicate whether the audio data 1804 has been processed by the processing module 1802. According to some examples, the processing data 1803 may indicate the degree of processing applied by the processing module 1802, such as the degree or extent of an equalization process.

[0209] In this example, the noise compensation module 1805 is configured to select and / or modify a noise compensation method based at least in part on the processing data 1803. In some examples, the noise compensation module 1805 may be configured to select a timbre-preserving noise compensation mode, such as the "music" noise compensation mode described herein, if the processing data 1803 indicates that the audio data 1804 has been processed by the processing module 1802. According to some examples, the noise compensation module 1805 may be configured to select an unconstrained noise compensation mode, such as the "movie" noise compensation mode described herein, if the processing data 1803 indicates that the audio data 1804 has not been processed by the processing module 1802. In this example, the noise compensation module 1805 is configured to output noise-compensated audio data 1806.

[0210] Fig.19 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dialogue enhancement process has occurred. In some examples, the dialogue enhancement module 1902 and the noise compensation module 1905 can be implemented by an instance of the control system 210 that is implemented on the decoder side, such as as part of the decoder 1403 or as described above with reference to Figure 14-16 The operation of the decoder 1403 is described later. In some examples, the dialogue enhancement module 1902 can be implemented by an instance of the control system 210 that is implemented on the encoder side, for example as part of the encoder 1401.

[0211] According to this example, audio data 1901 is input to dialogue enhancement module 1902. In some examples, audio data 1903 has been processed by dialogue enhancement module 1902. In this example, dialogue enhancement module 1902 also provides processing data 1904 to noise compensation module 1905. In some examples, processing data 1904 may indicate whether audio data 1903 has been processed by dialogue enhancement module 1902. According to some examples, processing data 1904 may indicate the degree of processing, for example, the degree or extent of the dialogue enhancement process applied by dialogue enhancement module 1902.

[0212] In this example, the noise compensation module 1905 is configured to select and / or modify a noise compensation method based at least in part on the processing data 1904. In some examples, the noise compensation module 1905 may be configured to select a timbre-preserving or "music" noise compensation mode if the processing data 1904 indicates that the audio data 1903 has been processed by the dialogue enhancement module 1902. In some implementations, the noise compensation module 1905 may be configured to select a timbre-preserving or "music" noise compensation mode if the processing data 1904 indicates that at least a threshold amount of dialogue enhancement is caused by the dialogue enhancement module 1902. In some examples, the noise compensation module 1905 may be configured to act only on a subset of the audio 1903 (e.g., a dialogue stream) to ensure that the audio subset is clearer if the dialogue enhancement module 1902 is turned on. According to some examples, the noise compensation module 1905 may be configured to select an unconstrained or "movie" noise compensation mode if the processing data 1904 indicates that the audio data 1903 has not been processed by the dialogue enhancement module 1902. In this example, the noise compensation module 1905 is configured to output noise compensated audio data 1906 .

[0213] Fig. 20 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream virtualization process has occurred. According to some examples, the virtualization module 2002 can be configured to provide a high degree of virtualization, for example, processing the audio data 2001 so that when played back by speakers of an audio system that does not include ceiling speakers, some of the reproduced audio appears to be emitted from ceiling speakers. Alternatively or additionally, in some examples, the virtualization module 2002 can be configured to provide other types of speaker virtualization, for example, processing the audio data 2001 so that when played back by physical speakers of an audio system, some of the reproduced audio appears to be emitted from additional virtual speakers that are not actual physical speakers.

[0214] In some examples, the virtualization module 2002 and the noise compensation module 2005 can be implemented by an instance of the control system 210 that is implemented on the decoder side, such as as part of the decoder 1403 or as described above with reference to Figure 14-16 After describing the operation of the decoder 1403. In some examples, the virtualization module 2002 can be implemented by an instance of the control system 210, which is implemented on the encoder side, for example as a part of the encoder 1401.

[0215] According to this example, audio data 2001 is input to virtualization module 2002. In some examples, audio data 2003 has been processed by virtualization module 2002. In this example, virtualization module 2002 also provides processing data 2004 to noise compensation module 2005. In some examples, processing data 2004 may indicate whether audio data 2003 has been processed by virtualization module 2002. According to some examples, processing data 2004 may indicate the degree of processing, such as the degree or scope of the virtualization process applied by virtualization module 2002.

[0216] In this example, the noise compensation module 2005 is configured to select and / or modify the noise compensation method based at least in part on the processing data 2004. In some examples, the noise compensation module 2005 can be configured to select the timbre retention or "music" noise compensation mode if the processing data 2004 indicates that the audio data 2003 has been processed by the virtualization module 2002. In some embodiments, the noise compensation module 2005 can be configured to select the timbre retention or "music" noise compensation mode if the processing data 2004 indicates that the virtualization module 2002 has caused at least a threshold amount of virtualization. According to some examples, the noise compensation module 2005 can be configured to select the unconstrained or "movie" noise compensation mode if the processing data 2004 indicates that the audio data 2003 has not been processed by the virtualization module 2002. In this example, the noise compensation module 2005 is configured to output the noise-compensated audio data 2006.

[0217] Fig.21 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream downmix or upmix process has occurred. In some examples, the downmix or upmix module 2102 and the noise compensation module 2105 can be implemented by an instance of the control system 210.

[0218] According to this example, audio data 2101 is input to downmix or upmix module 2102. In this example, audio data 2103 has been processed by downmix or upmix module 2102. In this example, downmix or upmix module 2102 also provides processing data 2104 to noise compensation module 2105. In some examples, processing data 2104 can indicate whether audio data 2103 is downmixed or upmixed. According to some examples, processing data 2104 can indicate the degree of processing, for example, the degree or range of downmix or upmix processing applied by downmix or upmix module 2102. An active mixer (for example, a mixer that mixes up or downmixes by checking content) can, for example, determine how the relevant parts of an audio signal are, so as to decide where to place the components of an audio signal. For static downmix or upmix, the exact coefficients used and / or the type of processing performed can also be applicable to the noise compensation process.

[0219] In this example, the noise compensation module 2105 may be configured to select and / or modify a noise compensation method based at least in part on the processing data 2104. In some examples, the noise compensation module 2105 may be configured to select a timbre-preserving or "music" noise compensation mode if the processing data 2104 indicates that the audio data 2103 was downmixed or upmixed by the upmix module 2102. According to some instances, the noise compensation module 2105 may be configured to select a timbre-preserving or "music" noise compensation mode if the processing data 2104 indicates that the audio data 2103 was downmixed or upmixed by the upmix module 2102.

[0220] In some examples, the downmix or upmix module 2102 may be configured to render audio data (e.g., audio data including audio objects and associated spatial metadata) as speaker feed signals. According to some such examples, the object rendering process may be based at least in part on the number of audio objects and the locations where they are rendered, including adjusting the type of noise compensation. For example, if the balance of the mix is ​​primarily in the surround speakers (which is determined by the location of the object or by metadata that determines the front / rear balance), the object rendering involves virtualization, and the audio data may be largely virtualized. In some such implementations, the noise compensation module 2205 may be configured to select a timbre-preserving or "music" noise compensation mode based on the associated processing data 2104. For situations where the balance of the mix is ​​primarily in the front speakers (or where the output speakers are located), virtualization may be relatively less. In some such implementations, the noise compensation module 2205 may be configured to select a non-timbre-preserving or "movie" noise compensation mode based on the associated processing data 2104. In this example, the noise compensation module 2105 is configured to output noise-compensated audio data 2106.

[0221] Fig. 22 An example is shown in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dynamic range compression process has occurred. In some examples, the dynamic range compression module 2202 and the noise compensation module 2205 can be implemented by an instance of the control system 210.

[0222] According to this example, audio data 2201 is input to a dynamic range compression module 2202. In this example, audio data 2204 has been processed by the dynamic range compression module 2202. In this example, the dynamic range compression module 2202 also provides processing data 2203 to the noise compensation module 2205. In some examples, the processing data 2203 may indicate whether the audio data 2204 has been processed by the dynamic range compression module 2202. According to some examples, the processing data 2203 may indicate the degree of processing, such as the degree or range of dynamic range compression applied by the dynamic range compression module 2202. In some such examples, the processing data 2203 may indicate the maximum, minimum, or average compression applied by the dynamic range compression module 2202. In some such examples, the processing data 2203 may indicate the weighted average compression applied by the dynamic range compression module 2202. Alternatively or additionally, the dynamic range compression module 2202 may provide an overall level to the noise compensation system (e.g., measured in the same manner as described elsewhere herein with reference to dialogue normalization metadata).

[0223] In this example, the noise compensation module 2205 is configured to select and / or modify a noise compensation method based at least in part on the processed data 2203. In some instances, the noise compensation module 2205 may be configured to select a timbre-preserving or "music" noise compensation mode if the processed data 2203 indicates that the dynamic range compression module 2202 applied at least a threshold level of compression. According to some instances, the noise compensation module 2205 may be configured to select a timbre-preserving or "movie" noise compensation mode if the processed data 2203 indicates that the dynamic range compression module 2202 applied less than a threshold level of compression. In this example, the noise compensation module 2205 is configured to output noise-compensated audio data 2206.

[0224] Fig.23 Another example is shown, in which the noise compensation module is configured to base the noise compensation process at least in part on whether an upstream dynamic range compression process has occurred. Fig. 22Some of the concepts described can be further extended by utilizing a level estimator within the encoder. In the case of Dolby Digital Plus, the output of the level estimator is indicated by a "dialog normalization" metadata value. In some cases, the dialog normalization metadata may be set by other mechanisms, such as by a fixed value indicating whether the audio data should be compressed at all.

[0225] Fig.23 An example of a system including dialogue normalization is shown. In some examples, the dynamic range compression module 2302 and the noise compensation module 2305 can be implemented by an instance of the control system 210 on the decoder side, such as by the above reference Figure 14-16 In some examples, the encoder 2310 may be the decoder 1403 described above. Figure 14-16 An example of encoder 1401 is described.

[0226] exist Fig.23 , unencoded audio 2301 is provided to encoder 2310. In this example, encoder 2310 is configured to implement level measurement of content (including audio 2301 in this example) and output dialogue normalization metadata value 2314. In some such examples, dialogue normalization metadata value 2314 allows different types of content to be appropriately leveled by downstream audio processing equipment. In the United States, the Commercial Advertising Loudness Mitigation (CALM) Act requires broadcast systems to employ such a level control process.

[0227] In systems that are not configured to comply with the CALM Act, in some cases, the dialogue normalization metadata value 2314 may be set to a fixed value. If the dialogue normalization metadata value 2314 is set to a fixed value, the noise compensation system may fall back to a timbre-preserving or "music" noise compensation mode in some cases because the fixed dialogue normalization metadata value is typically set to achieve the maximum amount of compression. If the dialogue normalization metadata value 2314 is set to a fixed value that results in the least compression, then in some examples, the noise compensation system may implement an unconstrained or "movie" noise compensation mode.

[0228] exist Fig.23 In the example, the audio to be leveled 2313 is provided to the dynamic range compression module 2302, which in some examples can be implemented by an instance of the decoder 1403. In this example, the dialogue normalization metadata value 2314 is also provided to the dynamic range compression module 2302.

[0229] In this example, audio data 2304 has been processed by dynamic range compression module 2302. In this example, dynamic range compression module 2302 also provides processed data 2303 to noise compensation module 2305. In some examples, processed data 2303 may indicate whether audio data 2304 has been processed by dynamic range compression module 2302. According to some examples, processed data 2303 may indicate the extent of processing, such as the extent or range of dynamic range compression applied by dynamic range compression module 2302. In some such examples, processed data 2303 may indicate a maximum, minimum, or average compression applied by dynamic range compression module 2302. In some such examples, processed data 2303 may indicate a weighted average compression applied by dynamic range compression module 2302. In some such examples, processed data 2303 may indicate an output target compression level.

[0230] In this example, the noise compensation module 2305 is configured to select and / or modify a noise compensation method based at least in part on the processing data 2303. In some instances, the noise compensation module 2305 may be configured to select a timbre-preserving or "music" noise compensation mode when the processing data 2303 indicates that the dynamic range compression module 2302 has applied any compression. In some instances, the noise compensation module 2305 may be configured to select a timbre-preserving or "music" noise compensation mode when the processing data 2303 indicates that the dynamic range compression module 2302 has applied at least a threshold level of compression. According to some examples, the noise compensation module 2305 may be configured to select an unconstrained or "movie" noise compensation mode when the processing data 2303 indicates that the dynamic range compression module 2302 has applied less than a threshold level of compression. In this example, the noise compensation module 2305 is configured to output noise-compensated audio data 2306 to the speaker 104.

[0231] In some examples, such as Fig.23As shown, the dialogue normalization metadata value 2314 can be provided to the noise compensation module 2305. If the dialogue normalization metadata value 2314 indicates that the content is at a high level (in other words, the content has a small amount of "headroom" between the current level and the applicable level limit), then in some examples, the noise compensation module 2305 can be configured to implement a timbre-preserving or "music" noise compensation mode. If the dialogue normalization metadata value 2314 indicates that the content is at a low level (in other words, the content has a large amount of headroom), then in some examples, the noise compensation module 2305 can be configured to implement an unconstrained or "movie" noise compensation mode. If the dialogue normalization metadata value 2314 indicates that the content is at an intermediate level (in other words, the content has an intermediate amount of headroom), then in some examples, the noise compensation module 2305 can be configured to implement a hybrid noise compensation mode, such as a partial timbre-preserving noise compensation mode. In some examples, the dialogue normalization metadata value 2314 can be set to a fixed value, which can indicate that the content has been authored by a specific tool (e.g., a DVD encoding tool). In some such examples, the noise compensation module 2305 can be configured to implement a noise compensation mode corresponding to a fixed value.

[0232] Some aspects of the disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include a programmable general-purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0233] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed or otherwise configured) to perform the desired processing on the audio signal, including the execution of one or more examples of the disclosed method. Alternatively, embodiments of the disclosed system (or its elements) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations including one or more examples of the disclosed method. Alternatively, elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP that is configured (e.g., programmed) to perform one or more examples of the disclosed method, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed method may be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0234] Another aspect of the disclosure is a computer-readable medium (eg, a disk or other tangible storage medium) storing code for executing (eg, executable to implement) one or more examples of the disclosed method or steps thereof.

[0235] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those skilled in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A content stream processing method, comprising: receiving, by the control system and via the interface system, a content stream including audio data; determining, by the control system, a content type corresponding to the content stream; determining, by the control system and based at least in part on the type of content, a noise compensation method; determining, by the control system, a noise estimate of an audio environment in which the content stream is to be reproduced, wherein determining the noise estimate comprises determining one or more metrics corresponding to a reverberation of the environment, a frequency response of the environment, a playback characteristic of one or more audio reproduction transducers of the environment; performing, by the control system and based at least in part on the noise estimate, the noise compensation method on the audio data to produce noise compensated audio data; rendering, by a control system, the noise compensated audio data for reproduction via a set of audio reproduction transducers of an audio environment to produce a rendered audio signal; and providing the rendered audio signal to at least some of the set of audio reproduction transducers of the audio environment via an interface system, Wherein, in the noise compensation, the gain applied to the lowest frequency band of the high frequency range is equal to the gain applied to the highest frequency band of the low frequency range, or is within a predetermined range of the gain. 2 . The method of claim 1 , wherein determining the noise compensation method comprises selecting a noise compensation method from a plurality of noise compensation methods, each noise compensation method corresponding to one or more content types among a plurality of content types.

3. The method of claim 2, wherein the plurality of content types includes at least one music content type. 4 . The method according to claim 3 , wherein the noise compensation method corresponding to the at least one music content type includes a first method corresponding to a low frequency range and a second method corresponding to a high frequency range. 5 . The method of claim 4 , wherein the first method comprises allowing a first gain applied to a first frequency band within the low frequency range to be independent of a second gain applied to a second frequency band within the low frequency range.

6. The method according to claim 4, wherein: The second method consists in constraining all gains in the frequency band applied in the high frequency range to be equal.

7. A method according to claim 4, wherein the second method includes constraining the difference between a first gain and a second gain to be less than or equal to a threshold amount, wherein the first gain is applied in a first frequency band of the high frequency range and the second gain is applied in a second frequency band of the high frequency range, the first frequency band being adjacent to the second frequency band.

8. The method of claim 2, wherein the plurality of content types comprises at least one of a movie content type or a television program content type.

9. The method of claim 1, wherein determining the content type is based at least in part on determining a content provider of the content stream.

10. The method of claim 1, wherein determining the content type is based at least in part on user input regarding the content stream.

11. The method of claim 10, wherein the user input is received via a voice command to a virtual assistant.

12. The method of claim 1, wherein the content stream includes content metadata, and wherein determining the content type is based at least in part on the content metadata.

13. The method of claim 1, wherein the control system is configured to implement an audio classifier, and wherein determining the content type is based at least in part on an audio classification generated by the audio classifier.

14. The method according to claim 1, wherein: Determining the noise estimate includes at least one of receiving the noise estimate from a noise estimation module or computing the noise estimate based on microphone signals of one or more microphones in the audio environment.

15. The method of claim 1, wherein determining a noise compensation method is based at least in part on a time of day.

16. The method of claim 15, wherein the nighttime noise compensation method comprises controlling playback levels of at least some frequency bands of the reproduced audio data to be lower than playback levels of corresponding frequency bands of the reproduced audio data corresponding to the daytime noise compensation method.

17. The method according to claim 16, wherein: The at least some frequency bands correspond to bass frequency bands.

18. The method of claim 1, wherein determining a noise compensation method is based at least in part on a type of ambient noise in the audio environment.

19. The method of claim 18, wherein the type of ambient noise corresponds to a conversation, and wherein the noise compensation method comprises controlling a playback level of the reproduced audio data to be lower than the ambient noise level.

20. The method according to claim 1, wherein: The noise estimate indicates an estimated noise level for each of the plurality of frequency bands.

21. The method of claim 1, wherein determining the content type is based at least in part on determining an application that implements decoding of the content stream.

22. The method of claim 1, wherein at least one noise compensation method comprises a demonstration version and a regular version, and wherein the demonstration version involves applying a higher gain in one or more frequency bands than the regular version.

23. The method of any one of claims 1-22, further comprising receiving metadata with the content stream, wherein determining a noise compensation method is based at least in part on the metadata.

24. The method of claim 23, wherein the metadata corresponds to a dynamic range or a reference level of the audio data.

25. The method according to claim 23 or 24, wherein: The metadata is dialogue-normalized metadata for Dolby Digital, Dolby Digital Plus or AC-4 audio technology, or reference-level metadata for the "High Efficiency Advanced Audio Coding" audio coding format.

26. The method of any of claims 1-22, wherein determining a noise compensation method is based at least in part on a dynamic range of at least one audio reproduction transducer of the audio environment.

27. An apparatus configured to implement the method according to any one of claims 1-26.

28. A system configured to implement the method according to any one of claims 1-26.

29. One or more non-transitory media having stored thereon software comprising instructions for controlling one or more devices to perform the method of any one of claims 1-26.

30. An audio processing system, comprising: A control system configured to: determining a content type corresponding to a content stream including audio data; determining a noise compensation method based at least in part on the type of content; determining a noise estimate of an audio environment in which the content stream is to be reproduced, wherein determining the noise estimate comprises determining one or more metrics corresponding to a reverberation of the environment, a frequency response of the environment, a playback characteristic of one or more audio reproduction transducers of the environment; performing the noise compensation method on the audio data based at least in part on the noise estimate to produce noise compensated audio data; as well as rendering the noise compensated audio data for reproduction via a set of audio reproduction transducers of an audio environment to produce a rendered audio signal; as well as an interface system configured to receive at least some of the set of audio reproduction transducers of the audio environment including the content stream and to provide the rendered audio signal to the audio environment, Wherein, in the noise compensation, the gain applied to the lowest frequency band of the high frequency range is equal to the gain applied to the highest frequency band of the low frequency range, or is within a predetermined range of the gain.

31. The audio processing system of claim 30, wherein: The control system is configured to perform the method according to any one of claims 1-22 and 26.

32. An audio processing system according to claim 30 or 31, wherein: The interface system is configured to receive metadata and the content stream, Therein, the control system is configured to determine a noise compensation method based at least in part on the metadata.

33. A content stream processing device, comprising: one or more processors; and One or more non-transitory storage media having instructions stored thereon, which when executed by the one or more processors cause the one or more processors to perform the method according to any one of claims 1-26.

34. A computer program product having instructions which, when executed by a computing device or system, cause the computing device or system to perform the method according to any one of claims 1-26.

Images