Audio processing method, apparatus, system, and storage medium

Abstract

Attenuation or "gaps" can be inserted into at least a first frequency range of at least first and second audio playback signals of a content stream during at least a first time interval to generate at least first and second modified audio playback signals. Corresponding audio device playback sounds can be provided by at least first and second audio devices. At least one microphone can detect at least first and second audio device playback sounds and can generate corresponding microphone signals. Audio data can be extracted from the microphone signals at least in the first frequency range to produce extracted audio data. A far-field audio environment impulse response and / or an audio environment noise can be estimated based at least in part on the extracted audio data.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. Provisional Application No. 63 / 201561, filed May 4, 2021, and U.S. Provisional Application No. 63 / 120887, filed December 8, 2020, all of which are incorporated herein by reference in their entirety. Technical Field

[0003] This disclosure relates to audio processing systems and methods. Background Technology

[0004] Audio devices and systems are widely deployed. While existing systems and methods for estimating the impulse response and ambient noise of audio environments can provide satisfactory results in some cases, improvements in systems and methods are desired.

[0005] Symbols and nomenclature

[0006] Throughout this disclosure, including in the claims, the terms "speaker," "loudspeaker," and "audio reproduction transducer" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may undergo different processing in different branches of the circuitry coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, the phrase “to” a signal or data (e.g., to filter, scale, transform, or apply gain to the signal or data) is broadly used to mean performing an operation directly on the signal or data, or on a processed version of the signal or data (e.g., on a signal version that has been pre-filtered or pre-processed before the operation was performed on it).

[0008] Throughout this disclosure, including in the claims, the term "system" is used broadly to refer to a device, system, or subsystem. For example, a subsystem implementing a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, wherein the subsystem generates M inputs and the other XM inputs are received from an external source) may also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used broadly to mean a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field-programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipelined processing of audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0010] Throughout this disclosure, including in the claims, the terms "coupled" or "coupled" are used to indicate a direct or indirect connection. Therefore, if a first device is coupled to a second device, the connection can be either a direct connection or an indirect connection via other devices and connections.

[0011] As used herein, a "smart device" is an electronic device typically configured to communicate with one or more other devices (or networks) that can interact with and / or operate autonomously to some extent via various wireless protocols such as Bluetooth, Zigbee, Near Field Communication, Wi-Fi, Li-Fi, 3G, 4G, and 5G. Some notable types of smart devices include smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, tablets and phablets, smartwatches, smart bracelets, smart keychains, and smart audio devices. The term "smart device" can also refer to devices that exhibit some properties of ubiquitous computing, such as artificial intelligence.

[0012] In this paper, we use the term "intelligent audio device" to refer to an intelligent device, which can be either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant's functionality). A single-purpose audio device is one that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to perform a single purpose (e.g., a television (TV)). For example, while a TV can typically play (and is considered capable of playing) audio from program material, in most cases, modern TVs run some kind of operating system on which applications, including TV-watching applications, run natively. In this sense, a single-purpose audio device with one or more speakers and one or more microphones is often configured to run native applications and / or services to directly utilize one or more speakers and one or more microphones. Some single-purpose audio devices may be configured to be grouped together to enable audio playback in zones or user-configured areas.

[0013] A common type of multipurpose audio device is an audio device that implements at least some aspects of virtual assistant functionality, but other aspects of virtual assistant functionality can be implemented by one or more other devices, such as one or more servers configured to communicate with the multipurpose audio device. Such a multipurpose audio device may be referred to herein as a "virtual assistant." A virtual assistant is a device that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and / or at least one camera) (e.g., a smart speaker or voice assistant integrated device). In some examples, a virtual assistant can provide the ability to use multiple devices (different from the virtual assistant) for applications that are, in some sense, cloud-enabled or otherwise not fully implemented within or on top of the virtual assistant itself. In other words, at least some aspects of virtual assistant functionality, such as speech recognition, can be (at least partially) implemented by one or more servers or other devices with which the virtual assistant can communicate via a network such as the Internet. Virtual assistants can sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants can work together in the sense that one (e.g., the one most confident in hearing the wake word) responds to the wake word. In some implementations, the connected virtual assistants can form a constellation that can be managed by a main application, which can be (or implement) the virtual assistants.

[0014] In this document, "wake word" is used broadly to refer to any sound (e.g., a word spoken by a person, or some other sound) in which a smart audio device is configured to wake up in response to the detection ("hearing") of a sound (using at least one microphone contained in or coupled to the smart audio device, or at least one other microphone). In this context, "wake up" means that the device enters a state of waiting for (in other words, listening for) a sound command. In some cases, what may be referred to as a "wake word" in this document may include more than one word, such as a phrase.

[0015] In this paper, the term "wake word detector" refers to a device (or software including instructions for configuring the device) configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability of detecting a wake word exceeds a predefined threshold. For example, the threshold could be a pre-determined threshold adjusted to provide a reasonable trade-off between false acceptance and false rejection rates. Following a wake word event, the device may enter a state (which may be referred to as a "wake-up" state or a "focused" state) in which it listens for commands and passes the received commands to a larger, more computationally intensive recognizer.

[0016] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some cases, video signals, at least in part of which are intended to be heard together. Examples include music anthologies, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized speech responses from intelligent assistants, and so on. In some cases, a content stream may include multiple versions of at least a portion of an audio signal, such as the same dialogue in more than one language. In this case, only one version of the audio data or a portion thereof is intended to be reproduced at a time (e.g., a version corresponding to one language). Summary of the Invention

[0017] At least some aspects of this disclosure can be implemented via one or more audio processing methods. In some cases, one or more methods may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transient media. Some methods may involve the control system causing a first gap to be inserted into a first frequency range of a first audio playback signal of the content stream during a first time interval of the content stream to generate a first modified audio playback signal for a first audio device in an audio environment. The first gap may be, or may cause, attenuation of the first audio playback signal in the first frequency range.

[0018] Some such methods may involve a control system causing a first audio device to play back a first modified audio playback signal to generate a sound played back by the first audio device. Some such methods may involve a control system causing a first gap during a first time interval of the content stream to be inserted into a first frequency range of a second audio playback signal of the content stream to generate a second modified audio playback signal for a second audio device in an audio environment. Some such methods may involve a control system causing a second audio device to play back a second modified audio playback signal to generate a sound played back by the second audio device.

[0019] Some such methods may involve a control system causing at least one microphone in an audio environment to detect at least a first audio device playback sound and a second audio device playback sound, and generating a microphone signal corresponding to the at least first audio device playback sound and the second audio device playback sound. Some such methods may involve a control system extracting audio data from the microphone signal in at least a first frequency range to generate the extracted audio data. Some such methods may involve a control system estimating at least one of a far-field audio environment impulse response or audio environment noise based at least in part on the extracted audio data.

[0020] Some such methods may also involve causing an unmodified audio playback signal of a content stream to be played back by a target audio device to generate a sound played back by the target audio device. Some such methods may also involve a control system estimating at least one of the audibility of a target audio device or the location of a target audio device based at least in part on extracted audio data. In some such examples, the unmodified audio playback signal does not include a first gap. In some such examples, the unmodified audio playback signal may not include gaps inserted into any frequency range. According to some such examples, a microphone signal may also correspond to the sound played back by the target audio device.

[0021] Based on some examples, generating the first modified audio playback signal may involve a control system causing the content stream to be modified in the second to third... N During the time interval, the second to the third N The gaps are inserted into the second to third intervals of the first audio playback signal. N Within the frequency range, N It is an integer greater than 2. In some such examples, generating a second modified audio playback signal may involve the control system causing the second to third modified audio playback signals in the content stream to be generated. N During the time interval, the second to the third N The gap is inserted into the second to the third of the second audio playback signal. N Within the frequency range.

[0022] Some methods may involve a control system that causes a first gap to be inserted into the third to fourth intervals of the content stream during a first time interval. M In the first frequency range of the audio playback signal, to generate the third to the fourth frequencies for the audio environment. M The third to the fourth audio equipment M Modify the audio playback signal, where M It is an integer greater than three. Some such methods may involve a control system making the third to the... M Audio device playback of the third to the fourth M Modify the corresponding instance of the audio playback signal to generate the third to the fourth. M The audio device plays back sound. In some such examples, generating a microphone signal may involve a control system causing at least one microphone in the audio environment to detect a third to a fourth microphone. M The audio device plays back the sound. In some such examples, the first to the second... M Modifying the audio playback signal can involve the control system causing changes in the second to third content stream. N During the time interval, the second to the third N The gap was inserted into the first to the second M The second to the third audio playback signal N Within the frequency range.

[0023] In some examples, at least the first gap can be perceived as masked. According to some examples, inserting the first gap may involve transmitting an instruction to insert the first gap. In other examples, inserting the first gap may involve inserting the first gap itself.

[0024] In some examples, at least a first frequency range may correspond to a frequency band. In some such examples, a frequency band may be one of several frequency bands equally spaced on a mel scale. However, in some cases, at least a first frequency range may correspond to a frequency bin.

[0025] According to some examples, causing a first audio device to play back a first modified audio playback signal may involve transmitting instructions to the first audio device to play back the first modified audio playback signal. In some examples, the first modified audio playback signal and the second modified audio playback signal may be at least partially related.

[0026] Some or all of the operations, functions, and / or methods described herein can be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. M Devices, etc. Therefore, some innovative aspects of the subject matter described in this disclosure can be realized via one or more non-transitory media on which software is stored.

[0027] At least some aspects of this disclosure can be implemented via apparatus. For example, one or more devices may be configured to at least partially perform the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0028] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become clear from the description, drawings, and claims. Note that the relative dimensions in the following figures may not be drawn to scale. Attached Figure Description

[0029] The same reference numerals and names in the various figures indicate the same elements.

[0030] Figure 1A An example system for estimating background noise in an audio environment is shown.

[0031] Figure 1B An example of an audio environment is shown.

[0032] Figure 1C This is a block diagram illustrating examples of components of an apparatus capable of implementing various aspects of the present invention.

[0033] Figure 2A This is an example of modifying the spectrogram of an audio playback signal.

[0034] Figure 2B This is a graph showing an example of gaps in the frequency domain.

[0035] Figure 2C This is a diagram showing an example of a gap in the time domain.

[0036] Figure 2D This is an example of modifying the audio playback signal, including the spacing between multiple audio devices arranged for an audio environment.

[0037] Figure 3A This is a graph showing examples of filter responses used to create gaps and filter responses used to measure frequency ranges of microphone signals used during a measurement session.

[0038] Figure 3B , 3C 3D, 3E, 3F, 3G, 3H, 3I, and 3J are diagrams illustrating examples of gap allocation strategies.

[0039] Figure 4 , 5A Figures 5 and 5B are flowcharts illustrating an example of how multiple audio devices coordinate a measurement session according to some implementations.

[0040] Figure 6 An example of two orchestrated audio devices participating in a measurement session and sharing reference data is shown.

[0041] Figure 7 An example of an audibility map corresponding to an audio device in an audio environment is shown.

[0042] Figure 8A This shows another example of an audio environment.

[0043] Figure 8B This shows another example of an audio environment.

[0044] Figure 8C An overview can be provided by, for example Figure 1C The flowchart shows an example of a method performed by the apparatus shown.

[0045] Figure 9 A block diagram of an example system for arranging gap insertions is presented.

[0046] Figure 10 This is a flowchart outlining another example of the disclosed methods. Detailed Implementation

[0047] To achieve compelling spatial reproduction of media and entertainment content, the physical layout and relative capabilities of available speakers should be evaluated and considered. Similarly, to provide high-quality voice-driven interaction (with virtual assistants and remote speakers), users need to both be heard and hear the dialogue reproduced via loudspeakers. The combined utility for users is expected to increase as more collaborative devices are added to the audio environment, as devices will be more generally within convenient voice range. A greater number of speakers allows for a better sense of immersion due to the spatiality of media presentation.

[0048] Sufficient coordination and cooperation between devices can make these opportunities and experiences possible. Acoustic information about each audio device is a key component of this coordination and cooperation. Such acoustic information can include the audibility of each loudspeaker from different locations in the audio environment, as well as the amount of noise in the audio environment.

[0049] Some previous methods of mapping and calibrating constellations of smart audio devices required a dedicated calibration process that involved playing known stimuli from the audio devices (often one device at a time) while one or more microphones recorded. While this process could be appealing to specific user groups through creative sound design, the need to repeatedly re-perform the process as devices are added, removed, or even simply repositioned hinders widespread adoption. Imposing such a process on users can interfere with normal device operation and may frustrate some users. A more basic but also popular approach is manual user intervention via a guided process, guided by a software application (“Application”) and / or user-instructed physical location of the audio devices in the audio environment. This approach presents further barriers to user adoption and provides relatively less information to the system compared to a dedicated calibration process.

[0050] Calibration and mapping algorithms typically require some basic acoustic information about each audio device in the audio environment. Many such methods have been proposed, using a range of different basic acoustic measurements and the acoustic properties being measured. Examples of acoustic properties derived from microphone signals for use in such algorithms include: o Estimation of the physical distance between devices (acoustic ranging); estimation of the angle between devices (direction of arrival (DoA)); Estimation of the impulse response between devices (e.g., via swept-frequency sine wave stimulation or other measurement signals); and o Estimation of background noise.

[0051] However, existing calibration and mapping algorithms are often unable to respond to changes in the acoustic environment of the audio environment, such as the movement of people within the audio environment or the repositioning of audio devices within the audio environment.

[0052] The problem of estimating background noise (indicating both background noise and playback content) from a microphone output signal has been proposed by attempting to correlate the playback content with the microphone output signal and subtracting an estimate of the playback content captured by the microphone (referred to as "echo") from the microphone output. The content of the microphone output signal generated when the microphone captures sound, indicating the playback content X emitted from one or more speakers and the background noise N, can be expressed as WX + N, where W is a transfer function determined by the sound emitted from one or more speakers indicating the playback content, the microphone, and the environment (e.g., a room) in which the sound propagates from one or more speakers to the microphone. For example, methods proposed academically for estimating noise N (see reference...) Figure 1A In the description, a linear filter W' is adapted to facilitate the estimation of the echo (the playback content captured by the microphone) WX, W'X, so as to be subtracted from the microphone output signal. Even though nonlinearity exists in the system, nonlinear implementations of the filter W' are rarely implemented due to computational costs.

[0053] Figure 1A An example of a system for estimating background noise in an audio environment is shown. In this example, Figure 1A This is a diagram of a system used to implement the conventional method (sometimes called echo cancellation) mentioned above for estimating background noise in an audio environment in which one or more speakers emit sound indicative of playback content. A playback signal X is presented to a speaker system S (e.g., a single speaker) in an audio environment E. A microphone M is located in the same audio environment E. In response to the playback signal X, the speaker system S emits sound that reaches the microphone M, having ambient noise N and user speech L. The microphone output signal is Y = WX + N + L, where W represents the transfer function, which is the combined response of the speaker system S, the playback environment E, and the microphone M.

[0054] Depend on Figure 1A A common approach to system implementation is to adaptively infer the transfer function W from Y and X using any of various adaptive filter methods. For example... Figure 1AAs indicated in the diagram, the linear filter W' is adaptively determined as an approximation of the transfer function W'. The playback signal content ("echo") indicated by the microphone signal M is estimated as W'X, and W'X is subtracted from Y to produce estimates of noise N and user speech L, Y' = WX - W'X + N + L. For noise compensation applications, it is of interest that if a positive bias exists in the estimation, then adjusting the level of X proportionally to Y' creates a feedback loop. An increase in Y', in turn, increases the level of X, which introduces an upward bias in the estimates of N and L (Y'), which in turn increases the level of X, and so on. This form of solution will rely heavily on the ability of the adaptive filter W' to subtract W'X from Y to remove a significant amount of echo WX from the microphone signal M.

[0055] In order to maintain Figure 1A System stability typically requires further filtering of the signal Y'. Since most noise compensation implementations in this field exhibit lackluster performance, most solutions will likely, in practice, down-bias the noise estimate and introduce aggressive time smoothing to maintain system stability. This comes at the cost of reduced and very slow compensation.

[0056] Systems that claim to implement the academic methods for noise estimation mentioned above (see reference) Figure 1A Conventional implementations of the described type typically ignore problems that arise during implementation, including some or all of the following: While academic simulations of the solution indicate an echo reduction of over 40 dB, practical implementations typically achieve echo reductions far below 40 dB due to nonlinearity, the presence of background noise, and the non-stationarity of the echo path W. This means that any measurement of background noise is biased by residual echoes. Sometimes, ambient noise and specific playback content can cause "leakage" in such systems (e.g., when playback content excites the nonlinear region of the playback system due to buzzing, rattling, and distortion). In these cases, the microphone output signal contains a large amount of residual echo, which will be incorrectly interpreted as background noise. In such cases, as the residual error signal increases, the adaptation of filter W' also becomes unstable. Moreover, the adaptation of filter W' becomes unstable when the microphone signal is affected by high levels of noise; and The computational complexity required to generate the noise estimate (Y') is high, but this noise estimate (Y') is useful for performing noise-compensated media playback (NCMP) that runs over a wide frequency range, such as the frequency range that covers the playback of typical music.

[0057] Noise compensation for compensating for ambient noise conditions (e.g., automatic leveling of speaker playback content) is a well-known and desirable feature, but it has not been optimally implemented previously. Measuring both ambient noise conditions and speaker playback content using microphones poses a significant challenge to noise estimation (e.g., online noise estimation) required for noise compensation.

[0058] Because people in an audio environment can typically be located beyond the critical acoustic distance of any given room, echoes introduced from other devices at similar distances can still represent a significant echo effect. Even if sophisticated multi-channel echo cancellation is available and somehow achieves the desired performance, the logistics of providing a remote echo reference for the canceller incurs unacceptable bandwidth and complexity costs.

[0059] Some disclosed implementations provide methods for continuously calibrating the constellation of audio devices via a persistent (e.g., continuous or at least ongoing) characterization of the acoustic space, including people, devices, and audio conditions (e.g., noise and / or echo). In some disclosed examples, such a process continues even when media is being played back through audio devices in the audio environment.

[0060] As used herein, a “gap” in a playback signal refers to a time (or time interval) in the playback signal at which playback content is lost (or has a level below a predetermined threshold). For example, a “gap” could be an attenuation of playback content within a frequency range during the time interval. In some disclosed embodiments, gaps can be inserted in one or more frequency ranges of the audio playback signal of the content stream to produce a modified audio playback signal that can be reproduced or “played back” in an audio environment. In some such embodiments, gaps can be inserted in... N During each time interval N A gap is inserted into the audio playback signal. N Within a frequency range.

[0061] According to some of these implementations, M Audio devices can be programmed with gaps in time and frequency, allowing for accurate detection of the far field (for each device) at the gap frequencies and time intervals. These "programmed gaps" are an important aspect of this disclosure. In some examples, M It can be a number corresponding to all audio devices in the audio environment. In some cases, M This can be a number corresponding to all audio devices in the audio environment except the target audio device, where the audio being played back is the audio device in the audio environment. M One or more microphones in a choreographed device (e.g., the audio environment). MThe audio device (one or more microphones of the orchestrated audio device) samples the audio device to, for example, evaluate the relative audibility, position, nonlinearity, and / or other characteristics of the target audio device. In some examples, the target audio device can reproduce an unmodified audio playback signal excluding gaps inserted into any frequency range. In other examples, M It can be a number corresponding to a subset of the audio devices in the audio environment (e.g., multiple participating non-target audio devices).

[0062] The desired outcome is that the spacing in the arrangement has a low perceptual impact (e.g., negligible perceptual impact) on the listener in the audio environment. Therefore, in some examples, the spacing parameter can be selected to minimize the perceptual impact.

[0063] In some examples, when a modified audio playback signal is being played back in an audio environment, the target device can reproduce the unmodified audio playback signal, excluding gaps inserted into any frequency range. In such examples, it is possible to reproduce the unmodified audio playback signal from the audio environment where the modified audio playback signal is being played back. M The relative audibility and / or location of the target device are estimated from the perspective of each audio device.

[0064] Figure 1B An example of an audio environment is shown. Similar to the other figures provided in this article, Figure 1B The types and quantities of elements shown are provided as examples only. Other implementations may include more, fewer, and / or different types and quantities of elements.

[0065] According to this example, the audio environment 100 includes a main living space 101a and a room 101b adjacent to the main living space 101a. Here, a wall 102 and a door 111 separate the main living space 101a from the room 101b. In this example, the amount of acoustic separation between the main living space 101a and the room 101b depends on whether the door 111 is open or closed, and if it is open, to what extent the door 111 is open.

[0066] In Figure 1B At the corresponding time, the smart TV (TV) 103a is located within the audio environment 100. According to this example, the smart TV 103a includes a left loudspeaker 103b and a right loudspeaker 103c.

[0067] In this example, in relation to Figure 1B At the corresponding time, smart audio devices 104, 105, 106, 107, 108, and 109 are also located within the audio environment 100. According to this example, each of smart audio devices 104-109 includes at least one microphone and at least one amplifier. However, in this case, smart audio devices 104-109 include amplifiers of various sizes and with various capabilities.

[0068] According to this example, at least one acoustic event is occurring in audio environment 100. In this example, an acoustic event is caused by speaker 110, who is uttering voice command 112.

[0069] In this example, another acoustic event is caused at least in part by variable element 103. Here, variable element 103 is a door to the audio environment 100. According to this example, as door 103 opens, sound 105 from outside the environment can be perceived more clearly inside the audio environment 100. Moreover, the changing angle of door 103 alters some echo paths within the audio environment 100. According to this example, element 104 represents a variable element of the impulse response of the audio environment 100 caused by the changing position of door 103.

[0070] Figure 1C This is a block diagram illustrating examples of components of a device capable of implementing various aspects of this disclosure. As with the other figures provided herein, Figure 1C The types and quantities of elements shown are provided by way of example only. Other embodiments may include more, fewer, and / or different types and quantities of elements. According to some examples, device 150 may be configured to perform at least some of the methods disclosed herein. In some embodiments, device 150 may be or may include one or more components of an audio system. For example, in some embodiments, device 150 may be an audio device, such as a smart audio device. Figure 1B In the examples shown, smart TV 103a and smart audio devices 104-109 are instances of device 150. According to some examples, Figure 1B The audio environment 100 may include orchestration devices, such as what may be referred to herein as a smart home hub. A smart home hub (or other orchestration device) may be an instance of device 150. In other examples, device 150 may be a mobile device (such as a cellular phone), a laptop computer, a tablet device, a television, or another type of device.

[0071] According to some alternative implementations, device 150 may be or may include a server. In some such examples, device 150 may be or may include an encoder. Thus, in some cases, device 150 may be a device configured for use in an audio environment such as a home audio environment, while in other cases, device 150 may be a device configured for use in the “cloud,” such as a server.

[0072] In this example, device 150 includes an interface system 155 and a control system 160. In some embodiments, the interface system 155 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 155 may be configured to exchange control information and associated data with audio devices in the audio environment. In some examples, the control information and associated data may be related to one or more software applications being executed by device 150.

[0073] In some implementations, the interface system 155 may be configured to receive or provide content streams. Content streams may include audio data. Audio data may include, but is not limited to, audio signals. In some cases, audio data may include spatial data, such as channel data and / or spatial metadata. For example, the metadata may already be provided by something referred to herein as an "encoder." In some examples, content streams may include video data and audio data corresponding to the video data.

[0074] Interface system 155 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some implementations, interface system 155 may include one or more wireless interfaces. Interface system 155 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, interface system 155 may include a control system 160 and a memory system (such as...) Figure 1C The system 160 may include one or more interfaces between the optional memory system 165 shown. However, in some cases, the control system 160 may include a memory system. In some embodiments, the interface system 155 may be configured to receive input from one or more microphones in the environment.

[0075] In some embodiments, the control system 160 may be configured to perform at least partially the methods disclosed herein. The control system 160 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0076] In some implementations, the control system 160 may reside in more than one device. For example, in some implementations, a portion of the control system 160 may reside in a device within one of the environments described herein, while another portion of the control system 160 may reside in a device outside that environment (such as a server, mobile device (e.g., a smartphone or tablet computer)). In other examples, a portion of the control system 160 may reside in a device within one of the environments described herein, while another portion of the control system 160 may reside in one or more other devices in the environment. For example, the control system functionality may be distributed across multiple smart audio devices in the environment, or it may be shared by an orchestration device (such as a smart home hub, referred to herein) and one or more other devices in the environment. In other examples, a portion of the control system 160 may reside in a device implementing a cloud-based service (such as a server), while another portion of the control system 160 may reside in another device implementing a cloud-based service (such as another server, storage device, etc.). In some examples, the interface system 155 may also reside in more than one device.

[0077] Some or all of the methods described herein can be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices, such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. One or more non-transitory media may, for example, reside on... Figure 1C The optional memory system 165 and / or control system 160 shown herein. Thus, various innovative aspects of the subject matter described herein can be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to perform some or all of the methods disclosed herein. For example, the software may be provided by, for example, a... Figure 1C The control system 160 is executed by one or more components of a control system.

[0078] In some examples, device 150 may include Figure 1CThe optional microphone system 170 is shown. The optional microphone system 170 may include one or more microphones. According to some examples, the optional microphone system 170 may include an array of microphones. In some cases, the microphone array may be configured to receive side beamforming, for example, according to instructions from the control system 160. In some examples, the microphone array may be configured to determine, for example, direction of arrival (DoA) and / or time of arrival (ToA) information according to instructions from the control system 160. Alternatively or additionally, the control system 160 may be configured to determine, for example, direction of arrival (DoA) and / or time of arrival (ToA) information based on microphone signals received from the microphone system 170.

[0079] In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker in a speaker system, a smart audio device, etc. In some examples, device 150 may not include microphone system 170. However, in some such implementations, device 150 may be configured to receive microphone data from one or more microphones in an audio environment via interface system 160. In some such implementations, cloud-based implementations of device 150 may be configured to receive microphone data or data corresponding to microphone data from one or more microphones in an audio environment via interface system 160.

[0080] According to some embodiments, device 150 may include Figure 1C The optional loudspeaker system 175 is shown. The optional loudspeaker system 175 may include one or more loudspeakers, which may also be referred to herein as “loudspeakers” or more generally as “audio reproduction transducers.” In some examples (e.g., cloud-based implementations), the device 150 may not include the loudspeaker system 175.

[0081] In some embodiments, device 150 may include an optional sensor system 180, such as Figure 1CAs shown in the diagram. Optional sensor system 180 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, optional sensor system 180 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of optional sensor system 180 may reside in a smart audio device, which may be a standalone audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 180 may reside in a television, mobile phone, or smart speaker. In some examples, device 150 may not include sensor system 180. However, in some such embodiments, device 150 may be configured to receive sensor data from one or more sensors in the audio environment via interface system 160.

[0082] In some embodiments, device 150 may include Figure 1C The optional display system 185 is shown. The optional display system 185 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, the optional display system 185 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 185 may include one or more displays of a smart audio device. In other examples, the optional display system 185 may include a television display, a laptop display, a mobile device display, or other types of displays. In some examples where the device 150 includes the display system 185, the sensor system 180 may include a touch sensor system and / or gesture sensor system proximate to one or more displays of the display system 185. According to some such embodiments, the control system 160 may be configured to control the display system 185 to present one or more graphical user interfaces (GUIs).

[0083] According to some of these examples, device 150 may be or may include a smart audio device. In some such implementations, device 150 may be or may include a wake word detector. For example, device 150 may be or may include a virtual assistant.

[0084] As described above, in some embodiments, one or more "gaps" (also referred to herein as "forced gaps" or "parameterized forced gaps") can be inserted into one or more frequency ranges of the audio playback signal of the content stream to produce a modified audio playback signal. The modified audio playback signal can then be reproduced or "played back" in an audio environment. In some such embodiments, it is possible to... N During each time interval N A gap is inserted into the audio playback signal. NWithin a frequency range. According to some such implementations, M Each audio device can program its gaps in time and frequency, allowing for accurate detection of the far field (for each device) in the gap frequencies and time intervals.

[0085] In some examples, a series of forced gaps are inserted into the playback signal, each forced gap located in a different frequency band (or a set of frequency bands) of the playback signal, so as to allow a pervasive listener to monitor the non-playback sound occurring "in" each forced gap in the sense that it occurs during the time interval in which the gap appears and in the frequency band(s) in which the gap is inserted. Figure 2A This is an example of modifying the spectrogram of an audio playback signal. In this example, based on a previous example, the modification of the audio playback signal is created by inserting gaps into the audio playback signal. More specifically, in order to generate Figure 2A The spectrum of the audio playback signal is used to perform the disclosed method on the audio playback signal to introduce forced gaps in its frequency band (e.g., Figure 2A The gaps G1, G2, and G3 shown in the diagram generate a modified audio playback signal. Figure 2A In the spectrum diagram shown, the position along the horizontal axis indicates time, and the position along the vertical axis indicates the frequency of the modified audio playback signal content at a given moment. The density of points in each small region (in this example, each such region is centered at a point with both vertical and horizontal coordinates) indicates the energy of the modified audio playback signal content at the corresponding frequency and time: denser regions indicate content with higher energy, while less dense regions indicate content with lower energy. Therefore, gap G1 occurs earlier than the time when gaps G2 or G3 occur (in other words, during the time interval between the occurrences of gaps G2 or G3), and gap G1 has been inserted into a frequency band higher than the frequency band into which gaps G2 or G3 were inserted.

[0086] According to some disclosed methods, a forced gap is introduced into the playback signal, which differs from simplex device operation where the device pauses the playback stream (e.g., to better hear the user and their environment). Introducing a forced gap into the playback signal according to some disclosed methods can be optimized to significantly reduce (or eliminate) the perceptibility of artifacts caused by the introduced gap during playback, preferably such that the forced gap has no or minimal perceptible impact on the user, but the microphone output signal in the playback environment indicates the forced gap (e.g., thus enabling a pervasive listening method). By using a forced gap introduced according to some disclosed methods, a pervasive listening system can monitor non-playback sounds (e.g., sounds indicating background activity and / or noise in the playback environment), even without the use of an acoustic echo canceller.

[0087] refer to Figure 2B and 2C We will now describe examples of parameterized forced gaps that can be inserted into the frequency band of an audio playback signal, and the criteria for selecting the parameters of such forced gaps. Figure 2B This is a graph showing an example of gaps in the frequency domain. Figure 2C This is a graph illustrating examples of gaps in the time domain. In these examples, the parameterized forced gap is achieved using a method with attenuation. G Attenuation of playback content, with attenuation G The distribution in time and frequency is similar to Figure 2B and 2C The distribution shown is illustrated here. This is achieved through attenuation. G Applied to by center frequency (exist Figure 2B (indication) and bandwidth B (also in) Figure 2B The playback signal within the frequency range (“band”) defined by the instruction in the text is used to force the gap, and the attenuation varies as a function of time at each frequency in the band (e.g., within each frequency interval of the band), having a similar effect to... Figure 2C The distribution shown can be controlled for attenuation. G The maximum value (as a function of the crossband frequency) increases from 0 dB (at the lowest frequency of the band) to the center frequency. Maximum attenuation (suppression depth) at that point Z (like Figure 2B As indicated in the text), and decrease (as the frequency increases above the center frequency) to 0dB (at the highest frequency of the band).

[0088] In this example, Figure 2B The curve indicates band attenuation G The distribution of frequency (i.e., frequency range) is applied to the frequency components of an audio signal to force the gaps in the audio content of the in-band signal. The audio signal can be a playback signal (e.g., the channels of a multi-channel playback signal), and the audio content can be playback content.

[0089] Based on this example, Figure 2C The graph shows the frequency band attenuation. G The distribution, as a function of time, is applied to the center frequency. The frequency components at that point force the audio content of the signal in the band. Figure 2B The gap indicated in the middle. For each other frequency component in the band, the band gain as a function of time can have the same... Figure 2C Similar distributions as shown, but Figure 2C Inhibition depth Z The suppression depth can be determined by interpolation. kZ Replacement, among which kIn this example, the factor is in the range of 0 to 1 (as a function of frequency), such that kZ have Figure 2B The distribution is shown in the figure. In some examples, for each frequency component, the suppression depth can also range from 0 dB to [the specified value]. kZ (For example, k =1, such as Figure 2C As shown, interpolated attenuation at the center frequency G For example, to reduce musical artifacts caused by introducing gaps. Figure 2C The image shows three regions (time intervals) t1, t2, and t3 for the latter interpolation.

[0090] Therefore, when targeting a specific frequency band (e.g., at the center frequency) The central belt, such as Figure 2B When a gap-forced operation occurs (as shown in the example), in this case, attenuation... G Each frequency component applied to the band (e.g., each interval within the band) follows the following... Figure 2C The trajectory shown. Starting at 0 dB, it descends to depth within t1 seconds. kZ dB, and remains there for t2 seconds, finally rising back to 0 dB within t3 seconds. In some implementations, the choice of the total time t1+t2+t3 can be considered to analyze the temporal resolution of the microphone feed using whatever frequency transformation is used, as well as a reasonable duration that is not too intrusive to the user. Table 1 below shows some examples of t1, t2, and t3 for a single-device implementation.

[0091] Some disclosed methods involve inserting forced gaps according to a predetermined, fixed band structure that covers the entire spectrum of the audio playback signal and includes B. count One band (of which B) count It is a number, for example, B count =49). To force a gap in any band, band attenuation is applied in the band in such examples. Specifically, for the first... j A band, and the attenuation Gj can be applied to the frequency region defined by that band.

[0092] Table 1 below shows the parameters t1, t2, t3, and depth for each band. Z Example values, and the number of bands B for a single-device implementation. count Examples.

[0093]

[0094] Table 1

[0095] When determining the number of bands and the width of each band, there is a trade-off between perceived impact and the usefulness of gaps: Narrower bands with gaps are better across all frequency bands of the full spectrum, for example, in response to changes in background noise or playback environment conditions, because they generally have less perceived impact, while wider bands with gaps are better for achieving noise estimation (and other pervasive listening methods) and reducing the time required to converge to a new noise estimate (or other values ​​monitored by pervasive listening) (“convergence” time). If only a limited number of gaps can be forced at a time, then forcing gaps sequentially in a large number of small bands takes longer than forcing gaps sequentially in a small number of large bands, resulting in a relatively longer convergence time. Larger bands (with gaps) provide a large amount of information about background noise (or other values ​​monitored by pervasive listening) but generally have a larger perceived impact.

[0096] In the inventors' earlier work, gaps were created in a single-device context where the echo effect was primarily (or entirely) near-field. Near-field echoes are largely influenced by the direct path of audio from the speaker to the microphone. This property holds true for almost all compact duplex audio devices, such as smart audio devices, except for those with large enclosures and significant acoustic decoupling. By introducing short, perceptibly masked gaps in playback, such as those shown in Table 1, the audio device can gain a glimpse into the acoustic space in which it is deployed through its own echo.

[0097] However, when other audio devices are also playing content in the same audio environment, the inventors found that the gaps between individual audio devices become less useful due to far-field echo corruption. Far-field echo corruption often degrades the performance of local echo cancellation, thus significantly worsening overall system performance. Far-field echo corruption is difficult to remove for various reasons. One reason is that obtaining a reference signal may require increased network bandwidth and increase the complexity of additional delay estimation. Moreover, estimating the far-field impulse response becomes more difficult as noise conditions increase and response times become longer (more reverberation and time spread). Furthermore, far-field echo corruption is often associated with near-field echoes and other far-field echo sources, further challenging far-field impulse response estimation.

[0098] The inventors have also discovered that if multiple audio devices in an audio environment are spaced out in time and frequency, a clearer perception of the far field (relative to each audio device) can be achieved when multiple audio devices reproduce modified audio playback signals. Furthermore, the inventors have found that if a target audio device plays an unmodified audio playback signal while multiple audio devices reproduce modified audio playback signals, the relative audibility and position of the target device can be estimated from the perspective of each of the multiple audio devices, even while media content is playing.

[0099] Moreover, perhaps contrary to intuition, the inventors have discovered that breaking with previous guidelines for single-device implementations (e.g., keeping the gap open for a longer period than indicated in Table 1) makes the implementation suitable for multiple devices to perform collaborative measurements via orchestrated gaps.

[0100] For example, in some orchestration gap implementations, t2 can be longer than indicated in Table 1 to accommodate various acoustic path lengths (acoustic delays) between multiple distributed devices in an audio environment. These acoustic path lengths can be on the order of meters (as opposed to the fixed microphone-speaker acoustic path length on a single device, which can be at most tens of centimeters apart). In some examples, the default t2 value can be, for example, 25 milliseconds larger than the 80 milliseconds value indicated in Table 1 to allow for separations of up to 8 meters between orchestrated audio devices. In some orchestration gap implementations, the default t2 value can be longer than the 80 milliseconds value indicated in Table 1 for another reason: in orchestration gap implementations, t2 is preferably longer to accommodate timing misalignments of the orchestrated audio devices, ensuring that a sufficiently long time elapses during which all orchestrated audio devices reach the Z-decay value. In some examples, an additional 5 milliseconds can be added to the default value of t2 to accommodate timing misalignments. Thus, in some orchestration gap implementations, the default value of t2 can be 110 milliseconds, the minimum is 70 milliseconds, and the maximum is 150 milliseconds.

[0101] In some orchestration gap implementations, t1 and / or t3 may also differ from the values ​​indicated in Table 1. In some examples, t1 and / or t3 may be adjusted because listeners cannot perceive the different times when a device enters or exits its decay period, due to timing issues and differences in physical distance. Listeners' ability to perceive the different times when orchestrated audio devices enter or exit their decay period is often lower than in a single-device scenario, at least in part due to spatial masking (caused by multiple devices playing audio from different locations). Therefore, in some orchestration gap implementations, the minimum values ​​of t1 and t3 may be reduced, while the maximum values ​​of t1 and t3 may be increased, compared to the single-device example shown in Table 1. According to some such examples, the minimum values ​​of t1 and t3 may be reduced to 2, 3, or 4 milliseconds, and the maximum values ​​of t1 and t3 may be increased to 20, 25, or 30 milliseconds.

[0102] Example of measuring the spacing of the arrangement

[0103] Figure 2D illustrates an example of a modified audio playback signal, including orchestrated gaps, for use with multiple audio devices in an audio environment. In this implementation, multiple smart devices in the audio environment orchestrate gaps to estimate their relative audibility to each other. In this example, a measurement session corresponding to one gap is performed during the time interval, and the measurement session only includes... Figure 1BThe device in the main living space 100a. Based on this example, previous audibility data has shown that the smart audio device 109 located in room 101b has been classified as almost inaudible to other audio devices and has been placed in a separate area.

[0104] exist Figure 2D In the example shown, the spacing of the arrangement is achieved using attenuation. G k Attenuation of replay content, among which k This indicates the center frequency of the frequency band being measured. Figure 2D The elements shown are as follows: Curve 203 is used for Figure 1B The smart audio device 103, measured in dB. G k The image; Curve 204 is used for Figure 1B The smart audio device 104 in dB. G k The image; Curve 205 is used for Figure 1B The smart audio device has a resolution of 105 dB. G k The image; Curve 206 is used for Figure 1B The smart audio device has a resolution of 106 dB. G k The image; Curve 207 is used for Figure 1B The smart audio device 107 in dB. G k The image; Curve 208 is used for Figure 1B The smart audio device has a resolution of 108 dB. G k The image; and Curve 209 is used for Figure 1B The smart audio device 109 in dB. G k The image.

[0105] As used herein, the term "session" (also referred to herein as a "measurement session") refers to a period of time during which measurements of a frequency range are performed. During a measurement session, a set of frequencies with associated bandwidths and a set of participating audio devices can be specified.

[0106] Optionally, an audio device can be nominated as the "target" audio device for the measurement session. If the target audio device is involved in the measurement session, then, according to some examples, the target audio device will be allowed to ignore forced gaps and will play an unmodified audio playback signal during the measurement session. According to some such examples, other participating audio devices will listen to the target device's playback sound, including the target device's playback sound within the frequency range being measured.

[0107] As used in this article, the term "audibility" refers to the degree to which a device can hear the speaker output of another device. Some examples of audibility are provided below.

[0108] according to Figure 2D In the example shown, at time t1, the orchestration device initiates a measurement session with the smart audio device 103, which is the target audio device, and selects one or more center frequencies of the intervals to be measured, including frequencies... k In some examples, the orchestration device can be a smart audio device that acts as the leader (e.g., see reference below). Figure 4 (As described). In other examples, the orchestration device could be another orchestration device, such as a smart home hub. This measurement session runs from time t1 to time t2. Other participating smart audio devices, smart audio devices 104-108, will apply gaps to their outputs and reproduce the modified audio playback signal, while smart audio device 103 will play the unmodified audio playback signal.

[0109] A subset of intelligent audio devices (intelligent audio devices 104-108) that are reproducing an audio environment 100 that includes modified audio playback signals with staggered spacing are referred to as M An example of an audio device. Based on this example, the smart audio device 109 will also play the unmodified audio playback signal. Therefore, the smart audio device 109 is not... M One of the audio devices. However, because smart audio device 109 is inaudible to other smart audio devices in the audio environment, smart audio device 109 is not the target audio device in this example, although both smart audio device 109 and the target audio device (smart audio device 103 in this example) will play back the unmodified audio playback signal.

[0110] During a measurement session, it is desirable that the spacing of the arrangement has a low perceptual impact (e.g., negligible perceptual impact) on listeners in the audio environment. Therefore, in some examples, the spacing parameter can be chosen to minimize the perceptual impact. See below for reference. Figure 3B-3J Some examples are described.

[0111] During this time period (the measurement session from time t1 to time t2), smart audio devices 104-108 will receive reference audio intervals from the target audio device (smart audio device 103) for the time-frequency data of this measurement session. In this example, the reference audio intervals correspond to the playback signal used by smart audio device 103 as a local reference for echo cancellation. Smart audio device 103 can access these reference audio intervals for both audibility measurement and echo cancellation purposes.

[0112] Based on this example, at time t2, the first measurement session ends and the orchestration device initiates a new measurement session, this time selecting one or more excluding frequencies. k The center frequency of the interval. In Figure 2D In the example shown, during the time period t2 to t3, for the frequency k There are no applied gaps, so the graph shows unity gain for all devices. In some such examples, the orchestration device could allow a series of gaps to be inserted into each of multiple frequency ranges, for frequencies not included in the spectrum. k A series of measurement sessions of the center frequency of the interval. For example, for the second to the... N The purpose of subsequent measurement sessions is to allow the orchestration equipment to be used in the second to third sessions. N During the time interval, the second to the third N The gaps are inserted into the second to third audio playback signals. N Within the frequency range, the intelligent audio device 103 remains the target audio device.

[0113] In some such examples, the orchestration device can then select another target audio device, such as smart audio device 104. The orchestration device can instruct smart audio device 103 to play back a modified audio playback signal with orchestrated intervals. M One type of intelligent audio device. An orchestration device can instruct a new target audio device to reproduce the unmodified audio playback signal. According to some such examples, the orchestration device has already enabled... N After a measurement session, the orchestration device can select another target audio device. In some such examples, the orchestration device can continue to allow measurement sessions to occur until a measurement session has been performed for each participating audio device in the audio environment.

[0114] exist Figure 2DIn the example shown, different types of measurement sessions occur between times t3 and t4. According to this example, at time t3, in response to user input (e.g., a voice command to a smart audio device acting as an orchestration device), the orchestration device initiates a new session to fully calibrate the amplifier settings of audio environment 100. Generally, users may be relatively more tolerant of orchestration gaps that have a relatively high perceived impact during a "setup" or "recalibration" measurement session, such as those occurring between times t3 and t4. Therefore, in this example, a large set of consecutive frequencies, including k, is selected for measurement. According to this example, smart audio device 106 is selected as the first target audio device during this measurement session. Thus, during the first phase of the measurement session from time t3 to t4, gaps will be applied to all smart audio devices except smart audio device 106.

[0115] gap bandwidth

[0116] Figure 3A This is an example of the filter response used to create the gap and a graph showing the filter response used to measure the frequency range of the microphone signal used during the measurement session. Based on this example, Figure 3A The elements are as follows: Element 301 represents the magnitude response of the filter used to create gaps in the output signal; Element 302 represents the magnitude response of the filter used to measure the frequency region corresponding to the gap caused by element 301; Elements 303 and 304 represent the -3dB point of 301, with frequencies f1 and f2; and Elements 305 and 306 represent the -3dB point of 302, with frequencies of f3 and f4.

[0117] The bandwidth (BW_gap) of the gap response 301 can be found by taking the difference between the -3dB points 303 and 304: BW_gap = f2 – f1 and BW_measure (measure the bandwidth of the response 302) = f4 – f3.

[0118] As an example, the measured mass can be represented as follows:

[0119] Since the bandwidth of the measurement response is typically fixed, the measurement quality can be adjusted by increasing the bandwidth of the gap filter response (e.g., widening the bandwidth). However, the bandwidth of the introduced gap is proportional to its perceptibility. Therefore, the bandwidth of the gap filter response should generally be determined based on the measurement quality and the perceptibility of the gap. Table 2 shows some examples of quality values:

[0120] Table 2

[0121] Although Table 2 indicates “minimum” and “maximum” values, these values ​​apply only to this example. Other implementations may involve mass values ​​lower than 1.5 and / or higher than 3.

[0122] Gap allocation strategy

[0123] The gap can be defined by the following: The underlying division of the spectrum includes a center frequency and a measurement bandwidth; These minimum measurement bandwidths are aggregated in a structure known as “banding”; Duration, attenuation depth, and one or more consecutive frequencies comprising an agreed-upon spectral division; and Other temporal behaviors, such as tilting the decay depth at the beginning and end of the gap.

[0124] According to some implementations, the gap can be selected based on a strategy aimed at measuring and observing as much audible spectrum as possible in the shortest possible time while satisfying applicable perceptibility constraints.

[0125] Figure 3B , 3C Figures 3D, 3E, 3F, 3G, 3H, 3I, and 3J are graphs illustrating examples of gap allocation strategies. In these examples, time is represented by distance along the horizontal axis and frequency by distance along the vertical axis. These graphs provide examples illustrating the patterns generated by various gap allocation strategies and the time they take to measure the complete audio spectrum. In these examples, the length of the gap measurement session for each orchestration is 10 seconds. As with other disclosed implementations, these graphs are provided by way of example only. Other implementations may include more, fewer, and / or different types, numbers, and / or sequences of elements. For example, in other implementations, the gap measurement session for each orchestration may be longer or shorter than 10 seconds. In these examples, Figure 3B-3J The unshaded area 310 (referred to herein as a "tile") in the time / frequency space represents the gap at the indicated time-frequency period (10 seconds). The moderately shaded area 315 represents a frequency tile that has been measured at least once. The lightly shaded area 320 has not yet been measured.

[0126] Suppose the task at hand requires the participating audio devices to be inserted into the interstitial space to "listen straight into the room" (e.g., assess noise, echo, etc. in the audio environment), then measuring session completion time would be as follows: Figure 3B-3JAs indicated in the document. If the task requires each audio device to become the target in turn and be listened to by other audio devices, then it needs to be multiplied by the number of audio devices involved in the process. For example, if each audio device becomes the target in turn, then... Figure 3B The figure shown in the middle represents three minutes and twenty seconds (3m20s) of the session completion time, which means that a system with 7 audio devices will be in 7... After 3m20s = 23m20s, it is fully mapped. When cycling between frequencies / bands and forcing multiple gaps simultaneously, in these examples, the gaps will be spaced as far apart as possible in terms of frequency to improve efficiency when covering the spectrum.

[0127] Figure 3B and 3C This is a graph illustrating an example sequence of gaps arranged according to a gap allocation strategy. In these examples, the gap allocation strategy involves gapening once during each consecutive measurement session. N All frequency bands (each band includes at least one frequency range, and in most cases, multiple frequency ranges). In Figure 3B N=1 and Figure 3C In N=3, the latter means Figure 3C Examples involve inserting three gaps within the same time interval. In these examples, the banding structure used is a 20-band Mel-interval arrangement. According to some of these examples, the sequence can be restarted after measuring all 20 bands. While 3m20s is a reasonable time to reach a complete measurement, the gaps made in the critical audio region of 300Hz–8kHz are very wide, and a significant amount of time is spent on measurements outside this region. Due to the relatively wide gaps in the 300Hz–8kHz frequency range, this particular strategy will be very easily perceived by the user.

[0128] Figure 3D and 3E This is a graph illustrating an example of a sequence of gaps arranged according to another gap allocation strategy. In these examples, the gap allocation strategy involves modifying... Figure 3B and 3C The banding structure shown is mapped to an "optimized" frequency range of approximately 300 Hz to 8 kHz. The overall allocation strategy is otherwise consistent with... Figure 3B and 3C The strategy remains unchanged, but the sequence ends slightly earlier because the 20th frequency band is now ignored. The bandwidth of the forced gaps here will still be perceptible. However, the advantage is that measurements of the optimized frequency region are very fast, especially when gaps are forced into multiple frequency bands simultaneously.

[0129] Figure 3F , 3GFigures 3H and 3H are graphs illustrating an example sequence of gaps arranged according to another gap allocation strategy. In these examples, the gap allocation strategy involves a "forced interval gap" method, where gaps are forced into a single frequency interval rather than across the entire frequency band. Figure 3F , 3G The horizontal line in 3H defines the boundary. Figure 3D and 3E The banding structure shown. Changing the gap allocation strategy from 19 frequency bands to 170 frequency bands significantly increases the time required to measure the optimized spectrum. Figure 3F In the example shown, a single measurement session now takes more than 25 minutes to complete, where N=1.

[0130] Figure 3F , 3G The main advantage of the gap allocation strategy represented by 3H is a significant reduction in process perceptibility. Choosing N=3 (e.g.) Figure 3G (As shown) or N=5 will make Figure 3F The example shows a reduction of 1 / N in the measurement session time, such as... Figure 3F and 3G As shown in the diagram, and the perceptibility remains manageable.

[0131] but, Figure 3F , 3G The gap allocation strategy represented by 3H still has two major drawbacks. First, the logarithmic property of the banding structure is ignored: based on real-world human perception, the bandwidth of the gaps at higher frequencies is overly conservative. Another drawback is that sequentially stepping through frequencies will result in each band being fully measured before moving to the next band. By interpolating missing data and averaging over the banding process, the algorithm can still operate with a certain degree of confidence even if the bands are not fully measured.

[0132] Figure 3I and 3J This is a graph illustrating an example sequence of gaps arranged according to another gap allocation strategy. In these examples, the bandwidth of the gaps increases with frequency, but is less than... Figure 3I and 3J The rate of the underlying bandgap structure represented by the mid-level line is more conservative. As the frequency increases, the bandwidth of the gap decreases, reducing the overall measurement session time without negatively impacting the perceptibility of the inserted gap. The second improvement is for each forced gap... Figure 3I and 3J The indicated gap allocation strategy involves selecting frequency intervals within a continuous frequency band (which in... Figure 3I(This is even more evident in the examples). Based on these examples, by remembering / tracking previously measured intervals within each band, the next successive interval within that band is measured when the band is revisited. This process does not affect the time spent measuring the entire spectrum, but it rapidly reduces the time spent measuring at least a portion of each band at least once. Figure 3I and 3J The represented gap allocation strategy also has patterns and structures that are less discernible than the gap allocation strategies mentioned above, further reducing the perceptible impact.

[0133] Figure 4 , 5A Figures 5 and 5B are flowcharts illustrating an example of how multiple audio devices coordinate a measurement session according to some implementations. Similar to the blocks of other methods described herein, Figure 4-5B The boxes shown may not be executed in the indicated order. For example, in some implementations, Figure 4 The operation of box 401 can be performed before the operation of box 400. Furthermore, such methods may include more or fewer boxes than those shown and / or described.

[0134] Based on these examples, the smart audio device is the orchestration device (which may also be referred to as the "leader" in this document), and only one device can be the orchestration device at a time. In other examples, the orchestration device can be the smart home hub referred to in this document. The orchestration device can be one of the above-mentioned references. Figure 1C An example of the described device 150.

[0135] Figure 4 The diagram depicts the actions performed by all participating audio devices according to this example. In this example, box 400 relates to obtaining a list of all other participating audio devices. According to some such examples, box 400 may relate to obtaining indications of the acoustic zone, group, etc., of each participating audio device. The list of box 400 can be created, for example, by aggregating information from other audio devices via network packets: other audio devices may, for example, broadcast their intention to participate in the measurement session. As audio devices are added and / or removed from the audio environment, the list of box 400 can be updated. In some such examples, the list of box 400 can be updated according to various heuristics to only relate to the most important devices (e.g., currently in...). Figure 1B The list of audio equipment in the main living space 101a is kept up-to-date.

[0136] exist Figure 4In the example shown, link 404 instructs the list in box 400 to be passed to box 401 for a leader negotiation process. This negotiation process in box 401 can take different forms, depending on the specific implementation. In the simplest embodiment, the leader can be determined by the alphanumeric order of the lowest or highest device ID code (or other unique device identifier) ​​without requiring multiple rounds of communication between devices, assuming all devices can implement the same scheme. In more complex implementations, devices can negotiate with each other to determine which device is best suited to be the leader. For example, it is convenient to have the device aggregating orchestrated information also be the leader for the purpose of facilitating measurement sessions. The device with the longest uptime, the most powerful computing capabilities, and / or the device connected to mains power can be good candidates for leader. In general, arranging such consensus among multiple devices is a challenging problem, but there are many existing and satisfactory protocols and solutions to this problem (e.g., the Paxos protocol). It will be understood that many such protocols exist and will be appropriate.

[0137] Then all participating audio devices proceed to box 403, meaning link 406 is unconditionally linked in this example. See below for reference. Figure 5B Description box 403. If the device is the leader, then it will execute box 402. In this example, link 405 involves checking the leader. See below for reference. Figure 5A Describe the leadership process. The outputs from this leadership process, including but not limited to messages to other audio devices, are... Figure 4 The link indicates a 407 error.

[0138] Figure 5A An example of a process performed by an orchestration device or leader is shown. Box 501 relates to selecting the target device to be measured and selecting a gap allocation strategy, such as the start and end times of the gaps to be used during the measurement session, as well as the location and size of the gaps in terms of frequency. In some examples, box 501 may relate to selecting times t1, t2, and / or t3, as referenced above. Figure 2C Different applications can inspire different strategies for the above selection. For example, in some cases, the target device to be measured may be selected in part based on the measurement of "urgency," such as a preference for devices and frequency bands that have not been measured recently. In some cases, a particular target device may be more important for measurements based on a specific application or use case. For example, the location of the speakers for the "left" and "right" channels in a spatial presentation can generally be important for measurement.

[0139] Based on this example, after the orchestration device makes selection 501, Figure 5AThe process continues to block 502. In this example, block 502 involves sending the information determined in block 501 to other participating audio devices. In some examples, block 502 may involve sending the information to other participating audio devices via wireless communication, such as via a local Wi-Fi network, via Bluetooth, etc. In some examples, block 502 may involve sending details of the gap allocation strategy to other participating audio devices, such as measuring the start and end times of the gaps to be used during the session, as well as the position and size of the gaps in the frequency spectrum. In other examples, the other participating audio devices may have already stored information about each of a plurality of gap allocation strategies. In some such examples, block 502 may involve sending an indication of the stored gap allocation strategy to select, for example, gap allocation strategy 1, gap allocation strategy 2, etc. In some examples, block 502 may involve sending a "session start" indication, for example, as referenced below. Figure 5B As stated above.

[0140] Based on this example, after the orchestration device executes block 502, Figure 5A The process continues to box 503, where the orchestration device waits for the current measurement session to end. In this example, in box 503, the orchestration device waits for confirmation that all other participating audio devices have ended their sessions.

[0141] In this example, after the orchestration device has received confirmation from all other participating audio devices in box 503, Figure 5A The process continues to block 500, where information about the measurement session is provided to the orchestration device. This information can influence the selection and timing of future measurement sessions. In some embodiments, block 500 relates to receiving measurements obtained from all other participating audio devices during the measurement session. The type of measurement received can vary depending on the specific implementation. According to some examples, the received measurement may be or may include microphone signals. Alternatively or additionally, in some examples, the received measurement may be or may include audio data extracted from microphone signals. In some implementations, the orchestration device may perform (or cause to perform) one or more operations on the received measurement. For example, the orchestration device may estimate (or cause to estimate) the audibility or location of a target audio device based at least in part on the extracted audio data. Some implementations may involve estimating the far-field audio environment impulse response and / or audio environment noise based at least in part on the extracted audio data.

[0142] exist Figure 5AIn the example shown, after executing box 500, the process returns to box 501. In some such examples, the process will return to box 501 after a predetermined time period following the execution of box 500. In some cases, the process may return to box 501 in response to user input.

[0143] Figure 5B An example of a process performed by participating audio devices other than the orchestration device is illustrated. Here, block 510 relates to each of the other participating audio devices sending a transmission (e.g., a network packet) to the orchestration device, signaling each device's intention to participate in one or more measurement sessions. In some embodiments, block 510 may also relate to sending the results of one or more previous measurement sessions to the leader.

[0144] In this example, box 515 follows box 510. According to this example, box 515 relates to waiting for notification that a new measurement session will begin, for example, as indicated via a “session start” packet.

[0145] According to this example, block 520 relates to applying a gap allocation strategy based on information provided by the orchestration device, for example, along with the “session start” data packet awaiting in block 515. In this example, block 520 relates to applying the gap allocation strategy to generate a modified audio playback signal, which will be played back by participating audio devices (if any, in addition to the target audio device) during the measurement session. According to this example, block 520 relates to detecting the audio device playback sound via the audio device's microphone and generating the corresponding microphone during the measurement session. As indicated by link 522, in some cases, block 520 may be repeated until all measurement sessions indicated by the orchestration device have been completed (e.g., according to a “stop” instruction received from the orchestration device (e.g., a stop data packet), or after a predetermined duration). In some cases, block 520 may be repeated for each of multiple target audio devices.

[0146] Finally, box 525 relates to stopping the insertion of gaps applied during the measurement session. In this example, after box 525... Figure 5B The process returns to box 510. In some such examples, the process will return to box 510 after a predetermined time period following the execution of box 525. In some cases, the process may return to box 510 in response to user input.

[0147] In some implementations, the frequency range, duration, and ordering of target devices within a set sequence can be determined using a simple algorithm based solely on unique device IDs / names. For example, the target devices can be ordered according to some agreed-upon word / alphanumeric sequence, and the frequency and interval duration can be based on the current time shared by all devices. This simplified embodiment offers lower system complexity but may not be suitable for more dynamic system requirements.

[0148] Example measurements of microphone signals revealed through gaps

[0149] The subband signals measured during the orchestrated interval measurement sessions correspond to the noise in the room, and if a target device is specified, then a direct stimulus from the target device is added. In this section, we show examples of acoustic properties and related information determined from these subband signals for further use in mapping, calibration, noise suppression, and / or echo attenuation applications.

[0150] Distance measurement

[0151] Based on some examples, sub-band signals measured during an intermittent measurement session can be used to estimate approximate distances between audio devices, for example, based on estimated direct reverberation ratios. For instance, if the target audio device can advertise an output sound pressure level (SPL), and if the distance from the speaker to the microphone of the measuring audio device is known, then it can be based on 1 / r... 2 The law estimates the approximate distance.

[0152] DoA

[0153] In some examples, sub-band signals measured during the intermittent measurement session can be used to estimate the direction of arrival (DoA) and / or time of arrival (ToA) of sounds emitted by one or more people (e.g., speaking) and / or one or more audio devices in the audio environment. In some such examples, the acoustic zone corresponding to the current location of one or more people and / or one or more audio devices can be estimated. See below for reference. Figure 8A Examples are described below.

[0154] Background noise

[0155] According to some examples, background noise can be estimated based on sub-band signals measured during the intervals of an orchestration measurement session, even at the time when a loudspeaker in an audio environment is reproducing music or other audio data. According to some such examples, background noise can be estimated by running a minimum follower (a filter that extracts the minimum value of the signal during a time window) on the energy found in each data frame, for example, according to the following expression:

[0156] In the above expression,K This indicates the total number of frequency ranges or bands in the application, and k This indicates the frequency range or band considered in the current measurement round. After a sufficient number of measurement rounds, BackGroundNoise It will include an estimate of the complete spectrum.

[0157] Reference sharing

[0158] If all participating audio devices are listening and one audio device is playing during the measurement session, then all audio devices will receive a relatively clean recording of the playback content during the measurement session. As discussed in the preceding headings, many acoustic properties can be derived from such a microphone signal. Another class of acoustic properties can be derived if the “reference” signal used for this playback signal is processed together with the microphone signal. The reference signal (also referred to herein as the “reference interval,” indicating the frequency range corresponding to the reference signal) can be, for example, a copy of the audio information played by the target device during the measurement session. In some examples, the reference signal can be used by the target device for echo suppression.

[0159] In some examples, the measurement session may be based on a narrow frequency range. Therefore, a bandwidth smaller than the full bandwidth of the reference information may be required to produce such acoustic properties. If a bandwidth smaller than the full bandwidth of the reference information is required, this makes it easier to provide such reference data over a network connection. For example, a typical frequency range with a bandwidth of 50 Hz corresponds to the original signal: 100% (50 / 24000) 2 ~= 0.5% (Assuming a sampling rate of 48kHz and the use of a complex frequency range representation). Furthermore, the same reference information can be presented to all other participating audio devices, thereby further utilizing associated network efficiency when broadcasting messages.

[0160] Nonlinear

[0161] Based on some examples of gap measurement sessions involving orchestration for one or more target audio devices, the presence of nonlinearity in the played-out audio data can be detected. Some such examples may involve obtaining audibility estimates at a range of playback levels and determining whether the audibility estimates are linear.

[0162] Figure 6 An example is shown where two orchestrated audio devices participate in a measurement session and share reference data. Similar to other figures provided in this article, Figure 6 The types and quantities of elements shown are provided as examples only. Other implementations may include more, fewer, and / or different types and quantities of elements.

[0163] For audio device 601a, Figure 6 The figure number in the diagram has an "a" suffix, and for audio device 601b, it has a "b" suffix. Figure 6 The elements include: 600: A system of two audio devices participating in a measurement session; 601a: Audio devices participating in a measurement session, it is Figure 1C An instance of device 150 and its implementation Figure 1C An example of the control system 160 shown; 601b: Another audio device participating in the measurement session, it is another instance of device 150, which implements Figure 1C The example shown is of a control system 160, and in this example, it is the target device; 602: A media playback engine that produces audio content (such as music, movie scores, or podcasts); 603: Gap Insertion Module; 604: Network connectivity module, used to send and receive network data packets via Wi-Fi, Bluetooth or other wireless protocols; 605: Acoustic property calculation block, configured to receive microphone signals and reference signals, and generate any acoustic properties described in this disclosure; 606a: One or more loudspeakers attached to audio device 601a; 606b: One or more loudspeakers attached to audio device 601b; 607a: The playback sound generated by the loudspeaker 606a corresponds to the (modified) audio content of the insertion gap; 607b: The playback sound produced by loudspeaker 606b corresponds to the unmodified (gapless) audio content, since audio device 601b is the target device in this example; 608a: One or more microphones attached to audio device 601a, detecting 607a and 607b; 608b: One or more microphones attached to audio device 601b, detecting 607a and 607b; 609: A network data packet transmitted from audio device 601b to audio device 601a contains at least a reference audio related to the current measurement session; 610: Media signal generated by media playback engine 602; 611: Add an additional modified (one or more gaps) media signal 610; 612: Reference signal extracted from network packet 609, equivalent to 614; 613: The microphone signal corresponding to the measurement area used in the current measurement session; and 614: Reference signal.

[0164] exist Figure 6 In this measurement session, the measurement session is active. Audio device 601b acts as the target device and is permitted to play media content in one or more measurement regions (one or more gaps in the media content inserted by audio device 601a) during the measurement session. Audio device 601a also participates in the measurement session. In this example, audio device 601a has been instructed by the control system of the orchestrated device to insert one or more appropriate gaps (using gap insertion module 603) into the outgoing media signal 610a.

[0165] During playback, in this example, audio device 601b extracts a reference signal 614b from playback media 610b, which corresponds to the same frequency measurement area relevant to the current measurement session. Reference signal 614b can, for example, be inserted into a network packet and sent (broadcast) as 609 over the local network to all other participating audio devices. Reference signal 614b can be streamed gradually while the measurement session is active, or alternatively, it can be sent as a single large transmission at the end of the measurement session. Other participating audio devices receive this reference signal 614b and also extract the corresponding microphone signal 613 (608) using their microphones. The target device (audio device 601b) also records the microphone signal and receives the reference signal, but skips the network transmission of 609 because the information resides on the same device.

[0166] exist Figure 6 In the example shown, signals 613 and 612 / 614 are presented to acoustic properties block 605, which is configured to use both signals simultaneously to calculate acoustic properties. It should be noted that the timing and synchronization aspects may vary depending on the implementation details of the specific embodiment, and the data may be appropriately aligned for further analysis using network packet timestamps and cross-correlation of the reference signal and microphone signal.

[0167] audibility and impulse response

[0168] According to some examples (e.g., in such as Figure 6 In the embodiment shown, during the measurement session, the reference signal r and microphone signal m All can be recorded and P The audio frames are tightly time-aligned over their periods. We can represent this as:

[0169] In the above expression, Indicates dimension (size)n complex space, r(t) and m(t) Indicates length n A complex vector, and n This represents the number of complex frequency intervals used for a given measurement session. Therefore, m(t) This represents the sub-band microphone signal. We can also represent:

[0170] In the above expression, Let represent the set of all integers, and t Indicates 1- P Any integer within the range, including 1, P .

[0171] This formula can solve the classic channel identification problem, attempting to estimate the channel from... r Predicted signal m linear transfer function H Existing solutions to this problem include adaptive finite impulse response (FIR) filters, offline (non-causal) Wiener filters, and many other statistical signal processing methods. Transfer function H The value of can be called audibility This is a useful acoustic property that can be used in some applications to rank devices based on their degree of "mutual audibility." According to some examples, the transfer function can be determined within the range of the audio device's playback level. H The value of the quantity is used to determine whether the played-back audio data indicates that the audio device is non-linear, for example, as described above.

[0172] Figure 7 An example of an audibility graph corresponding to an audio device in an audio environment is shown. In this case, Figure 7 The results of experiments are presented, depicting multiple measurement sessions conducted on a group of seven audio devices positioned at different locations around a typical open-plan living environment. Figure 7 The horizontal axis represents frequency (Hz), and the vertical axis represents dB. H The overall level, also referred to in this disclosure as "audibility". When a specific audio device named "DOLBY-OBSIDIAN / Kitchen" is the target audio device... Figure 7All acoustic measurements shown in the figure (as an aggregate) correspond to the arranged measurement sessions. The audibility of each audio device is shown as a bold dashed line, indicating the audio device's audibility as a function of frequency, and dashed lines with the same pattern but not shown in bold indicate the average audio device audibility level. From this figure, the differences in overall audibility or level between the "kitchen" audio device and various other audio devices can be seen. Furthermore, in Figure 7 The varying audibility at different frequencies reveals a level of detail achievable in the acoustic property measurements of this example. Lines 701a and 701b, representing "self-audibility," measure the echo level of the kitchen audio equipment itself; these are appropriately the loudest. The audio equipment "Kitchen 2," closest to the "kitchen," is only about 2 dB quieter on average and occasionally louder than the "kitchen" audio equipment at some frequencies. Audio equipment located in a distant room was measured with very low audibility, averaging 45 dB lower than its own audibility. The remaining audio equipment, located at different locations within the same room, recorded audibility measurements at some intermediate point.

[0173] An orchestration system comprising multiple smart audio devices can be configured to determine when speech from a user is detected. For example, when playing media content, speech can be detected in a frequency band associated with orchestration intervals, even if echo cancellation is not used or is insufficient.

[0174] Figure 8A This shows another example of an audio environment. Figure 8A This is a schematic diagram of an audio environment (in this example, a living space) including a system comprising a collection of smart audio devices (device 1.1) for audio interaction, speakers (1.3) for audio output, microphones 1.5, and controllable lights (1.2). In some cases, one or more of the microphones 1.5 may be part of or associated with one of the devices 1.1, 1.2, or 1.3. Alternatively or additionally, one or more of the microphones 1.5 may be attached to another part of the environment, such as a wall, ceiling, furniture, appliance, or another device in the environment. In this example, each smart audio device 1.1 includes (and / or is configured to communicate with) at least one microphone 1.5. Figure 8A The system can be configured to implement one or more embodiments of this disclosure. Various methods can be used to... Figure 8A The microphone 1.5 together acquires information and provides it to a device (e.g., a classifier) ​​configured to provide a location estimate of the user speaking.

[0175] In living spaces (e.g., Figure 8AWithin a living space, there exists a set of natural activity zones where people can perform tasks or activities, or cross thresholds. In some examples, these zones may be referred to herein as user zones, and can be defined by the user without specifying the coordinates or other markings of their geometric location. Figure 8A In the example shown, the user area may include: 1. Kitchen sink and food preparation area (in the upper left area of ​​the living space); 2. Refrigerator door (on the right side of the sink and food preparation area); 3. Dining area (located in the lower left area of ​​the living space); 4. Open areas of the living space (to the right of the sink and food preparation area and the dining area); 5. TV sofa (on the right side of the open area); 6. The television itself; 7. Table; and 8. Doorway or entrance passage (in the upper right area of ​​the living space).

[0176] According to some embodiments, a system that estimates where a user's voice (e.g., speech or noise) originates or is located can have a certain level of confidence (or multiple assumptions) in the estimate. For example, if a user happens to be near a boundary between zones of the system environment, then an uncertain estimate of the user's location could include a certain level of confidence in the user's location in each zone.

[0177] Figure 8B This shows another example of an audio environment. Figure 8B In this context, environment 809 (acoustic space) includes a user (801) speaking directly 802, and an example of a system including a set of intelligent audio devices (803 and 805), speakers for audio output, and microphones. This system can be configured according to embodiments of this disclosure. The speech uttered by user 801 (sometimes referred to herein as the speaker) can be recognized by one or more elements of the system within staggered time-frequency intervals.

[0178] More specifically, Figure 8B The system's elements include: 802: Direct local voice (generated by user 801); 803: Voice assistant device (coupled to one or more loudspeakers). Device 803 is closer to user 801 than device 805, therefore device 803 is sometimes referred to as the "near" device, while device 805 is referred to as the "far" device; 804: Multiple microphones in (or coupled to) near device 803; 805: Voice assistant device (coupled to one or more loudspeakers); 806: Multiple microphones in (or coupled to) remote device 805; 807: Household appliances (e.g., light bulbs); and 808: Multiple microphones in (or coupled to) home appliance 807. In some examples, each microphone 808 may be configured to communicate with a device configured to implement a classifier (in some cases, this may be at least one of device 803 or 805).

[0179] Figure 8B The system may also include at least one classifier. For example, device 803 (or device 805) may include a classifier. Alternatively or additionally, the classifier may be implemented by another device that can be configured to communicate with device 803 and / or 805. In some examples, the classifier may be implemented by another local device (e.g., a device within environment 809), while in other examples, the classifier may be implemented by a remote device located outside environment 809 (e.g., a server).

[0180] In some implementations, the control system (e.g., Figure 1C The control system 160 can be configured to implement a classifier, such as those disclosed herein. Alternatively or additionally, the control system 160 can be configured to determine an estimate of the user's current user region based at least in part on the output from the classifier.

[0181] Figure 8C An overview can be provided by, for example Figure 1C The flowchart illustrates an example of a method performed by the apparatus shown. As with other methods described herein, the blocks of method 830 are not necessarily executed in the indicated order. Furthermore, such methods may include more or fewer blocks than those shown and / or described. In this embodiment, method 830 involves estimating the user's position in the environment.

[0182] In this example, box 835 relates to receiving an output signal from each of a plurality of microphones in the environment. In this case, each of the plurality of microphones resides at a microphone location in the environment. According to this example, the output signal corresponds to the user's current utterance measured during the interludes in the playback content. For example, box 835 could relate to a control system (such as...) Figure 1C The control system 160) is accessed via an interface system (such as... Figure 1C The interface system 155 receives output signals from each of the multiple microphones in the environment.

[0183] In some examples, at least some of the microphones in the environment can provide an output signal asynchronously relative to the output signals provided by one or more other microphones. For example, a first microphone among a plurality of microphones may sample audio data according to a first sample clock, while a second microphone among a plurality of microphones may sample audio data according to a second sample clock. In some cases, at least one microphone in the environment may be included in or configured to communicate with a smart audio device.

[0184] According to this example, block 840 relates to determining multiple current acoustic features from the output signal of each microphone. In this example, "current acoustic features" are acoustic features derived from the "current utterance" in block 835. In some implementations, block 840 may relate to receiving multiple current acoustic features from one or more other devices. For example, block 840 may relate to receiving at least some of the multiple current acoustic features from one or more speech detectors implemented by one or more other devices. Alternatively or additionally, in some implementations, block 840 may relate to determining multiple current acoustic features from the output signal.

[0185] Whether acoustic features are determined by a single device or multiple devices, they can be determined asynchronously. If acoustic features are determined by multiple devices, they will generally be determined asynchronously unless the devices are configured to coordinate the process of determining the acoustic features. If acoustic features are determined by a single device, in some implementations, the acoustic features can still be determined asynchronously because the single device can receive the output signal from each microphone at different times. In some examples, acoustic features can be determined asynchronously because at least some of the microphones in the environment can provide output signals asynchronously relative to the output signals provided by one or more other microphones.

[0186] In some examples, acoustic features may include a speech confidence measure corresponding to speech measured during the intervals in the arrangement of the output playback signal.

[0187] Alternatively or additionally, acoustic features may include one or more of the following: Band power within a frequency band weighted for human speech. For example, acoustic features can be based solely on a specific frequency band (e.g., 400Hz–1.5kHz). In this example, higher and lower frequencies can be ignored.

[0188] The confidence level of the speech activity detector for each band or interval relative to the frequency band or interval arranged in the playback content.

[0189] Acoustic characteristics can be based at least in part on long-term noise estimates in order to ignore microphones with poor signal-to-noise ratios.

[0190] Kurtosis is a measure of the peak value of a speech. Kurtosis can also be an indicator of the length of a long reverberation tail.

[0191] According to this example, box 845 relates to applying a classifier to multiple current acoustic features. In some such examples, applying a classifier may involve applying a model trained on previously determined acoustic features derived from multiple previous utterances made by the user in multiple user zones within the environment. Various examples are provided in this paper.

[0192] In some examples, the user area may include a sink area, food preparation area, refrigerator area, dining area, sofa area, television area, bedroom area, and / or doorway area. According to some examples, one or more of the user areas may be pre-defined user areas. In some such examples, one or more pre-defined user areas may have been selectable by the user during the training process.

[0193] In some implementations, the applied classifier may involve applying a Gaussian mixture model trained on previous utterances. According to some such implementations, the applied classifier may involve applying a Gaussian mixture model trained on one or more of normalized speech confidence, normalized average reception level, or maximum reception level of previous utterances. However, in alternative implementations, the applied classifier may be based on a different model, such as one of the other models disclosed herein. In some cases, the model may be trained using training data labeled with user regions. However, in some examples, the applied classifier involves applying a model trained using unlabeled training data with unlabeled user regions.

[0194] In some examples, the preceding utterance may already be or may include the utterance of the speech. According to some such examples, the preceding utterance and the current utterance may be utterances of the same speech.

[0195] In this example, box 850 relates to determining an estimate of the user's current location based at least in part on the output from the classifier. In some such examples, the estimate can be determined without reference to the geometric positions of multiple microphones. For example, the estimate can be determined without reference to the coordinates of the individual microphones. In some examples, the estimate can be determined without estimating the user's geometric position. However, in alternative implementations, position estimation may involve estimating the geometric positions of one or more people and / or one or more audio devices in an audio environment, e.g., with reference to a coordinate system.

[0196] Some embodiments of method 830 may involve selecting at least one speaker based on an estimated user area. Some such embodiments may involve controlling at least one selected speaker to provide sound to the estimated user area. Alternatively or additionally, some embodiments of method 830 may involve selecting at least one microphone based on the estimated user area. Some such embodiments may involve providing a signal output by at least one selected microphone to a smart audio device.

[0197] Figure 9 A block diagram of an example system for interpolation in orchestration is presented. Figure 9 The system includes audio device 901a, which is Figure 1C An example of device 150, and includes a control system 160a configured to implement a noise estimation subsystem (noise estimator) 64, a noise compensation gain application subsystem (noise compensation subsystem) 62, and a forced gap application subsystem (forced gap applicator) 70. In this example, audio devices 901b-901n are also present in the playback environment E. In this embodiment, each of the audio devices 901b-901n is Figure 1C Examples of the apparatus 150, and each includes a control system configured to implement an instance of the noise estimation subsystem 64, the noise compensation subsystem 62, and the forced gap application subsystem 70.

[0198] Based on this example, Figure 9 The system also includes orchestration equipment 905, which is also Figure 1C Examples of the device 150. In some examples, the orchestration device 905 may be an audio device for a playback environment, such as a smart audio device. In some such examples, the orchestration device 905 may be implemented via one of the audio devices 901a-901n. In other examples, the orchestration device 905 may be another type of device, such as a device referred to herein as a smart home hub. According to this example, the orchestration device 905 includes a control system configured to receive noise estimates 910a-910n from the audio devices 901a-901n and provide urgency signals 915a-915n to the audio devices 901a-901n to control each corresponding instance of the forced gap applicator 70. In this embodiment, each instance of the forced gap applicator 70 is configured to determine, based on the urgency signals 915a-915n, whether to insert a gap, and if so, what type of gap to insert.

[0199] According to this example, audio devices 901a-901n are also configured to provide current gap data 920a-920n to orchestration equipment 905, indicating what gap (if any) each of the audio devices 901a-901n is implementing. In some examples, the current gap data 920a-920n may indicate a series of gaps being applied by the audio devices and their corresponding times (e.g., the start time and time interval of each or all gaps). In some implementations, the control system of orchestration equipment 905 may be configured to maintain data structures indicating, for example, recent gap data, which audio devices have received the most recent urgency signal, etc. Figure 9 In the system, each instance of the forced gap application subsystem 70 operates in response to urgency signals 915a-915n, causing the orchestration device 905 to control the forced gap insertion based on the gap requirement in the playback signal.

[0200] Based on some examples, urgency signals 915a-915n can indicate a set of urgency values ​​[U0, U1, ... U...]. N ], where N is a predetermined number of frequency bands (the entire frequency range of the playback signal) in which the subsystem 70 can insert forced gaps (e.g., one forced gap in each band), and U i Subsystem 70 can insert the first "" of the forced gap i The urgency value of each band. The urgency value of each set of urgency values ​​(corresponding to time) can be generated according to any disclosed embodiment for determining urgency, and can indicate the urgency of inserting a forced gap in N bands (by subsystem 70) at that time.

[0201] In some implementations, urgency signals 915a-915n can indicate a fixed (time-invariant) set of urgency values ​​[U0, U1, ... U...]. N The urgency signal 915a-915n is determined by a probability distribution defining the gap insertion probability for each of the N frequency bands. According to some examples, the probability distribution is implemented using a pseudo-random mechanism, so the result (the response of each instance of subsystem 70) is deterministic (the same) across all receiver audio devices 901a-901n. Therefore, in response to this fixed set of urgency values, subsystem 70 can be configured to insert fewer forced gaps (on average) in those bands with lower urgency values ​​(i.e., lower probability values ​​determined by the pseudo-random probability distribution) and more forced gaps (on average) in those bands with higher urgency values ​​(i.e., higher probability values). In some implementations, the urgency signal 915a-915n can indicate a set of urgency values ​​[U0, U1, ... U2]. NFor example, a set of different urgency values ​​for each different time in the sequence. Each such set of different urgency values ​​can be determined by different pseudo-random probability distributions for each time in the different time.

[0202] We will now describe a method for determining an urgency value or a signal (U) indicating an urgency value (which may be implemented in any of the many different embodiments of the disclosed pervasive listening method).

[0203] The urgency value of a frequency band indicates the need for a forced gap within the band. We provide a method for determining the urgency value U. k Three strategies, among which U k Indicates that in the belt k The urgency of forced intermittent insertion, and U This indicates that it contains information for B. count A vector of urgency values ​​for all bands in a set of frequency bands:

[0204] The first strategy (sometimes referred to as Method 1 in this paper) determines a fixed urgency value. This method is the simplest, simply allowing a urgency vector. U This becomes a predetermined, fixed quantity. When used in conjunction with a fixed perceptual freedom measure, this can be used to implement systems that randomly insert forced gaps over time. Some of these methods do not require a time-related urgency value provided by the pervasive listening application. Therefore:

[0205] in X =B count And each value u k (For from) k =1 to k = B count Within range k ) indicates used for " k "A predetermined, fixed urgency value. All u..." k Setting it to 1.0 will express the same level of urgency across all frequency bands.

[0206] The second strategy (sometimes referred to herein as Method 2) determines a urgency value that depends on the time elapsed since the previous gap occurred. In some implementations, the urgency gradually increases over time and returns to a low value once a forced or existing gap causes an update to the pervasive listening results (e.g., an update to the background noise estimate).

[0207] Therefore, each frequency band (band) k The urgency value U in ) k Can be used with self-brought kThe duration (e.g., number of seconds) from the start of the gap is perceived by the listener (in a distributed manner). In some examples, the urgency value U in each frequency band corresponds to this. k The following can be determined:

[0208] in t g Indicates seeing the belt k The time of the last gap, and U max This indicates that the urgency is limited to the maximum tuning parameters. It should be noted that... t g Updates can be made based on gaps initially present in the playback content. For example, in noise compensation, the current noise level in the playback environment can determine what is considered a gap in the output playback signal. That is, the playback signal must be quieter when the environment is quieter than when the environment is noisier for gaps to appear. Similarly, when implementing pervasive listening methods that rely on the presence or absence of user speech in the playback environment, the urgency of the frequency bands typically occupied by human speech is often more important.

[0209] The third strategy (sometimes referred to as Method 3 in this paper) determines the event-based urgency value. In this context, "event-based" means dependent on an event or activity (or a need for information) outside the playback environment, or detected or inferred to have occurred within the playback environment. The urgency determined by the pervasive listening subsystem can change abruptly with the onset of new user behavior or a change in the playback environment conditions. For example, such a change might make one or more devices configured for pervasive listening urgently need to observe background activity to make a decision, quickly adjust the playback experience to adapt to the new situation, or implement a change in the time between the general urgency or desired density and the gaps in each band. Table 3 below provides several examples of contexts and scenarios and the corresponding changes in event-based urgency:

[0210] Table 3

[0211] The fourth strategy (sometimes referred to as Method 4 in this paper) uses a combination of two or more of Methods 1, 2, and 3 to determine the urgency value. For example, each of Methods 1, 2, and 3 can be combined into a joint strategy, represented by a general formula of the following type:

[0212] in u k This represents a fixed, unitless weighting factor that controls the relative importance of each frequency band. V kThis represents a scalar value modulated in response to a context requiring rapid change or a change in user behavior. t g and U max The definition is as above. In some examples, the value... V k It is expected to remain at a value of 1.0 under normal operating conditions.

[0213] In some examples within a multi-device context, forced gap applicators for smart audio devices in an audio environment can collaborate in an orchestrated manner to achieve an accurate estimation of ambient noise N. In some such implementations, determining where to introduce the forced gap in time and frequency can be accomplished by orchestration device 905, which is implemented by separate orchestration devices, such as a smart home hub as referred to elsewhere herein. In some alternative implementations, the determination of where to introduce the forced gap in time and frequency can be made by one of the smart audio devices acting as a leader (e.g., the smart audio device acting as orchestration device 905).

[0214] In some embodiments, the orchestration device 905 may include a control system configured to receive noise estimates 910a-910n and provide gap commands to the audio devices 901a-901n that may be at least partially based on the noise estimates 910a-910n. In some such examples, the orchestration device 905 may provide gap commands instead of urgency signals. According to some such embodiments, the forced gap applicator 70 does not need to determine whether to insert a gap, and if so, what type of gap, based on urgency signals, but can instead simply act according to the gap commands.

[0215] In some such implementations, the gap command may indicate the characteristics of one or more specific gaps to be inserted (e.g., frequency range or B). count (t1, t2, and / or t3) and the time(s) used to insert one or more specific gaps. For example, a gap command can indicate a series of gaps and corresponding time intervals, such as t1, t2, and / or t3. Figure 3B-3J One of those shown and described above. In some examples, the gap command may instruct a data structure from which the receiving audio device can access the characteristics of the gap sequence to be inserted and the corresponding time intervals. The data structure may, for example, have been previously provided to the receiving audio device. In some such examples, the orchestration device 905 may include a control system configured to perform urgency calculations to determine when and what type of gap command to send.

[0216] According to some examples, the urgency signal can be estimated at least in part by noise estimation elements 64 of one or more of the audio devices 901a-901n and can be transmitted to the orchestration device 905. In some examples, the decision to orchestrate a forced gap at a specific frequency region and time location can be determined at least in part by the aggregation of these urgency signals from one or more of the audio devices 901a-901n. For example, the disclosed algorithm that makes a selection based on urgency can be replaced by using a maximum urgency calculated using urgency signals across multiple audio devices, for example, urgency = maximum(urgency A, urgency B, urgency C, ...), where urgency A / B / C is understood as urgency signals from three separate example devices for implementing noise compensation.

[0217] Noise compensation system (e.g., Figure 9 A noise compensation system can operate with weak or nonexistent echo cancellation (e.g., when implemented as described in U.S. Provisional Patent Application No. 62 / 663,302, which is incorporated herein by reference), but will be affected by content-dependent response times, especially in the case of music, television, and movie content. The time it takes for a noise compensation system to respond to changes in the distribution of background noise in the playback environment can be very important to the user experience, sometimes even more important than the accuracy of the actual noise estimate. When the playback content provides little or no gaps to glimpse the background noise, the noise estimate may remain unchanged even as the noise conditions change. While interpolating and imputing missing values ​​in the noise estimate spectrum is often helpful, large areas of the noise estimate spectrum can still become locked and stale.

[0218] Figure 9Some embodiments of the system are operable to provide forced gaps (in the playback signal) that occur sufficiently frequently (e.g., in each frequency band of interest at the output of forced gap applicator 70) so that the background noise estimate (via noise estimator 64) can be updated sufficiently frequently in response to typical changes in the profile of the background noise N in the playback environment E. In some examples, subsystem 70 can be configured to introduce forced gaps in the compensated audio playback signal (having K channels, where K is a positive integer) output from noise compensation subsystem 62. Here, noise estimator 64 can be configured to search for gaps (including forced gaps inserted by subsystem 70) in each channel of the compensated audio playback signal and generate a noise estimate for the frequency band (and time interval) in which the gaps occur. In this example, noise estimator 64 of audio device 901a is configured to provide noise estimate 910a to noise compensation subsystem 62. According to some examples, the noise estimator 64 of the audio device 901a can also be configured to use the obtained information about the detected gaps to generate (and provide to the orchestration device 905) an estimated urgency signal whose urgency value tracks the urgency of inserting a forced gap in the frequency band of the compensated audio playback signal.

[0219] In this example, noise estimator 64 is configured to receive a microphone feed Mic (output of microphone M in playback environment E) and a reference to the compensated audio playback signal (input of speaker system S in playback environment E). According to this example, the noise estimate generated in subsystem 64 is provided to noise compensation subsystem 62, which applies compensation gain to the input playback signal 23 (from content source 22) to level each of its frequency bands to the desired playback level. In this example, the noise-compensated audio playback signal (output from subsystem 62) and a urgency measure for each band (indicated by a urgency signal output from orchestration device 905) are provided to forced gap applicator 70, which forces gaps in the compensated playback signal (preferably according to an optimization process). Speaker feeds(s), each indicating the content of a different channel of the noise-compensated playback signal (output from forced gap applicator 70), are provided to each speaker in speaker system S.

[0220] Although Figure 9 Some implementations of the system may perform echo cancellation as a component of their noise estimation, but Figure 9 Other implementations of the system do not perform echo cancellation. Therefore, Figure 9 The specific elements used to achieve echo cancellation are not shown in the text.

[0221] exist Figure 9The time-domain to frequency-domain (and / or frequency-domain to time-domain) transformation of the signal is not shown, but the application of noise compensation gain (in subsystem 62), the analysis of the content used for gap forcing (in orchestration device 905, noise estimator 64, and / or forced gap applicator 70), and the insertion of forced gaps (via forced gap applicator 70) can be performed in the same transform domain for convenience. The resulting output audio is resynthesized as PCM (time-domain) audio or further encoded for transmission before playback. According to some examples, each participating device uses the methods described elsewhere in this document to coordinate this gap forcing. In some such examples, the introduced gaps may be exactly the same. In some examples, the introduced gaps may be synchronized.

[0222] By using the forced gap applicator 70 present on each participating device, a gap can be inserted, which can increase the compensated playback signal (from...). Figure 9 The number of gaps in each channel of the system's noise compensation subsystem 62 output (relative to the number of gaps that would occur without the forced gap applicator 70) is calculated to significantly reduce the impact on noise compensation. Figure 9 The system implements any echo canceller requirement, and in some cases even eliminates the need for echo cancellation entirely.

[0223] In some disclosed embodiments, a simple post-processing circuitry, such as time-domain peak limiting or speaker protection, can be implemented between the forced gap applicator 70 and the speaker system S. However, post-processing capable of boosting and compressing the speaker feed may cancel or degrade the quality of the forced gap inserted by the forced gap applicator; therefore, these types of post-processing are preferably implemented at a point in the signal processing path prior to the forced gap applicator 70.

[0224] Figure 10 This is a flowchart outlining another example of the disclosed method. As with other methods described herein, the blocks of method 1000 are not necessarily executed in the indicated order. Furthermore, such methods may include more or fewer blocks than those shown and / or described. In this example, method 1000 is an audio processing method.

[0225] Method 1000 can be made by a device or system (such as...) Figure 1CThe method 1000 is performed by the apparatus 150 shown and described above. In some examples, blocks of method 1000 may be performed by one or more devices within an audio environment, such as an orchestration device like an audio system controller (e.g., referred to herein as a smart home hub) or by another component of the audio system (such as a smart speaker, television, television control module, laptop computer, mobile device (such as a cellular phone), etc.). In some implementations, the audio environment may include one or more rooms of a home environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative implementations, at least some blocks of method 1000 may be performed by a device implementing a cloud-based service (such as a server).

[0226] In this embodiment, block 1005 relates to a control system inserting a first gap into a first frequency range of a first audio playback signal of the content stream during a first time interval of the content stream to generate a first modified audio playback signal for a first audio device in an audio environment. In this example, the first gap corresponds to an attenuation of the first audio playback signal in the first frequency range. In this example, block 1010 relates to a control system causing the first audio device to play back the first modified audio playback signal to generate a first audio device playback sound.

[0227] In this example, block 1015 relates to a control system interpolating a first gap into a first frequency range of a second audio playback signal of the content stream during a first time interval of the content stream to generate a second modified audio playback signal for a second audio device in an audio environment. According to this example, block 1020 relates to a control system causing the second audio device to play back the second modified audio playback signal to generate playback sound from the second audio device.

[0228] According to this embodiment, block 1025 relates to a control system causing at least one microphone in an audio environment to detect at least a first audio device playback sound and a second audio device playback sound, and generating a microphone signal corresponding to the at least first audio device playback sound and the second audio device playback sound. In this example, block 1030 relates to a control system extracting audio data from the microphone signal in at least a first frequency range to generate extracted audio data. According to this embodiment, block 1035 relates to a control system estimating at least one of a far-field audio environment impulse response or audio environment noise, at least in part, based on the extracted audio data.

[0229] In some implementations, method 1000 may involve causing a target audio device to play back an unmodified audio playback signal of a content stream to generate a target audio device playback sound. Some such implementations may involve a control system estimating at least one of the target audio device's audibility or location based at least in part on extracted audio data. In some such examples, the unmodified audio playback signal does not include a first gap. In some cases, the unmodified audio playback signal does not include gaps inserted in any frequency range. In some such examples, a microphone signal also corresponds to the target audio device playback sound.

[0230] According to some implementations, generating the first modified audio playback signal may involve the control system causing the second to third... N The gap is in the second to third of the content stream. N The time interval was inserted into the second to third of the first audio playback signal. N Within the frequency range. In some such examples, generating a second modified audio playback signal may involve the control system making the second to the third... N The gap is in the second to third of the content stream. N The time interval was inserted into the second to the second audio playback signal during the second to third time intervals. N Within the frequency range. Based on some examples, at least the first gap (and in some cases all gaps) can be perceived as masked.

[0231] In some implementations, method 1000 may involve the control system causing a first gap to be inserted into the third to fourth intervals of the content stream during a first time interval. M In the first frequency range of the audio playback signal, to generate the third to the fourth frequencies for the audio environment. M The third to the fourth audio equipment M Modify the audio playback signal. Some examples of this could involve the control system making the third to fourth... M Audio device playback of the third to the fourth M Modify the corresponding instance of the audio playback signal to generate the third to the fourth. M The audio device plays back sound, wherein generating a microphone signal involves the control system causing at least one microphone in the audio environment to detect the third to fourth microphones. M The audio device plays back the sound. In some such examples, the first to the second... M Modifying the audio playback signal involves the control system causing the second to third... N The gap is in the second to third of the content stream. N The time interval was inserted into the first to the second. N The second to the third audio playback signal N Within the frequency range.

[0232] In some examples, at least a first frequency range may correspond to a frequency band. In some such examples, the frequency band may be one of several frequency bands equally spaced on a Mel scale. However, in some cases, at least a first frequency range may correspond to a frequency interval.

[0233] In some implementations, method 1000 may involve transmitting a reference interval from a first device to a second device. In some examples, the first device may be a target device. For example, the reference interval may correspond to the output of a target device within a first frequency range.

[0234] According to some examples, inserting the first gap may involve transmitting an instruction to insert the first gap. In some alternative embodiments, inserting the first gap may involve inserting the first gap.

[0235] In some implementations, causing a first audio device to play back a first modified audio playback signal may involve transmitting an instruction to the first audio device to play back the first modified audio playback signal. According to some examples, the first modified audio playback signal and the second modified audio playback signal may be at least partially related.

[0236] Some aspects of this disclosure include systems or devices configured (e.g., programmed) to perform one or more examples of the disclosed methods, and tangible computer-readable media (e.g., disks) storing code for implementing one or more examples of the disclosed methods or steps thereof. For example, some disclosed systems may be or include programmable general-purpose processors, digital signal processors, or microprocessors, programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or steps thereof. Such general-purpose processors may be or include computer systems including input devices, memory, and processing subsystems programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0237] Some embodiments may be implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed or otherwise configured) to perform desired processing on (one or more) audio signals, including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor that may include input devices and memory), programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the invention may be implemented as a general-purpose processor or DSP configured (e.g., programmed) to execute one or more examples of the disclosed methods, and the system may also include other elements (e.g., one or more speakers and / or one or more microphones). A general-purpose processor configured to execute one or more examples of the disclosed methods may be coupled to an input device (e.g., a mouse and / or keyboard), memory, and a display device.

[0238] Another aspect of this disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code for performing (e.g., executable code to perform) one or more examples of the disclosed methods or steps thereof.

[0239] While specific embodiments and applications of this disclosure have been described herein, it will be apparent to those skilled in the art that many variations can be made to the described embodiments and applications and claimed herein without departing from the scope of the disclosure. It should be understood that although certain forms of this disclosure have been shown and described, this disclosure is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. An audio processing method, comprising: The control system causes a first gap to be inserted into a first frequency range of a first audio playback signal of the content stream during a first time interval of the content stream to generate a first modified audio playback signal for a first audio device in an audio environment, the first gap including attenuation of the first audio playback signal in the first frequency range; The control system causes the first audio device to play back the first modified audio playback signal to generate the sound played back by the first audio device. The control system causes a first gap to be inserted into a first frequency range of the second audio playback signal of the content stream during a first time interval of the content stream, so as to generate a second modified audio playback signal for a second audio device in an audio environment; The control system causes the second audio device to play back the second modified audio playback signal to generate the playback sound of the second audio device; The control system causes at least one microphone in the audio environment to detect at least the sound played back by a first audio device and the sound played back by a second audio device, and generates a microphone signal corresponding to the sound played back by the first audio device and the sound played back by the second audio device. The control system extracts audio data from the microphone signal in at least a first frequency range to generate the extracted audio data; as well as The far-field audio environment impulse response is estimated by the control system based at least in part on the extracted audio data.

2. The audio processing method as described in claim 1, further comprising: To generate playback sound for the target audio device, the unmodified audio playback signal of the content stream is used to enable the target audio device to play back the sound. as well as The control system estimates at least one of the audibility of the target audio device or the location of the target audio device, at least in part, based on the extracted audio data, wherein: The unmodified audio playback signal does not include the first gap; and The microphone signal also corresponds to the sound played back by the target audio device.

3. The audio processing method of claim 2, wherein the unmodified audio playback signal does not include gaps inserted into any frequency range.

4. The audio processing method as described in claim 1, wherein: Generating the first modified audio playback signal involves the control system causing the second to third content streams to... N During the time interval, the second to the third N The gaps are inserted into the second to third intervals of the first audio playback signal. N In the frequency range; as well as Generating the second modified audio playback signal involves the control system causing the second to third content streams to... N During the time interval, the second to the third N The gap is inserted into the second to the third of the second audio playback signal. N Within the frequency range.

5. The audio processing method as described in claim 1, further comprising: The control system causes the first gap to be inserted into the third to fourth gaps of the content stream during the first time interval of the content stream. M In the first frequency range of the audio playback signal, to generate the third to the fourth frequencies for the audio environment. M The third to the fourth audio equipment M Modify the audio playback signal; as well as The control system enables the third to the fourth M Audio device playback of the third to the fourth M Modify the corresponding instance of the audio playback signal to generate the third to the fourth. M The audio device plays back sound, wherein generating a microphone signal involves the control system causing at least one microphone in the audio environment to detect a third to a fourth microphone. M Audio devices play back sound.

6. The audio processing method as described in claim 5, wherein the first to the second generation are... M Modifying the audio playback signal involves the control system causing changes in the second to third content stream. N During the time interval, the second to the third N The gap was inserted into the first to the second M The second to the third audio playback signal N Within the frequency range.

7. The audio processing method of claim 1, wherein at least the first gap is perceptually masked.

8. The audio processing method of claim 1, wherein at least a first frequency range corresponds to a frequency band.

9. The audio processing method of claim 8, wherein the frequency band is one of a plurality of frequency bands equally spaced on a Mel scale.

10. The audio processing method of claim 1, wherein at least the first frequency range corresponds to a frequency interval.

11. The audio processing method according to any one of claims 1-10, further comprising transmitting a reference interval from a first device to a second device, the reference interval corresponding to the output of a target device in a first frequency range.

12. The audio processing method according to any one of claims 1-10, wherein inserting the first gap includes inserting the first gap or transmitting an instruction to insert the first gap.

13. The audio processing method according to any one of claims 1-10, wherein causing the first audio device to play back the first modified audio playback signal includes transmitting an instruction to the first audio device to play back the first modified audio playback signal.

14. The audio processing method according to any one of claims 1-10, wherein the first modified audio playback signal and the second modified audio playback signal are at least partially correlated.

15. An audio processing apparatus comprising a processor and a memory, the memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to perform steps in the audio processing method of any one of claims 1-14.

16. An audio processing system comprising a processor and a memory, the memory storing instructions, wherein the instructions, when executed by the processor, cause the processor to perform steps in the audio processing method of any one of claims 1-14.

17. One or more computer-readable non-transitory storage media storing instructions that, when executed by a processor, cause the processor to perform the steps of the audio processing method as described in any one of claims 1-14.

Images