Distributed Audio Device Ducking

Abstract

The audio processing method may include receiving an output signal from each microphone of a plurality of microphones in an audio environment, the output signal corresponding to a current speech of a person. The method may include determining, in response to the output signal, one or more audio processing modifications to apply to audio data being rendered into loudspeaker feed signals for two or more audio devices in the audio environment based at least in part on the audio device location information and the echo management system information. The audio processing modifications may include reducing loudspeaker playback levels for one or more loudspeakers in the audio environment. The method may include applying one or more types of audio processing modifications. The audio processing modifications may have the effect of increasing a voice-to-echo ratio at one or more microphones.

Description

[Technical field]

[0001] [CROSS REFERENCE TO RELATED APPLICATIONS] This application claims priority to U.S. Provisional Patent Application No. 63 / 278,003, filed November 10, 2021, U.S. Provisional Patent Application No. 63 / 362,842, filed April 12, 2022, and European Patent Application No. 22167857.6, filed April 12, 2022, all of which are incorporated by reference in their entireties herein. [Technical field]

[0002] The present disclosure relates to systems and methods for orchestrating and implementing audio devices, such as smart audio devices, and for controlling the Sound-to-Echo Ratio (SER) in such audio devices. [Background technology]

[0003] Audio devices, including but not limited to smart audio devices, are being widely deployed and are becoming a common feature in many homes. While existing systems and methods for controlling audio devices provide benefits, improved systems and methods are desirable.

[0004] Notation and Nomenclature Throughout this disclosure, including the claims, the terms "speaker," "loudspeaker," and "audio-reproducing transducer" are used interchangeably to refer to any sound-emitting transducer (or set of transducers). A typical headphone set includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feed(s) may receive different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the phrase performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to a signal or data) is used broadly to indicate performing an operation on the signal or data directly, or on a processed version of the signal or data (e.g., on a version of the signal that has undergone preliminary filtering or preprocessing prior to performing the operation on it).

[0006] Throughout this disclosure, including the claims, the term "system" is used broadly to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or, in some cases, configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0008] Throughout this disclosure, including the claims, the terms "couples" or "coupled" are used to mean either a direct connection or an indirect connection. Thus, when a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0009] As used herein, a "smart device" is generally an electronic device that is configured to communicate with one or more other devices (or networks) via various wireless protocols, such as Bluetooth, Zigbee, Near Field Communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc., and can operate interactively and / or autonomously to some degree. Some notable types of smart devices include smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bands, smart keychains, and smart audio devices. The term "smart device" may also refer to devices that exhibit some properties of ubiquitous computing, such as artificial intelligence.

[0010] In this specification, the expression "smart audio device" is used to indicate a smart device that is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device (e.g., a television (TV)) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is designed to achieve a single purpose, primarily or primarily. For example, a TV can typically play (and is considered to be able to play) audio from program material, but in most cases, modern TVs run some operating system on which applications run locally, including applications for watching television. In this sense, a single-purpose audio device having speaker(s) and microphone(s) is often configured to run local applications and / or services to directly use the speaker(s) and microphone(s). Some single-purpose audio devices may be configured to be grouped to achieve audio playback across a zone or user-configured area.

[0011] One common type of multipurpose audio device is an audio device that implements at least some aspects of the virtual assistant functionality, while other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multipurpose audio device is configured to communicate. Such multipurpose audio devices may be referred to herein as "virtual assistants." A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may be cloud-enabled in some sense or provide the ability to utilize multiple devices (different from the virtual assistant) for applications that are not otherwise fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, e.g., voice recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant may communicate over a network such as the Internet. Virtual assistants may also operate together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them, e.g., the one that is most certain to have heard the wake word, will respond to the wake word. The connected virtual assistants may form a kind of constellation that, in some implementations, may be managed by one main application that may be (or may implement) a virtual assistant.

[0012] As used herein, a "wake word" is used broadly to refer to any sound (e.g., a word uttered by a human being, or any other sound) that the smart audio device is configured to awaken in response to detecting ("listening") for the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "awakening" refers to the device entering a state in which it waits for (or is listening for) a sound command. In some instances, what may be referred to as a "wake word" herein may include more than one word, e.g., a phrase.

[0013] As used herein, the term "wake word detector" refers to a device (or software including instructions for configuring the device) configured to continuously search for a match between real-time sound (e.g., voice) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that a wake word has been detected exceeds a predefined threshold. For example, the threshold may be a predetermined threshold tuned to provide a reasonable compromise between false acceptance and false rejection rates. Following a wake word event, the device may enter a state (sometimes referred to as an "awake" or "attentiveness") in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0014] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals, and in some cases video signals, at least some of which are intended to be heard together. Examples include selections of music, movie soundtracks, movies, television programs, audio portions of television programs, podcasts, live voice calls, synthesized voice responses from smart assistants, and the like. In some cases, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same dialogue in two or more languages. In such cases, only one version of the audio data or a portion thereof (e.g., the version corresponding to a single language) is intended to be played at a time. Summary of the Invention

[0015] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some cases, the method(s) may be implemented, at least in part, by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some such methods may include receiving, by the control system, an output signal from one or more microphones in the audio environment. The output signal may, in some cases, include a signal corresponding to a person's current speech. In some such examples, the current speech may be or may include a wake word utterance.

[0016] Some such methods may include determining, by the control system, in response to the output signal, one or more audio processing modifications to apply to audio data being rendered into loudspeaker feed signals for two or more audio devices in the audio environment based at least in part on the audio device location information and the echo management system information. In some examples, the audio processing modifications may include reducing loudspeaker playback levels for one or more loudspeakers in the audio environment. Some such methods may include causing, by the control system, one or more types of audio processing modifications to be applied.

[0017] In some examples, at least one of the audio processing modifications may correspond to an increase in the signal-to-echo ratio. According to some such examples, the echo management system information may include a model of echo management system performance. For example, the model of echo management system performance may include an acoustic echo canceller (AEC) performance matrix. In some examples, the model of echo management system performance may include a measure of expected echo return loss enhancement provided by the echo management system.

[0018] According to some examples, determining the one or more types of audio processing modifications may be based at least in part on optimizing a cost function. Alternatively or additionally, in some examples, the one or more types of audio processing modifications may be based at least in part on acoustic models of inter-device echo and intra-device echo. Alternatively or additionally, in some examples, the one or more types of audio processing modifications may be based at least in part on inter-audibility of audio devices in the audio environment, e.g., on an inter-audibility matrix.

[0019] In some examples, the one or more types of audio processing modifications may be based at least in part on an estimated location of the person. In some such examples, the estimated location of the person may be based at least in part on output signals from multiple microphones in the audio environment. According to some such examples, the audio processing modifications may include modifying a rendering process to warp the rendering of the audio signal away from the estimated location of the person.

[0020] Alternatively or additionally, in some examples, one or more types of audio processing modifications may be based at least in part on a listening objective, which in some such examples may include spatial components, frequency components, or both spatial and frequency components.

[0021] Alternatively or additionally, in some examples, the one or more types of audio processing modifications may be based at least in part on one or more constraints. In some such examples, the one or more constraints may be based at least in part on a perceptual model. Alternatively or additionally, the one or more constraints may be based at least in part on audio content energy preservation, audio spatiality preservation, an audio energy vector, a regularization constraint, or a combination thereof. Alternatively or additionally, some examples may include updating an acoustic model of the audio environment, a model of echo management system performance, or both after applying the one or more types of audio processing modifications.

[0022] In some examples, the one or more types of audio processing modifications may include spectral modifications, hi some such examples, the spectral modifications may include reducing a level of the audio data in a frequency band from 500 Hz to 3 KHz.

[0023] Aspects of some disclosed implementations include a control system configured (e.g., programmed) to perform one or more of the disclosed methods or steps thereof, and a tangible, non-transitory computer-readable medium (e.g., a disk or other tangible storage medium) implementing non-transitory storage of data that stores code (e.g., executable code to perform) for performing one or more of the disclosed methods or steps thereof. For example, some disclosed embodiments may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including one or more of the disclosed methods or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0024] Some or all of the operations, functions, and / or methods described herein may be executed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices, such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in software-stored non-transitory media.

[0025] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the following description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0026] [Figure 1A] FIG. 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the present disclosure. [Figure 1B] An example of an audio environment is shown. [Diagram 2] 1C illustrates the echo paths between three of the audio devices of FIG. 1B. [Diagram 3] FIG. 2 is a system block diagram illustrating components of an audio device according to an example. [Figure 4] 1 illustrates elements of a ducking module according to an example. [Diagram 5] FIG. 1 is a block diagram illustrating an example of an audio device including a ducking module. [Figure 6] FIG. 13 is a block diagram illustrating an alternative example of an audio device including a ducking module. [Figure 7] FIG. 1 is a flow diagram outlining an example of a method for determining a ducking solution. [Figure 8] FIG. 11 is a flow diagram outlining another example of a method for determining a ducking solution. [Figure 9] FIG. 1 is a flow diagram outlining an example of the disclosed method. [Figure 10] 1B is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. [Figure 11] FIG. 2 is a block diagram of elements of an example embodiment configured to implement a zone classifier. [Figure 12] 1B is a flow diagram outlining an example of a method that may be performed by an apparatus such as apparatus 150 of FIG. 1A. [Figure 13] 1B is a flow diagram outlining another example of a method that may be performed by an apparatus such as apparatus 150 of FIG. 1A. [Figure 14] 1B is a flow diagram outlining another example of a method that may be performed by an apparatus such as apparatus 150 of FIG. 1A. [Figure 15]FIG. 1 illustrates an example set of speaker activation and object rendering positions. [Figure 16] FIG. 1 illustrates an example set of speaker activation and object rendering positions. [Figure 17] 1B is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as that shown in FIG. 1A. [Figure 18] 1 is a graph of speaker activation in an exemplary embodiment. [Figure 19] 1 is a graph of object rendering positions in an example embodiment. [Figure 20] 1 is a graph of speaker activation in an exemplary embodiment. [Figure 21] 1 is a graph of object rendering positions in an example embodiment. [Figure 22] 1 is a graph of speaker activation in an exemplary embodiment. [Diagram 23] 1 is a graph of object rendering positions in an example embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0027] Some embodiments are configured to implement a system including coordinated audio devices, also referred to herein as orchestrated audio devices. In some implementations, the orchestrated audio devices may include smart audio devices. According to some such implementations, two or more of the smart audio devices may be or may be configured to implement a wake word detector. Thus, in such examples, multiple microphones (e.g., asynchronous microphones) may be present in the audio environment.

[0028] Currently, designers generally view audio devices as a single interface point for audio that may be a mix of entertainment, communication, and information services. Using audio for notifications and voice control has the advantage of avoiding visual or physical intrusion. In all forms of interactive audio, the problem of increasing full-duplex (input and output) audio capabilities remains a challenge. When there is audio output in a room that is not related to in-room transmission or information-based capture, it is desirable to remove this audio from the captured signal (e.g., by echo cancellation and / or echo suppression).

[0029] Some disclosed embodiments provide techniques for managing the listener or "user" experience to improve a key criterion for full-duplex success with one or more audio devices. This criterion is known herein as the signal-to-echo ratio (SER), also referred to as the voice-to-echo ratio, and may be defined as the ratio between the voice signal or other desired signal to be captured in an audio environment (e.g., a room) via one or more microphones, and the "echo" presented at the audio device, including the signal from one or more microphones corresponding to output program content, interactive content, etc. being played by one or more loudspeakers of the audio environment. Those skilled in the art will recognize that in this context, "echo" is not necessarily reflected before being captured by the microphone.

[0030] Such an embodiment may be useful in situations where there are two or more audio devices within acoustic range of a user, each capable of presenting audio program material at an appropriate volume at the user's location for a desired entertainment, communication, or information service. The value of such an embodiment may be particularly high when there are three or more audio devices in the vicinity of the user as well. If an audio device is closer to the user, the audio device may be more advantageous in terms of its ability to accurately localize sounds or convey specific audio signaling and imaging to the user. However, if these audio devices include one or more microphones, one or more of these audio devices may also have a microphone system that is preferred for picking up the user's voice.

[0031] Audio devices often need to respond to a user's voice commands while the audio device is playing content, in which case the microphone system of the audio device will detect the content played by the audio device. In other words, the audio device will hear its own "echo." The special nature of wake word detectors may enable such devices to outperform more common speech recognition engines in the presence of this echo. A common mechanism implemented in these audio devices, commonly referred to as "ducking," involves reducing the audio device's playback level after detecting the wake word so that the audio device can better recognize post-wake word commands issued by the user. Such ducking generally results in an improvement in SER, a common metric for predicting speech recognition performance.

[0032] In the context of distributed and orchestrated audio devices, where multiple audio devices are located in a single acoustic space (also referred to herein as an "audio environment"), ducking only the playback of a single audio device may not be an optimal solution. This may be true in part because "echoes" (detected audio playback) from other unducked audio devices in the audio environment may limit the maximum achievable SER by ducking only the playback of a single audio device.

[0033] Thus, some disclosed embodiments may cause audio processing modifications for two or more audio devices of an audio environment to increase the SER at one or more microphones of the audio environment. In some examples, the audio processing modification(s) may be determined according to the results of an optimization process. According to some examples, the optimization process may include trading off objective sound capture performance goals against constraints that preserve one or more aspects of a user's listening experience. In some examples, the constraints may be perceptual constraints, objective constraints, or a combination thereof. Some disclosed examples include implementing a model that describes the echo management signal chain, sometimes referred to herein as a "capture stack," the acoustic space, and the perceptual impact of the audio processing modification(s) and explicitly trading them off (e.g., finding a solution that takes all such factors into account).

[0034] According to some examples, the process may include, for example, a closed-loop system in which the acoustic and capture stack models are updated after each audio processing change (such as each change of one or more rendering parameters). Some such examples may include iteratively improving the performance of the audio system over time.

[0035] Some disclosed implementations may be based at least in part on one or more of the following factors, or a combination thereof: A model of the acoustic environment (in other words the acoustics of the audio environment) and the echo management signal chain that is capable of predicting the achieved SER for a given configuration and output solution; Constraints that limit the output solution according to both objective and subjective metrics, including: - content energy conservation; - Spatiality preservation; - energy vector; and / or - Regularization such as: ◇ Level 1 regularization (L1), Level 2 regularization (L2), or any level of regularization (LN) distortion of two-dimensional or three-dimensional arrays of loudspeaker activation, sometimes referred to herein as "waffle"; and / or ◇ L1, L2 or LN regularization of ducking gain; Listening objectives that may determine both the spatial and level components of the target solution.

[0036] According to some examples, the ducking solution may be based at least in part on one or more of the following factors, or a combination thereof: In some cases, a simple gain may be applied to already rendered audio content; These gains may be full-band or frequency dependent, depending on the particular implementation; Input to a renderer that, in some instances, may be used in combination with Waffle; and / or Inputs to a waffle generating device, module, etc., which may be referred to herein as a waffle maker, and which in some instances may be a component of a renderer. The waffle maker, in some instances, may use such inputs to generate ducked waffles.

[0037] In some implementations, such input to the wafflemaker and / or renderer may be used to modify audio playback such that audio objects may appear to be "moved away" from the location where the wake word was detected. Some such implementations may include determining relative activation of a set of loudspeakers in an audio environment by optimizing a cost that is a function of (a) a model of the perceived spatial location of the audio signal played when played on a set of loudspeakers in the audio environment, (b) a measure of proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker in the set of loudspeakers, and (c) one or more additional dynamically configurable functions. Some such implementations are described in more detail below.

[0038] FIG. 1A is a block diagram illustrating an example of components of a device capable of implementing various aspects of the present disclosure. As with other figures provided herein, the types and numbers of elements shown in FIG. 1A are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. According to some examples, device 150 may be configured to perform at least some of the methods disclosed herein. In some implementations, device 150 may be or include one or more components of an audio system. For example, device 150 may be an audio device, such as a smart audio device, in some implementations. In other examples, device 150 may be a mobile device (such as a mobile phone), a laptop computer, a tablet device, a television, or another type of device.

[0039] According to some alternative implementations, the apparatus 150 may be or include a server. In some such examples, the apparatus 150 may be or include an encoder. Thus, in some instances, the apparatus 150 may be a device configured for use in an audio environment, such as a home audio environment, while in other instances, the apparatus 150 may be a device configured for use in the "cloud," e.g., a server.

[0040] In this example, device 150 includes interface system 155 and control system 160. Interface system 155 may be configured, in some implementations, for communication with one or more other devices of an audio environment. The audio environment may be, in some examples, a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, an automobile environment, a train environment, a street or sidewalk environment, a park environment, etc. Interface system 155 may be configured, in some implementations, for exchanging control information and associated data with audio devices of the audio environment. The control information and associated data may, in some examples, relate to one or more software applications that device 150 is executing.

[0041] The interface system 155 may be configured, in some implementations, to receive or provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, an audio signal. In some instances, the audio data may include spatial data, such as channel data and / or spatial metadata. The metadata may have been provided, for example, by what may be referred to herein as an “encoder.” In some examples, the content stream may include video data and audio data corresponding to the video data.

[0042] The interface system 155 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus (USB) interfaces). According to some implementations, the interface system 155 may include one or more wireless interfaces. The interface system 155 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 155 may include one or more interfaces between the control system 160 and a memory system, such as the optional memory system 165 shown in FIG. 1A. However, in some cases, the control system 160 may include the memory system. The interface system 155 may be configured to receive input from one or more microphones in the environment, in some implementations.

[0043] Control system 160 may include, for example, a general-purpose single or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0044] In some implementations, the control system 160 may reside in more than one device. For example, in some implementations, a portion of the control system 160 may reside in a device in one of the environments shown herein, and another portion of the control system 160 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer). In other examples, a portion of the control system 160 may reside in a device in one of the environments shown herein, and another portion of the control system 160 may reside in one or more other devices of the environment. For example, the control system functionality may be distributed across multiple smart audio devices of the environment, or may be shared by an orchestration device (such as what may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 160 may reside in a device implementing a cloud-based service, such as a server, and another portion of the control system 160 may reside in another device implementing a cloud-based service, such as another server, a memory device, etc. The interface system 155 may also reside in more than one device, in some examples.

[0045] In some implementations, the control system 160 may be configured to at least partially execute the methods disclosed herein. According to some examples, the control system 160 may be configured to determine and cause audio processing modifications for two or more audio devices of the audio environment to increase the SER at one or more microphones of the audio environment. In some examples, the audio processing modification(s) may be based at least in part on audio device location information and echo management system information. According to some examples, the audio processing modification(s) may be responsive to a microphone output signal corresponding to a person's current speech, such as speaking a wake word. In some examples, the audio processing modification(s) may be determined according to the results of an optimization process. According to some examples, the optimization process may include trading off objective sound capture performance goals against constraints that preserve one or more aspects of the user's listening experience. In some examples, the constraints may be perceptual constraints, objective constraints, or a combination thereof.

[0046] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices, such as memory devices described herein, including but not limited to random access memory (RAM) devices, read only memory (ROM) devices, and the like. The one or more non-transitory media may be present, for example, in optional memory system 165 and / or control system 160 shown in FIG. 1A. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. The software may include, for example, instructions for controlling at least one device to perform some or all of the methods disclosed herein. The software may be executable by one or more components of a control system, such as, for example, control system 160 of FIG. 1A.

[0047] In some examples, the device 150 may include an optional microphone system 170 shown in FIG. 1A. The optional microphone system 170 may include one or more microphones. According to some examples, the optional microphone system 170 may include an array of microphones. In some examples, the array of microphones may be configured to determine direction of arrival (DOA) and / or time of arrival (TOA) information, for example, according to instructions from the control system 160. The array of microphones may be configured for receive-side beamforming in some cases, for example, according to instructions from the control system 160. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, the device 150 may not include the microphone system 170. However, in some such implementations, the device 150 may nevertheless be configured to receive microphone data for one or more microphones in the audio environment via the interface system 160. In some such implementations, a cloud-based implementation of device 150 may be configured to receive microphone data, or data corresponding to microphone data, from one or more microphones in the audio environment via interface system 160.

[0048] According to some implementations, the device 150 may include an optional loudspeaker system 175 shown in FIG. 1A. The optional loudspeaker system 175 may include one or more loudspeakers, sometimes referred to herein as "speakers" or, more generally, "audio reproduction transducers." In some examples (e.g., cloud-based implementations), the device 150 may not include the loudspeaker system 175.

[0049] In some implementations, the device 150 may include an optional sensor system 180 as shown in FIG. 1A. The optional sensor system 180 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some implementations, the optional sensor system 180 may include one or more cameras. In some implementations, the camera may be a freestanding camera. In some examples, the one or more cameras of the optional sensor system 180 may be present in a smart audio device, which may be configured to at least partially implement a virtual assistant in some examples. In some such examples, the one or more cameras of the optional sensor system 180 may be present in a television, a mobile phone, or a smart speaker. In some examples, the device 150 may not include the sensor system 180. However, in some such implementations, the device 150 may nevertheless be configured to receive sensor data for one or more sensors in the audio environment via the interface system 160.

[0050] In some implementations, the device 150 may include an optional display system 185 shown in FIG. 1A. The optional display system 185 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some cases, the optional display system 185 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 185 may include one or more displays of a smart audio device. In other examples, the optional display system 185 may include a television display, a laptop display, a mobile device display, or another type of display. In some examples in which the device 150 includes a display system 185, the sensor system 180 may include a touch sensor system and / or a gesture sensor system in proximity to one or more displays of the display system 185. According to some such implementations, the control system 160 may be configured to control the display system 185 to present one or more graphical user interfaces (GUIs).

[0051] According to some such examples, device 150 may be or include a smart audio device. In some such implementations, device 150 may be or include a wake word detector. For example, device 150 may be or include a virtual assistant.

[0052] Figure IB illustrates an example of an audio environment. As with other figures provided herein, the types, numbers, and arrangements of elements shown in Figure IB are provided merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements, different arrangements of elements, etc.

[0053] According to this example, audio environment 100 includes audio devices 110A, 110B, 110C, 110D, and 110E. Audio devices 110A-110E may be instances of apparatus 150 of FIG. 1A in some examples. In this example, each audio device 110A-110E includes at least one respective one of microphones 120A, 120B, 120C, 120D, and 120E and at least one respective one of loudspeakers 121A, 121B, 121C, 121D, and 121E. In this example, individual instances of microphones 120A-120E and loudspeakers 121A-121E are shown. However, one or more of audio devices 110A-110E may include a microphone system including multiple microphones and / or a loudspeaker system including multiple loudspeakers. According to some examples, each of the audio devices 110A-110E may be a smart audio device, such as a smart speaker.

[0054] In some examples, some or all of the audio devices 110A-110E may be orchestrated audio devices that operate (at least in part) according to instructions from an orchestration device. According to some such examples, the orchestration device may be one of the audio devices 110A-110E. In other examples, the orchestration device may be another device, such as a smart home hub.

[0055] In this case, persons 101A and 101B are in an audio environment. In this example, an acoustic event is caused by a speaker 101A speaking near an audio device 110A. An element 102 is intended to represent the voice of person 101A. In this example, the voice 102 corresponds to the speaking of a wake word by person 101A.

[0056] Figure 2 shows the echo paths between three of the audio devices of Figure 1B. Elements of Figure 2 that are not described with reference to Figure 1B are as follows: 200AA: echo path from device 110A to device 110A (from loudspeaker 121A to microphone 120A); 200AB: echo path from device 110A to device 110B (loudspeaker 121A to microphone 120B); 200AC: echo path from device 110A to device 110C (from loudspeaker 121A to microphone 120C); 200BA: echo path from device 110B to device 110A (from loudspeaker 121B to microphone 120A); 200BB: echo path from device 110B to device 110B (from loudspeaker 121B to microphone 120B); 200BC: echo path from device 110B to device 110C (from loudspeaker 121B to microphone 120C); 200CA: echo path from device 110C to device 110A (from loudspeaker 121C to microphone 120A); 200CB: echo path from device 110C to device 110C (from loudspeaker 121C to microphone 120B); 200CC: Echo path from device 110C to device 110C (from loudspeaker 121C to microphone 120C).

[0057] These echo paths indicate the effect that the reproduced audio or "echo" of each audio device has on the other audio devices. This effect is sometimes referred to herein as the "mutual audibility" of the audio devices. Mutual audibility depends on various factors including the position and orientation of each audio device in the audio environment, the playback level of each audio device, the loudspeaker capabilities of each audio device, etc. Some implementations may include constructing a more accurate representation of the mutual audibility of the audio devices in the audio environment, such as an audibility matrix A that represents the energy of echo paths 200AA-200CC. For example, each column of the audibility matrix may represent an audio device loudspeaker and each row of the audibility matrix may represent an audio device microphone, or vice versa. In some such audibility matrices, the diagonal of the audibility matrix may represent the echo path from the loudspeaker(s) of an audio device to the microphone(s) of the same audio device.

[0058] From FIG. 2, it can be seen that if the echo paths 200CA and 200BA from audio devices 110C and 110B, respectively, to audio device 100A have strong coupling, the echo from audio device 110C and audio device 110B will become significant in the echo (residual) of audio device 110A. In such a case, if the audio system's response to detecting voice 102 (which corresponds to the wake word in this example) is simply to lower the volume of the nearest loudspeaker, this includes ducking playback only from loudspeaker(s) 121A. If this is the audio system's only response to detecting the wake word, potential SER improvements may be significantly limited. Thus, some disclosed examples may include other responses to detecting the wake word in such situations.

[0059] 3 is a system block diagram representing components of an audio device according to an example. In FIG. 3, the block representing audio device 110A includes loudspeaker 121A and microphone 120A. In some examples, loudspeaker 121A may be one of multiple loudspeakers in a loudspeaker system, such as loudspeaker system 175 of FIG. 1A. Similarly, according to some implementations, microphone 120A may be one of multiple microphones in a microphone system, such as microphone system 170 of FIG. 1A.

[0060] In this example, audio device 110A includes renderer 201A, echo management system (EMS) 203A, and audio processor / communications block 240A. In this example, EMS 203A may be or include an acoustic echo canceller (AEC), an acoustic echo suppressor (AES), or both an AEC and an AES. According to this example, renderer 201A is configured to render audio data 301 received by or stored in audio device 110A for playback on loudspeaker 121A. In some examples, the audio data may include one or more audio signals and associated spatial data. The spatial data may indicate, for example, an intended perceived spatial location corresponding to an audio signal. In some examples, the spatial data may be or include spatial metadata corresponding to an audio object. In this example, the renderer output 220A is provided to the loudspeaker 121A for playback, and the renderer output 220A is also provided to the EMS 203A as a reference for echo cancellation.

[0061] In addition to receiving the renderer output 220A, in this example, the EMS 203A also receives a microphone signal 223A from the microphone 120A. In this example, the EMS 203A processes the microphone signal 223A and provides an echo-canceled residual 224A (sometimes referred to herein as "residual output 224A") to the audio processor / communications block 240A.

[0062] In some implementations, the voice processor / communications block 240A may be configured for voice recognition functions. In some examples, the voice processor / communications block 240A may be configured to provide telecommunications services such as telephone calls, video conferencing, etc. Although not shown in FIG. 3, the voice processor / communications block 240A may be configured to communicate with one or more networks, the loudspeaker 121A and / or the microphone 120A, for example, via an interface system. The one or more networks may include, for example, a local Wi-Fi network, one or more types of telephone networks, etc.

[0063] 4 illustrates elements of a ducking module according to one example. In this implementation, the ducking module 400 is implemented by an instance of the control system 160 of FIG. 1A. In this example, the elements of FIG. 4 are as follows: 401: Acoustic models of inter-device and intra-device echo, including, in some examples, an acoustic model of user speech. According to some examples, the acoustic model 401 may be or may include a model of how playback from each audio device in an audio environment appears as echo detected by microphones of all audio devices (own and others) in the audio environment. In some examples, the acoustic model 401 may be based at least in part on an audio environment impulse response estimate, characteristics of the impulse response such as peak amplitude, decay time, etc. In some examples, the acoustic model 401 may be based at least in part on an audibility estimate. In some such examples, the audibility estimate may be based on microphone measurements. Alternatively or additionally, the audibility estimate may be inferred by audio device position, e.g., based on echo power being inversely proportional to the distance between audio devices. In some examples, the acoustic model 401 may be based at least in part on long-term estimates of AEC / AES filter taps. In some examples, the acoustic model 401 may be based at least in part on waffles, which may include information regarding the capabilities (loudness) of the loudspeaker(s) of each audio device; 452: spatial information including information regarding a position of each of a plurality of audio devices within the audio environment. In some examples, the spatial information 452 may include information regarding an orientation of each of the plurality of audio devices. According to some examples, the spatial information 452 may include information regarding a position of one or more persons within the audio environment. In some instances, the spatial information 452 may include information regarding an impulse response of at least a portion of the audio environment; 402: A model of EMS performance, which may indicate the performance of the EMS 203A of FIG. 3 or another EMS (such as that of FIG. 5 or FIG. 6). In this example, the EMS performance model 402 predicts how well an EMS (AEC, AES, or both) will perform. In some examples, the EMS performance model 402 may predict how well an EMS will perform given the current audio environment impulse response, the current noise level(s) of the audio environment, the type of algorithm(s) being used to implement the EMS, the type of content being played in the audio environment, the number of echo criteria being fed to the EMS algorithm(s), the capabilities / quality of the loudspeakers in the audio environment (nonlinearities in the loudspeakers will impose an upper limit on expected performance), or a combination thereof. According to some examples, the EMS performance model 402 may be based at least in part on empirical observations, for example, by observing how an EMS performs under various conditions, storing data points based on such observations, and building a model (e.g., by fitting a curve) based on these data points. In some examples, the EMS performance model 402 may be based at least in part on machine learning, such as by training a neural network based on empirical observations of EMS performance. Alternatively or additionally, the EMS performance model 402 may be based at least in part on a theoretical analysis of the algorithm(s) used by the EMS. In some examples, the EMS performance model 402 may indicate an ERLE (Echo Return Loss Enhancement) caused by the operation of the EMS, which is a useful metric for evaluating EMS performance. The ERLE may indicate, for example, the amount of additional signal loss applied by the EMS between each audio device; 403: Information regarding one or more current listening objectives. In some examples, the listening objective information 403 may set a goal, such as a SER goal or a SER improvement goal, for the ducking module to achieve. According to some examples, the listening objective information 403 may include both a spatial component and a level component; · 450: Target-related factors that may be used to determine the target, such as an external trigger, an acoustic event, or a mode indication; 404: One or more constraints to be applied during the process of determining a ducking solution, such as a constraint that trades off improved listening performance (in other words, improved ability of one or more microphones to capture audio, such as human speech) against other metrics (such as degradation of the listening experience of humans in the audio environment). For example, in one example, a constraint may prevent the ducking module 400 from reducing loudspeaker playback levels for some or all loudspeakers in the audio environment to an unacceptably low level, such as 0 dBFS (decibels relative to full scale); 451: Metadata about the current audio content, which may include spatiality metadata, level metadata, content type metadata, etc. Such metadata may provide information (directly or indirectly) about the effect that ducking one or more loudspeakers will have on the listening experience of a person in the audio environment. For example, if the spatiality metadata indicates that a "loud" audio object is being played by multiple loudspeakers in the audio environment, ducking one of those loudspeakers may not have an adverse effect on the listening experience. As another example, if the content metadata indicates that the content is a podcast, in some instances a monologue or dialogue in the podcast may be played by multiple loudspeakers in the audio environment, and therefore ducking one of those loudspeakers may not have an adverse effect on the listening experience. However, if the content metadata indicates that the audio content corresponds to a movie or television program, the dialogue of such content may be reproduced primarily or entirely through particular loudspeakers (e.g., "front" loudspeakers), and ducking those loudspeakers may have an undesirable effect on the listening experience; 405: The model used to derive the perceptually driven constraints. Some detailed examples are given elsewhere in this specification; 406: An optimization algorithm, which may vary according to the particular implementation. In some examples, the optimization algorithm 406 may be or include a closed-form optimization algorithm. In some cases, the optimization algorithm 406 may be or include an iterative process. Some detailed examples are described elsewhere herein; 480: Ducking solution output by ducking module 400. As described in more detail elsewhere herein (e.g., with reference to Figures 5 and 6), ducking solution 480 may vary according to a variety of factors, including whether ducking solution 480 is provided to a renderer or whether ducking solution 480 is provided for audio data output from a renderer.

[0064] According to some examples, the AEC model 402 and the acoustic model 401 may provide a means for estimating or predicting the SER, for example, as described below.

[0065] The constraint(s) 404 and the perceptual model 405 may be used to ensure that the ducking solution 480 output by the ducking module 400 is neither degenerate nor trivial. One example of a trivial solution is to set the playback level globally to 0. The constraint(s) 404 may be perceptual and / or objective. According to some examples, the constraint(s) 404 may be based at least in part on a perceptual model, such as a model of human hearing. In some examples, the constraint(s) 404 may be based at least in part on audio content energy conservation, audio spatiality conservation, audio energy vectors, or one or more combinations thereof. According to some examples, the constraint(s) 404 may be or include a regularization constraint. The listening intent information 403 may determine, for example, a current target SER improvement to be made by distributed ducking (in other words, ducking two or more audio devices of an audio environment).

[0066] Global Optimization In some examples, the selection of audio device(s) for ducking involves using the estimated SER and / or wake word information obtained when the wake word is detected to select the audio device to listen for the next utterance. If this audio device selection is incorrect, it is highly unlikely that the best listening device will be able to understand the command spoken after the wake word. This is because automatic speech recognition (ASR) is more difficult than wake word detection (WWD), which is one of the motivating factors for ducking. If the best listening device was not ducked, ASR is likely to fail for all of the audio devices. Thus, in some such examples, the ducking method involves optimizing the ASR stage by ducking the closest (or best estimated) audio device(s) using the previous estimate (from the WWD).

[0067] Thus, some ducking implementations include using a previous estimate when determining a ducking solution. However, in implementations such as that shown in FIG. 4, listening objectives and constraints can be applied to achieve more robust ASR performance. In some such examples, the ducking method may include configuring a ducking algorithm such that the SER improvement is significant at all potential user locations in the acoustic space. In this way, it can be ensured that at least one of the microphones in the room has sufficient SER for robust ASR performance. Such implementations may be advantageous when talker locations are unknown or there is uncertainty regarding talker locations. Some such examples may include considering variance of talker and / or audio device position estimates by spatially spreading the SER improvement zone to account for one or more uncertainties.

[0068] Some such examples may include the use of the δ parameter of the discussion below, or a similar parameter. Other examples may include multi-parameter models that describe or accommodate uncertainties in talker position and / or audio device position estimates.

[0069] In some embodiments, the ducking method may be performed in the context of one or more user zones. As described in more detail later in this document, a set of acoustic features

number

[0070] In some examples, the ducking method may use quantities related to one or more user zones in the process of selecting an audio device for ducking or other audio processing modification. If both z and p are available, an example audio device selection decision may be made according to the following formula:

number

[0071] In some implementations, the acoustic features W(j) may be used directly in the ducking method. For example, if the wake word confidence score associated with utterance j is w(j), n If (j), then audio device selection may be performed according to the following formula:

number

[0072] 5 is a block diagram illustrating an example of an audio device including a ducking module. The renderer 201A, the voice processor / communications block 240A, the EMS 203A, the loudspeaker(s) 121A, and the microphone(s) 120A may function substantially as described with reference to FIG. 3, except as described below. In this example, the renderer 201A, the voice processor / communications block 240A, the EMS 203A, and the ducking module 400 are implemented by an instance of the control system 160 described with reference to FIG. 1A. The ducking module 400 may be, for example, an instance of the ducking module 400 described with reference to FIG. 4. Thus, the ducking module 400 may be configured to determine one or more types of audio processing modifications (indicated by or corresponding to a ducking solution 480) to be applied to rendered audio data (e.g., audio data rendered to loudspeaker feed signals) for at least the audio device 110A. The audio processing modification may be or may include a reduction in loudspeaker playback levels of one or more loudspeakers in the audio environment. In some examples, the ducking module 400 may be configured to determine one or more types of audio processing modifications to apply to rendered audio data for two or more audio devices in the audio environment.

[0073] In this example, the renderer output 220A and the ducking solution 480 are provided to a gain multiplier 501. In some examples, the ducking solution 480 includes a gain for the gain multiplier 501 to apply to the renderer output 220A to generate processed audio data 502. According to this example, the processed audio data 502 is provided to the EMS 203A as a local reference for echo cancellation, where the processed audio data 502 is also provided to the loudspeaker(s) 121A for playback.

[0074] In some examples, ducking module 400 may be configured to determine ducking solution 480 as described below with reference to Figure 8. According to some examples, ducking module 400 may be configured to determine ducking solution 480 as described in the "Optimizing for a Specific Device" section below.

[0075] 6 is a block diagram illustrating an alternative example of an audio device including a ducking module. The renderer 201A, the audio processor / communications block 240A, the EMS 203A, the loudspeaker(s) 121A, and the microphone(s) 120A may function substantially as described with reference to FIG. 3, except as described below. The ducking module 400 may be, for example, an instance of the ducking module 400 described with reference to FIG. 4. In this example, the renderer 201A, the audio processor / communications block 240A, the EMS 203A, and the ducking module 400 are implemented by an instance of the control system 160 described with reference to FIG. 1A.

[0076] According to this example, ducking module 400 is configured to provide ducking solution 480 to renderer 201A. In some such examples, ducking solution 480 may cause renderer 201A to implement one or more types of audio processing modifications that may include a reduction in loudspeaker playback levels during the process of rendering received audio data 301 (or during the process of rendering audio data stored in a memory of audio device 110A). In some examples, ducking module 400 may be configured to determine ducking solution 480 for implementation of one or more types of audio processing modifications via one or more instances of renderer 201A in one or more other audio devices in the audio environment.

[0077] In this example, the renderer output 201A outputs processed audio data 502. According to this example, the processed audio data 502 is provided to the EMS 203A as a local reference for echo cancellation, where the processed audio data 502 is also provided to the loudspeaker(s) 121A for playback.

[0078] In some examples, the ducking solution 480 may include one or more penalties implemented by a flexible rendering algorithm, for example as described below. In some such examples, the penalty may be a loudspeaker penalty estimated to cause the desired SER improvement. According to some examples, determining the one or more types of audio processing modifications may be based on optimization of a cost function by the ducking module 400 or by the renderer 201A.

[0079] FIG. 7 is a flow diagram outlining an example of a method for determining a ducking solution. In some examples, the method 720 may be performed by an apparatus such as that shown in FIG. 1A, FIG. 5, or FIG. 6. In some examples, the method 720 may be performed by a control system of an orchestration device, which in some cases may be an audio device. In some examples, the method 720 may be performed, at least in part, by a ducking module, such as the ducking module 400 of FIG. 4, FIG. 5, or FIG. 6. According to some examples, the method 720 may be performed, at least in part, by a renderer. The blocks of the method 720, as well as other methods described herein, are not necessarily performed in the order shown. Additionally, such methods may include more or less blocks than those illustrated and / or described.

[0080] In this example, the process begins at block 725. In some cases, block 725 may correspond to a boot-up process or a time when the boot-up process is complete and a device configured to perform method 720 is ready to function.

[0081] According to this example, block 730 includes waiting for a wake word to be detected. If method 720 is being performed by an audio device, block 730 may also include playing rendered audio data corresponding to received or stored audio content, such as music content, a podcast, an audio soundtrack for a movie or television program, etc.

[0082] In this example, upon detection of the wake word (e.g., at block 730), the SER of the wake word is estimated at block 735. S(a) is an estimate of the speech-to-echo ratio at device a. By definition, the speech-to-echo ratio in dB is given by:

number

[0083] In the above formula,

number

number

number

number

number

number

[0084] According to this example, block 740 includes obtaining a target SER (in this case from block 745) and calculating a target SER improvement. In some implementations, the desired SER improvement (SERI) may be determined as follows:

number

[0085] In the above equation, m represents the device / microphone location for which the SER is being improved, and TargetSER represents a threshold that may be set according to the application in use in some examples. For example, a wake word detection algorithm may tolerate a lower operating SER than a command detection algorithm, which may tolerate a lower operating SER than a large vocabulary speech recognizer. Typical values ​​for TargetSER are on the order of -6 dB to 12 dB. In some cases, if S(m) is not known or is not easily estimated, a preset value based on offline measurements of speech and echo recorded in a typical echoic room or setting may be sufficient. Some embodiments may determine the audio device for which audio processing (e.g., rendering) should be modified by specifying f_n ranging from 0 to 1. Other embodiments may use a speech-to-echo ratio improvement decibel s, potentially calculated according to: n (also denoted herein as s_n), may include specifying the degree to which the audio processing (e.g., rendering) should be modified:

number

[0086] Some embodiments may compute f_n directly from the device geometry, for example as follows:

number

[0087] In the above equation, m represents the index of the device to be selected for maximum audio processing (e.g., rendering) modification, and H(m,i) represents the approximate physical distance between devices m and i. Other implementations may include other choices of relaxation or smoothing functions over device geometries.

[0088] Thus, H is a property of the physical location of the audio device within the audio environment. H may be determined or estimated according to various methods, depending on the particular implementation. Various examples of methods for estimating the location of the audio device within the audio environment are described below.

[0089] In this example, block 750 includes calculating what may be referred to herein as a "ducking solution." The ducking solution may include determining a reduction in loudspeaker playback levels for one or more loudspeakers in the audio environment, although the ducking solution may also include one or more other audio processing modifications, such as those disclosed herein. The ducking solution determined in block 750 is an example of ducking solution 480 of FIGS. 4, 5, and 6. Thus, block 750 may be performed by ducking module 400.

[0090] According to this example, the ducking solution determined in block 750 is based (at least in part) on the target SER, the ducking constraint (represented by block 755), the AEC model (represented by block 765), and the acoustic model (represented by block 760). The acoustic model may be an instance of the acoustic model 401 of FIG. 4. The acoustic model may be based at least in part on inter-device audibility, for example, sometimes referred to herein as inter-device echo or inter-audibility. The acoustic model may be based at least in part on intra-device echo, in some examples. In some cases, the acoustic model may be based at least in part on an acoustic model of the user's speech, such as acoustic characteristics of typical human speech, acoustic characteristics of human speech previously detected in the audio environment, etc. The AEC model may be an instance of the AEC model 402 of FIG. 4, in some examples. The AEC model may be indicative of the performance of the EMS 203A of FIG. 5 or FIG. 6. In some examples, the EMS performance model 402 may be indicative of an actual or expected ERLE (echo return loss enhancement) caused by the operation of the AEC. The ERLE may indicate, for example, the amount of additional signal loss applied by the AEC between each audio device. According to some examples, the EMS performance model 402 may be based at least in part on an expected ERLE for a given number of echo criteria. In some examples, the EMS performance model 402 may be based at least in part on an estimated ERLE calculated from actual microphone and residual signals.

[0091] In some examples, the ducking solution determined in block 750 may be an iterative solution, while in other examples, the ducking solution may be a closed-form solution. Examples of both iterative and closed-form solutions are disclosed herein.

[0092] In this example, block 770 includes applying the ducking solution determined in block 750. In some examples, the ducking solution may be applied to the rendered audio data, as shown in Figure 5. In other examples, the ducking solution determined in block 750 may be provided to a renderer, as shown in Figure 6. The ducking solution may be applied as part of the process of rendering the audio data input to the renderer.

[0093] In this example, block 775 includes detecting another utterance, which in some examples may be a command uttered after the wake word. According to this example, block 780 includes estimating the SER of the utterance detected in block 775. In this example, block 785 includes updating the AEC model and the acoustic model based at least in part on the SER estimated in block 780. According to this example, the process of block 785 occurs after the ducking solution is applied. In a perfect system, the actual SER improvement and the actual SER would be exactly what was targeted. In a real-world system, the actual SER improvement and the actual SER will likely differ from what was targeted. In such an implementation, method 720 includes using at least the SER to update the information and / or models used to calculate the ducking solution. According to this example, the ducking solution is based at least in part on the acoustic model of block 760. For example, the acoustic model of block 760 may have indicated very strong acoustic coupling between audio device X and microphone Y, and as a result, the ducking solution may have included significantly ducking the signal from microphone Y. However, after estimating the SER of the detected speech in block 775 while the ducking solution was applied, the control system may have determined that the actual SER and / or SERI was not as expected. If so, block 785 may include updating the acoustic model accordingly (in this example, by reducing the acoustic coupling estimate between audio device X and microphone Y). According to this example, the process then returns to block 730.

[0094] FIG. 8 is a flow diagram outlining another example of a method for determining a ducking solution. In some examples, the method 800 may be performed by an apparatus such as that shown in FIG. 1A, FIG. 5, or FIG. 6. In some examples, the method 800 may be performed by a control system of an orchestration device, which in some cases may be an audio device. In some examples, the method 800 may be performed, at least in part, by a ducking module, such as the ducking module 400 of FIG. 4, FIG. 5, or FIG. 6. According to some examples, the method 800 may be performed, at least in part, by a renderer. The blocks of the method 800, as well as other methods described herein, are not necessarily performed in the order shown. Additionally, such methods may include more or less blocks than those illustrated and / or described.

[0095] In this example, the process begins at block 805. In some cases, block 805 may correspond to a boot-up process or a time when the boot-up process is complete and a device configured to perform method 800 is ready to function.

[0096] According to this example, block 810 includes estimating a current echo level without application of a ducking solution. In this example, block 810 includes estimating the current echo level based (at least in part) on an acoustic model (represented by block 815) and an AEC model (represented by block 820). Block 810 may include estimating a current echo level resulting from a current ducking candidate solution. The estimated current echo level may be combined with a current speech level in some examples to generate a current estimated SER improvement.

[0097] The acoustic model may be an instance of acoustic model 401 of FIG. 4. The acoustic model may be based at least in part on inter-device audibility, for example, sometimes referred to herein as inter-device echo or inter-audibility. The acoustic model may, in some examples, be based at least in part on in-device echo. In some instances, the acoustic model may be based at least in part on an acoustic model of a user's speech, such as acoustic characteristics of typical human speech, acoustic characteristics of human speech previously detected in the audio environment, etc.

[0098] The AEC model may be an instance of the AEC model 402 of FIG. 4 in some examples. The AEC model may indicate the performance of the EMS 203A of FIG. 5 or FIG. 6. In some examples, the EMS performance model 402 may indicate an actual or expected ERLE (echo return loss enhancement) caused by the operation of the AEC. The ERLE may indicate, for example, the amount of additional signal loss applied by the AEC between each audio device. According to some examples, the EMS performance model 402 may be based at least in part on an expected ERLE for a given number of echo criteria. In some examples, the EMS performance model 402 may be based at least in part on an estimated ERLE calculated from actual microphone and residual signals.

[0099] According to this example, block 825 includes obtaining a current ducking solution (represented by block 850) and estimating the SER based on applying the current ducking solution. In some examples, the ducking solution may be determined as described with reference to block 750 of FIG.

[0100] In this example, block 830 includes calculating the difference or "error" between the current estimate of the SER improvement and the target SER improvement (represented by block 835). In some alternative examples, block 830 may include calculating the difference between the current estimate of the SER and the target SER.

[0101] Block 840, in this example, includes determining whether the difference or "error" calculated in block 830 is sufficiently small. For example, block 840 may include determining whether the difference calculated in block 830 is equal to or less than a threshold. The threshold may be in the range of 0.1 dB to 1.0 dB, such as 0.1 dB, 0.2 dB, 0.3 dB, 0.4 dB, 0.5 dB, 0.6 dB, 0.7 dB, 0.8 dB, 0.9 dB, or 1.0 dB, in some examples. In such examples, if block 840 determines that the difference calculated in block 830 is equal to or less than the threshold, the process ends at block 845. The current ducking solution may be output and / or applied.

[0102] However, if it is determined at block 840 that the difference calculated at block 830 is not small enough (e.g., not equal to or less than a threshold), in this example, the process proceeds to block 855. According to this example, the ducking solution is or includes a ducking vector. In this example, block 855 includes calculating gradients of the cost function and the constraint function with respect to the ducking vector. The cost function may correspond to (or describe) an error between the estimated SER improvement and the target SER improvement, for example, as determined at block 830.

[0103] In some implementations, the constraint function may penalize the influence of the ducking vector on one or more objective functions (e.g., an audio energy preservation function), one or more subjective functions (such as one or more perceptually-based functions), or a combination thereof. In some such examples, one or more of the constraints may be based on a perceptual model of human hearing. According to some examples, one or more of the constraints may be based on audio spatiality preservation.

[0104] In some examples, block 855 may include optimizing a cost that is a function of a model of a perceived spatial location of the audio signal played when played on a set of loudspeakers in the environment and a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker of the set of loudspeakers. In some such examples, the cost may be a function of one or more additional dynamically configurable features. In some such examples, at least one of the one or more additional dynamically configurable features corresponds to echo canceller performance. According to some such examples, at least one of the one or more additional dynamically configurable features corresponds to inter-audibility of loudspeakers in the audio environment. Detailed examples are provided below. However, other implementations may not include these types of cost functions.

[0105] According to this example, block 865 includes updating the current ducking solution using the gradient and one or more types of optimizers, such as the following algorithm, stochastic gradient descent, or another known optimizer.

[0106] In this example, block 870 includes evaluating a change in the ducking solution from a previous ducking solution. According to this example, if at block 870 it is determined that the change in the ducking solution from the previous solution is less than a threshold value, the process ends at block 875. According to some examples, the threshold value may be expressed in decibels. According to some such examples, the threshold value may be in the range of 0.1 dB to 1.0 dB, such as 0.1 dB, 0.2 dB, 0.3 dB, 0.4 dB, 0.5 dB, 0.6 dB, 0.7 dB, 0.8 dB, 0.9 dB, or 1.0 dB. In some examples, if at block 870 it is determined that the change in the ducking solution from the previous solution is less than or equal to the threshold value, the process ends.

[0107] However, in this example, if in block 870 it is determined that the change in the ducking solution from the previous solution is equal to or greater than the threshold, the ducking solution of block 850 is updated to become the current ducking solution and the process returns to block 825. In some examples, method 800 may continue until block 845 or block 875 is reached. According to some examples, method 800 may end if block 845 or block 875 is not reached within a time interval or within some number of iterations.

[0108] After method 800 is completed, the resulting ducking solution may be applied. In some examples, as shown in FIG. 5, the ducking solution may be applied to rendered audio data. In other examples, as shown in FIG. 6, the ducking solution determined via method 800 may be provided to a renderer. The ducking solution may be applied as part of the process of rendering audio data input to the renderer. However, as noted elsewhere herein, in some implementations, method 800 may be performed at least in part by a renderer. According to some such implementations, the renderer may both determine and apply the ducking solution.

[0109] The following algorithm is one example of obtaining a ducking solution. In some examples, the ducking solution may be, include, or indicate a gain to be applied to the rendered audio data. Thus, in some such examples, the ducking solution may be appropriate for ducking module 400, as shown in FIG.

[0110] The following symbols are defined as follows: · A represents the interaudibility matrix (audibility between each audio device); · P represents the nominal (unducked) playback level vector (across the audio devices); D represents a ducking solution vector (across devices), which may correspond to the ducking solution 480 output by the ducking module 400; · C represents the AEC performance matrix, which in this example indicates ERLE (Echo Return Loss Enhancement), which is the amount of additional signal loss applied by the AEC between each audio device.

[0111] The net echo at the microphone feed of audio device i may be expressed as:

number

[0112] In Equation 1, J represents the number of audio devices in the room, and A j i represents the audibility of audio device j with respect to audio device i, and P j Let denote the playback level of audio device j. Then, taking into account the effect of ducking any of the audio devices, the echo in the microphone feed can be expressed as:

number

[0113] In Equation 2, D j denotes the ducking gain. Considering also a naive model of the AEC, where the nominal ERLE is applied independently to each loudspeaker rendering, the echo power in the residual can be expressed as:

number

[0114] In formula 3, C j i represents the ability of audio device i to cancel echo from audio device j. In some examples, Cj i is assumed to be either -20dB or 0dB.

[0115] Generating C can be as trivial as setting an entry to be the nominal cancellation performance value when a particular device is performing cancellation for that particular non-local ("far") device entry in the C matrix. More complex models may take into account audio environment adaptive noise (any noise in the adaptive filter process) and inter-channel correlation. For example, such models may predict how the AEC will perform in the future if a given ducking solution is applied. Some such models may be based at least in part on the echo and noise levels of the audio environment.

[0116] In this example, the distributed ducking problem is formulated as an optimization problem that involves minimizing the total echo power in the AEC residual by varying the ducking vector, which can be expressed as:

number

number

number

number

[0117] Iterative ducking solution In one example, a gradient-based iterative solution to a distributed ducking optimization problem may take the following form:

number

[0118] In Equation 8, F represents the cost function that describes the distributed ducking problem, and D n denotes the ducking vector at the nth iteration. In this form, we constrain D(x) ∈ [0,1]. However, adding a regularization term to Equation 8 does not guarantee that D(x) ∈ [0,1] unless we add some heuristics and / or hard constraints. Another approach involves formulating a gradient-based iterative ducking solution as follows:

number

[0119] In some examples, Z∈[0,1]. However, if one wishes to improve the SER at audio device i by reducing the echo power in the microphone feed of audio device i while also maintaining (at least to some extent) the perceptual quality of the audio content being rendered, it may prove sufficient to ensure that D≧0 and Z≧0. This means allowing some audio devices to increase their full-band volume in order to maintain, at least to some extent, the quality of the rendered content. Thus, in some examples, Z may be defined as follows:

number

[0120] In Equation 10, F represents a cost function that describes the distributed ducking problem, and R is a regularization term that aims to maintain the quality of the rendered audio content. In some examples, R can be an energy conservation constraint. The regularization term R also allows us to find a sensible solution for D.

[0121] Considering T to be the target SER improvement at audio device i by ducking the audio device in the audio environment, F can be defined as:

number

[0122] In formula 11, E res,i n represents the echo in the residual for device i at the nth iteration evaluated using Eq. 3, and E res,i 0 represents the echo in the residual for device i when D is all ones. However, by defining F as shown in Equation 11, the step size becomes a function of the error and not just T. In some examples, F may be reformulated to remove the dependence on the target SER as follows:

number

[0123] It is potentially advantageous to adjust each element of the ducking vector in proportion to the sensitivity of E to each element. Additionally, it is potentially advantageous to allow the user some control over the step size. With these goals in mind, in some instances, F may be reformulated as follows:

number

[0124] In Equation 13, M scales the individual contribution of each device to the echo in the residual for audio device i. In some examples, M may be expressed as:

number

[0125] In Equation 14,

number

number

[0126] In Equation 15, λ represents the Lagrange multiplier. Equation 15 allows the control system to find a simple solution for D. However, in some instances, a ducking solution that maintains an acceptable listening experience and conserves the total energy in the audio environment can be determined by defining the regularization term R as:

number

[0127] Ducking Solver Algorithm With the above discussion in mind, the following ducking solver algorithm can be readily understood: The following algorithm is an example of the method 800 of FIG.

number

[0128] According to some such examples, THRESH may be 6 dB, 8 dB, 10 dB, 12 dB, 14 dB, and so on.

[0129] FIG. 9 is a flow diagram outlining an example of the disclosed method. In some examples, the method 900 may be performed by an apparatus such as that shown in FIG. 1A, FIG. 5, or FIG. 6. In some examples, the method 900 may be performed by a control system of an orchestration device, which in some cases may be an audio device. In some examples, the method 900 may be performed, at least in part, by a ducking module, such as ducking module 400 of FIG. 4, FIG. 5, or FIG. 6. According to some examples, the method 900 may be performed, at least in part, by a renderer. The blocks of the method 900, as well as other methods described herein, are not necessarily performed in the order shown. Additionally, such methods may include more or less blocks than those illustrated and / or described.

[0130] In this example, block 905 includes receiving, by the control system, an output signal from one or more microphones in the audio environment, where the output signal includes a signal corresponding to a current speech of a person. In some instances, the current speech may be or may include a wake word utterance.

[0131] According to this example, block 910 includes determining, by the control system, in response to the output signal, one or more audio processing modifications to apply to audio data being rendered to loudspeaker feed signals for two or more audio devices in the audio environment based at least in part on the audio device location information and the echo management system information. In this example, the audio processing modifications include reducing loudspeaker playback levels for one or more loudspeakers in the audio environment. Thus, the audio processing modifications may include or be indicated by what is referred to herein as a ducking solution. According to some examples, at least one of the one or more types of audio processing modifications may correspond to an increase in a signal-to-echo ratio.

[0132] However, in some examples, audio processing modifications may include or involve modifications other than reducing loudspeaker playback levels. For example, audio processing modifications may include shaping the spectrum of the output of one or more loudspeakers, sometimes referred to herein as "spectral modification" or "spectral shaping." Some such examples may include shaping the spectrum with a substantially linear equalization (EQ) filter designed to produce an output that differs from the spectrum of the audio desired to detect. In some examples, if the output spectrum is being shaped to detect human voices, the filter may reduce frequencies in the range of about 500-3 kHz (e.g., plus or minus 5% or 10% at each end of the frequency range). Some examples may include shaping the loudness to emphasize low and high frequencies and leave space in the mid-band (e.g., in the range of about 500-3 kHz).

[0133] Alternatively or additionally, in some examples, audio processing modifications may include modifying output upper limits or peaks to lower peak levels and / or reduce distortion products that may further degrade the performance of any echo cancellation that is part of the overall system creating the achieved SER for the audio detection, for example via a time domain dynamic range compressor or a multi-band frequency dependent compressor. Such audio signal modifications can effectively reduce the amplitude of the audio signal and can help limit loudspeaker excursion.

[0134] Alternatively or additionally, in some examples, the audio processing modifications may include spatially steering the audio in a manner that tends to reduce the energy or coupling of the output of one or more loudspeakers to one or more microphones that allows for a higher SER by the system (e.g., the audio processing manager). Some such implementations may include the "warping" examples described herein.

[0135] Alternatively or additionally, in some examples, the audio processing modifications may include energy conservation and / or creating continuity in a particular or broad set of listening locations. In some examples, energy removed from one loudspeaker may be compensated for by providing additional energy to another loudspeaker. In some cases, the overall loudness may remain the same, or essentially the same. This is not a required feature, but may be an effective means of allowing more severe changes to the audio processing of the "closest" device, or set of closest devices, without loss of content. However, continuity and / or energy conservation may be particularly important when dealing with complex audio outputs and audio scenes.

[0136] Alternatively or additionally, in some examples, the audio processing changes may include a time constant of activation. For example, changes to the audio processing may be applied a little faster (e.g., 100-200 ms) than they are reverted back to a normal state (e.g., 1000-10000 ms), such that the change(s) in the audio processing appear intentional, if noticeable, but the subsequent reversion from that change(s) may not appear to be related to any actual event or change (from the user's perspective) and in some cases may be so slow as to be barely noticeable.

[0137] In this example, block 915 includes causing the control system to apply one or more types of audio processing modifications. In some examples, such as shown in FIG. 5, the audio processing modifications may be applied to the rendered audio data according to ducking solutions 480 from the ducking module 400. According to some examples, such as shown in FIG. 5, the audio processing modifications may be applied by a renderer. In some such examples, the one or more types of audio processing modifications may include modifying a rendering process to warp the rendering of the audio signal away from the estimated location of the person. However, in some such examples, such audio processing modifications may nevertheless be based at least in part on ducking solutions 480 from the ducking module 400.

[0138] According to some examples, the echo management system information may include a model of echo management system performance. In some examples, the model of echo management system performance may be or may include an acoustic echo canceller (AEC) performance matrix. In some examples, the model of echo management system performance may be or may include a measure of expected echo return loss enhancement provided by the echo management system.

[0139] In some examples, the one or more types of audio processing modifications may be based at least in part on acoustic models of inter-device and intra-device echoes. Alternatively or additionally, in some examples, the one or more types of audio processing modifications may be based at least in part on an interaudibility matrix.

[0140] Alternatively or additionally, in some examples, one or more types of audio processing modifications may be based at least in part on an estimated location of the person. In some examples, the estimated location may correspond to a point, while in other examples, the estimated location may correspond to an area, such as a user zone. According to some such examples, the user zone may be a portion of the audio environment, such as a sofa area, a table area, a chair area, etc. In some examples, the estimated location may correspond to an estimated location of the person's head. According to some examples, the estimated location of the person may be based at least in part on output signals from multiple microphones in the audio environment.

[0141] In some implementations, one or more types of audio processing modifications may be based at least in part on listening intent, which may include, for example, spatial content, frequency content, or both.

[0142] According to some examples, the one or more types of audio processing modifications may be based at least in part on one or more constraints. The one or more constraints may be based on a perceptual model, such as a model of human hearing. Alternatively or additionally, the one or more constraints may include audio content energy preservation, audio spatiality preservation, an audio energy vector, a regularization constraint, or a combination thereof.

[0143] In some examples, the method 900 may include updating an acoustic model of the audio environment, a model of echo management system performance, or both, after applying one or more types of audio processing modifications.

[0144] In some examples, determining one or more types of audio processing modifications may be based at least in part on optimizing a cost function. According to some such examples, the cost function may correspond to or be similar to one of the cost functions of Equations 10-13. Other examples of audio processing modifications based at least in part on optimizing a cost function are described in detail below.

[0145] Audio Device Location Method Examples 9 and elsewhere herein, in some examples, audio processing modifications (such as those corresponding to ducking solutions) may be based at least in part on audio device location information. The locations of the audio devices within the audio environment may be determined or estimated by various methods, including but not limited to those described in the following paragraphs.

[0146] Some such methods may include receiving direct instructions by a user, for example, using a smartphone or tablet device to mark or indicate the approximate location of the device on a floor plan or similar graphical representation of the environment. Such digital interfaces are already common in managing the configuration, grouping, names, purposes, and identities of smart home devices. For example, such direct instructions may be provided via an Amazon Alexa smartphone application, a Sonos S2 controller application, or a similar application.

[0147] Some examples may include using measured signal strengths (sometimes referred to as Received Signal Strength Indicator or RSSI) of common wireless communication technologies such as Bluetooth, Wi-Fi, ZigBee, etc. to solve a basic trilateration problem to generate estimates of the physical distance between devices, for example, as disclosed in J. Yang and Y. Chen, “Indoor Localization Using Improved RSS-Based Lateration Methods,” GLOBECOM 2009 - 2009 IEEE Global Telecommunications Conference, Honolulu, HI, 2009, pp. 1-6, doi: 10.1109 / GLOCOM.2009.5425237, and / or in Mardeni,Mardeni, R. & Othman, Shaifull & Nizam (2010), “Node Positioning in ZigBee Network Using Trilateration Method Based on the Received Signal Strength Indicator (RSSI),” 46, both of which are incorporated herein by reference.

[0148] U.S. Patent No. 10,779,084, entitled "Automatic Discovery and Localization of Speaker Locations in Surround Sound Systems," which is incorporated herein by reference, describes a system that can automatically locate loudspeakers and microphones in a listening environment by acoustically measuring the time of arrival (TOA) between each speaker and microphone.

[0149] In International Publication No. 2021 / 127286 A1 entitled "Audio Device Auto-Location", which is incorporated herein by reference, a method for estimating an audio device location, a listener position and a listener location in an audio environment is disclosed. Some disclosed methods include estimating an audio device location in an environment via direction of arrival (DOA) data and by determining an interior angle for each of a plurality of triangles based on the DOA data. In some examples, each triangle has a vertex corresponding to an audio device location. Some disclosed methods include determining a side length of each side of each of the triangles and performing a forward alignment process to align each of the plurality of triangles to generate a forward alignment matrix. Some disclosed methods include determining to perform a backward alignment process to align each of the plurality of triangles in a reverse sequence to generate a backward alignment matrix. A final estimate of each audio device location may be based at least in part on values ​​of the forward alignment matrix and values ​​of the backward alignment matrix.

[0150] Other disclosed methods of International Publication No. WO 2021 / 127286 A1 include estimating a listener location, and in some cases, including a listener location. Some such methods include prompting a listener to make one or more utterances (e.g., via audio prompts from one or more loudspeakers in the environment) and estimating a listener location according to DOA data. The DOA data may correspond to microphone data acquired by multiple microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, estimating a listener location may include a triangulation process. Some such examples include triangulating the user's voice by finding an intersection between DOA vectors that pass through an audio device. Some disclosed methods of determining a listener's orientation include prompting a user to identify one or more loudspeaker locations. Some such examples include prompting a user to identify one or more loudspeaker locations by moving next to the loudspeaker location(s) and making an utterance. Another example includes prompting a user to identify one or more loudspeaker locations by pointing to each of the one or more loudspeaker locations with a handheld device such as a mobile phone that includes an inertial sensor system and a wireless interface configured to communicate with a control system (such as a control system of an orchestration device) that is controlling audio devices of the audio environment. Some disclosed methods include determining a listener's orientation by having loudspeakers render audio objects such that the audio objects appear to rotate around the listener, and prompting the listener to make an utterance (such as "Stop!") when the listener perceives that the audio object is within a location such as a loudspeaker location, a television location, or the like.Some disclosed methods include determining the location and / or orientation of the listener via the camera data, e.g., by determining a relative location of the listener to one or more audio devices of the audio environment in accordance with the camera data, by determining an orientation of the listener relative to one or more audio devices of the audio environment in accordance with the camera data (e.g., in accordance with the direction the listener is facing), etc.

[0151] Shi, Guangi et al., "Spatial Calibration of Surround Sound Systems including Listener Position Estimation," AES 137, incorporated herein by reference. thConvention, October 2014 describes a system in which a single linear microphone array associated with a playback system component whose location is predictable, such as a soundbar, front center speaker, measures the time difference of arrival (TDOA) of both the satellite loudspeakers and the listener to determine the location of both the loudspeakers and the listener. In this case, the listening orientation is essentially defined as the line connecting the detected listening position and the playback system component that contains the linear microphone array, such as a soundbar collocated with a television (installed directly above or below the television). Since the location of the soundbar is predictably directly above or below the video screen, the geometry of the measured distances and angles of incidence can be converted to absolute positions relative to any point in front of that reference soundbar location using simple trigonometric principles. The distances between the loudspeakers and the microphones of a linear microphone array can be estimated by playing a test signal and measuring the time of flight (TOF) between the radiating loudspeaker and the receiving microphone. For this purpose, the time delay of the direct component of the measured impulse response can be used. The impulse response between the loudspeaker and the microphone array elements can be obtained by playing a test signal through the loudspeaker under analysis. For example, either a maximum length sequence (MLS) or a chirp signal (also known as a logarithmic sine sweep) can be used as the test signal. The impulse response in the room can be obtained by calculating the circular cross-correlation between the captured signal and the MLS input. Figure 2 of this reference shows the echo impulse response obtained using the MLS input. This impulse response is said to be similar to measurements made in a typical office or living room. The delay of the direct component is used to estimate the distance between the loudspeaker and the microphone array elements. For loudspeaker distance estimation, any loopback latency of the audio device used to play the test signal should be calculated and removed from the measured TOF estimate.

[0152] Example of estimating location and orientation of a person in an audio environment The location and orientation of a person in an audio environment may be determined or estimated by a variety of methods, including but not limited to those described in the following paragraphs.

[0153] In "Head-Tracking Techniques for Virtual Acoustic Applications" (AES 133rd Convention, October 2012), Hess, Wolfgang, incorporated herein by reference, presents a number of commercially available techniques for tracking both the position and orientation of a listener's head in the context of a spatial audio playback system. One particular example described is the Microsoft Kinect. Its depth sensing and standard camera, together with publicly available software (Windows Software Development Kit (SDK)), allow for the simultaneous tracking of the head position and orientation of several listeners in space using a combination of skeletal tracking and facial recognition. Kinect for Windows has been discontinued, but the Azure Kinect Developer Kit (DK) is now available, which implements the next generation of Microsoft's depth sensor.

[0154] U.S. Patent No. 10,779,084, entitled "Automatic Discovery and Localization of Speaker Locations in Surround Sound Systems," which is incorporated herein by reference, describes a system that can automatically locate loudspeakers and microphones in a listening environment by acoustically measuring the time of arrival (TOA) between each speaker and microphone. A listening location can be detected by locating and positioning a microphone (e.g., a microphone in a mobile phone held by a listener) at a desired listening position, and an associated listening orientation can be defined by placing another microphone, e.g., on a TV, at a point in the listener's viewing direction. Alternatively, a listening orientation can be defined by positioning a loudspeaker in the viewing direction, e.g., a loudspeaker on a TV.

[0155] In International Publication No. 2021 / 127286 A1 entitled "Audio Device Auto-Location", which is incorporated herein by reference, a method for estimating an audio device location, a listener position and a listener location in an audio environment is disclosed. Some disclosed methods include estimating an audio device location in an environment via direction of arrival (DOA) data and by determining an interior angle for each of a plurality of triangles based on the DOA data. In some examples, each triangle has a vertex corresponding to an audio device location. Some disclosed methods include determining a side length of each side of each of the triangles and performing a forward alignment process to align each of the plurality of triangles to generate a forward alignment matrix. Some disclosed methods include determining to perform a backward alignment process to align each of the plurality of triangles in a reverse sequence to generate a backward alignment matrix. A final estimate of each audio device location may be based at least in part on values ​​of the forward alignment matrix and values ​​of the backward alignment matrix.

[0156] Other disclosed methods of International Publication No. WO 2021 / 127286 A1 include estimating a listener location, and in some cases, including a listener location. Some such methods include prompting a listener to make one or more utterances (e.g., via audio prompts from one or more loudspeakers in the environment) and estimating a listener location according to DOA data. The DOA data may correspond to microphone data acquired by multiple microphones in the environment. The microphone data may correspond to detection of one or more utterances by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, estimating a listener location may include a triangulation process. Some such examples include triangulating the user's voice by finding an intersection between DOA vectors that pass through an audio device. Some disclosed methods of determining a listener's orientation include prompting a user to identify one or more loudspeaker locations. Some such examples include prompting a user to identify one or more loudspeaker locations by moving next to the loudspeaker location(s) and making an utterance. Another example includes prompting a user to identify one or more loudspeaker locations by pointing to each of the one or more loudspeaker locations with a handheld device such as a mobile phone that includes an inertial sensor system and a wireless interface configured to communicate with a control system (such as a control system of an orchestration device) that is controlling audio devices of the audio environment. Some disclosed methods include determining a listener's orientation by having loudspeakers render audio objects such that the audio objects appear to rotate around the listener, and prompting the listener to make an utterance (such as "Stop!") when the listener perceives that the audio object is within a location such as a loudspeaker location, a television location, or the like.Some disclosed methods include determining the location and / or orientation of the listener via the camera data, e.g., by determining a relative location of the listener to one or more audio devices of the audio environment in accordance with the camera data, by determining an orientation of the listener relative to one or more audio devices of the audio environment in accordance with the camera data (e.g., in accordance with the direction the listener is facing), etc.

[0157] Shi, Guangi et al., "Spatial Calibration of Surround Sound Systems including Listener Position Estimation," AES 137, incorporated herein by reference. thConvention, October 2014 describes a system in which a single linear microphone array associated with a playback system component whose location is predictable, such as a soundbar, front center speaker, measures the time difference of arrival (TDOA) of both the satellite loudspeakers and the listener to determine the location of both the loudspeakers and the listener. In this case, the listening orientation is essentially defined as the line connecting the detected listening position and the playback system component that contains the linear microphone array, such as a soundbar collocated with a television (installed directly above or below the television). Since the location of the soundbar is predictably directly above or below the video screen, the geometry of the measured distances and angles of incidence can be converted to absolute positions relative to any point in front of that reference soundbar location using simple trigonometric principles. The distances between the loudspeakers and the microphones of a linear microphone array can be estimated by playing a test signal and measuring the time of flight (TOF) between the radiating loudspeaker and the receiving microphone. For this purpose, the time delay of the direct component of the measured impulse response can be used. The impulse response between the loudspeaker and the microphone array elements can be obtained by playing a test signal through the loudspeaker under analysis. For example, either a maximum length sequence (MLS) or a chirp signal (also known as a logarithmic sine sweep) can be used as the test signal. The impulse response in the room can be obtained by calculating the circular cross-correlation between the captured signal and the MLS input. Figure 2 of this reference shows the echo impulse response obtained using the MLS input. This impulse response is said to be similar to measurements made in a typical office or living room. The delay of the direct component is used to estimate the distance between the loudspeaker and the microphone array elements. For loudspeaker distance estimation, any loopback latency of the audio device used to play the test signal should be calculated and removed from the measured TOF estimate.

[0158] Estimating people's location according to user zone In some examples, the estimated location of a person in an audio environment may correspond to a user zone. This section describes methods for estimating a user zone in which a person is located based at least in part on microphone signals.

[0159] 10 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. The blocks of method 1000, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this implementation, method 1000 includes estimating a location of a user within an environment.

[0160] In this example, block 1005 includes receiving an output signal from each microphone of a plurality of microphones in the environment, where each of the plurality of microphones is present at a microphone location in the environment. According to this example, the output signal corresponds to a current speech of the user. In some examples, the current speech may be or may include a wake word speech. Block 1005 may include, for example, a control system (such as control system 120 of FIG. 1A) receiving an output signal from each microphone of the plurality of microphones in the environment via an interface system (such as interface system 205 of FIG. 1A).

[0161] In some examples, at least some of the microphones in the environment may provide output signals that are asynchronous with respect to output signals provided by one or more other microphones. For example, a first microphone of the plurality of microphones may sample audio data according to a first sample clock, and a second microphone of the plurality of microphones may sample audio data according to a second sample clock. In some instances, at least one of the microphones in the environment may be included within or configured to communicate with a smart audio device.

[0162] According to this example, block 1010 includes determining a plurality of current acoustic features from the output signal of each microphone. In this example, the “current acoustic features” are acoustic features derived from the “current speech” of block 1005. In some implementations, block 1010 may include receiving the plurality of current acoustic features from one or more other devices. For example, block 1010 may include receiving at least some of the plurality of current acoustic features from one or more wake word detectors implemented by the one or more other devices. Alternatively or additionally, in some implementations, block 1010 may include determining the plurality of current acoustic features from the output signal.

[0163] Regardless of whether the acoustic features are determined by a single device or multiple devices, the acoustic features may be determined asynchronously. When the acoustic features are determined by multiple devices, the acoustic features will generally be determined asynchronously unless the devices are configured to coordinate the process of determining the acoustic features. When the acoustic features are determined by a single device, in some implementations, the acoustic features may nevertheless be determined asynchronously because the single device may receive the output signals of each microphone at different times. In some examples, the acoustic features may be determined asynchronously because at least some of the microphones in the environment may provide output signals that are asynchronous with respect to the output signals provided by one or more other microphones.

[0164] In some examples, the acoustic features may include a wake word confidence metric, a wake word duration metric, and / or at least one receive level metric. The receive level metric may indicate a receive level of the sound detected by the microphone and may correspond to a level of the microphone output signal.

[0165] Alternatively or additionally, the acoustic features may include one or more of the following: Average state entropy (purity) for each wake word state along the 1-best (Viterbi) alignment with the acoustic model. Connectionist Temporal Classification Loss (CTC loss) for the acoustic model of the wake word detector. The wake word detector may be trained to provide an estimate of the talker's distance from the microphone and / or an RT60 estimate in addition to the wake word confidence. The distance estimate and / or the RT60 estimate may be acoustic features. Instead of or in addition to the broadband receive level / power at the microphone, the acoustic feature may be the receive level in several log / Mel / Bark-spaced frequency bands, which may vary according to the particular implementation (e.g., 2 frequency bands, 5 frequency bands, 20 frequency bands, 50 frequency bands, 1 octave frequency bands, or 1 / 3 octave frequency bands). · Cepstral representation of the spectral information at preceding points, calculated by taking the DCT (Discrete Cosine Transform) of the logarithm of the band powers. Band power in frequency bands weighted for human speech. For example, the acoustic features may be based only on a specific frequency band (e.g., 400 Hz to 1.5 kHz). In this example, higher and lower frequencies may be ignored. Confidence of the voice activity detector per band or per bin. The acoustic signature may be based at least in part on a long-term noise estimate so as to ignore microphones with insufficient signal-to-noise ratio. Kurtosis as a measure of the "peakiness" of speech. Kurtosis can be an indication of smearing due to long reverberation tails. Estimated onset time of the wake word. Onset and duration are expected to be equal across all microphones to within a frame or so. Outliers may give clues to unreliable estimates. This assumes a certain level of synchrony, not necessarily across samples, but across frames of e.g. tens of milliseconds.

[0166] According to this example, block 1015 includes applying a classifier to the plurality of current acoustic features. In some such examples, applying the classifier may include applying a trained model to previously determined acoustic features derived from a plurality of previous utterances made by the user in a plurality of user zones in the environment. Various examples are provided herein.

[0167] In some examples, the user zones may include a sink area, a food preparation area, a refrigerator area, a dining area, a sofa area, a television area, a sleeping area, and / or an entryway area. According to some examples, one or more of the user zones may be pre-defined user zones. In some such examples, the one or more pre-defined user zones may have been selected by a user during a training process.

[0168] In some implementations, applying the classifier may include applying a Gaussian mixture model trained on the previous utterance. According to some such implementations, applying the classifier may include applying a Gaussian mixture model trained on one or more of a normalized wake word confidence, a normalized average received level, or a maximum received level of the previous utterance. However, in alternative implementations, applying the classifier may be based on a different model, such as one of the other models disclosed herein. In some instances, the model may be trained using training data labeled with user zones. However, in some examples, applying the classifier includes applying a model trained using unlabeled training data that is not labeled with user zones.

[0169] In some examples, the previous utterance may have been or may have included an utterance of the wake word, and in some such examples, the previous utterance and the current utterance may be utterances of the same wake word.

[0170] In this example, block 1020 includes determining an estimate of a user zone in which the user is currently located based at least in part on the output from the classifier. In some such examples, the estimate may be determined without reference to the geometric locations of the multiple microphones. For example, the estimate may be determined without reference to coordinates of individual microphones. In some examples, the estimate may be determined without estimating the geometric location of the user.

[0171] Some implementations of the method 1000 may include selecting at least one speaker according to the estimated user zone. Some such implementations may include controlling the at least one selected speaker to provide sound to the estimated user zone. Alternatively or additionally, some implementations of the method 1000 may include selecting at least one microphone according to the estimated user zone. Some such implementations may include providing a signal output by the at least one selected microphone to a smart audio device.

[0172] 11 is a block diagram of elements of an example of an embodiment configured to implement a zone classifier. According to this example, a system 1100 includes multiple loudspeakers 1104 distributed in at least a portion of an environment (e.g., an environment as shown in FIG. 1A or FIG. 1B). In this example, the system 1100 includes a multi-channel loudspeaker renderer 1101. According to this implementation, the output of the multi-channel loudspeaker renderer 1101 serves as both a loudspeaker driving signal (speaker feed for driving the speaker 1104) and an echo reference. In this implementation, the echo reference is provided to an echo management subsystem 1103 via multiple loudspeaker reference channels 1102 that include at least some of the speaker feed signals output from the renderer 1101.

[0173] In this implementation, the system 1100 includes multiple echo management subsystems 1103. According to this example, the echo management subsystems 1103 are configured to implement one or more echo suppression processes and / or one or more echo cancellation processes. In this example, each of the echo management subsystems 1103 provides a corresponding echo management output 1103A to one of the wake word detectors 1106. The echo management output 1103A has an attenuated echo relative to the input to an associated one of the echo management subsystems 1103.

[0174] According to this implementation, the system 1100 includes N microphones 1105 (N is an integer) distributed in at least a portion of an environment (e.g., the environment shown in FIG. 1A or FIG. 1B). The microphones may include array microphones and / or spot microphones. For example, one or more smart audio devices located in the environment may include an array of microphones. In this example, the output of the microphones 1105 is provided as an input to the echo management subsystem 1103. According to this implementation, each of the echo management subsystems 1103 captures the output of an individual microphone 1105 or an individual group or subset of the microphones 1105.

[0175] In this example, the system 1100 includes multiple wake word detectors 1106. According to this example, each of the wake word detectors 1106 receives audio output from one of the echo management subsystems 1103 and outputs multiple acoustic features 1106A. The acoustic features 1106A output from each echo management subsystem 1103 may include (but are not limited to) wake word confidence, wake word duration, and receive level measurements. Although three arrows depicting three acoustic features 1106A are shown as being output from each echo management subsystem 1103, more or fewer acoustic features 1106A may be output in alternative implementations. Additionally, although the three arrows impinge on the classifier 1107 along a substantially vertical line, this does not indicate that the classifier 1107 necessarily receives acoustic features 1106A from all of the wake word detectors 1106 at the same time. As noted elsewhere herein, the acoustic features 1106A may, in some cases, be determined and / or provided to the classifier asynchronously.

[0176] According to this implementation, the system 1100 includes a zone classifier 1107, sometimes referred to as a classifier 1107. In this example, the classifier receives multiple features 1106A from multiple wake word detectors 1106 for multiple (e.g., all) microphones 1105 in the environment. According to this example, an output 1108 of the zone classifier 1107 corresponds to an estimate of a user zone in which the user is currently located. According to some such examples, the output 1108 may correspond to one or more posterior probabilities. The estimate of the user zone in which the user is currently located may be or may correspond to a maximum posterior probability according to Bayesian statistics.

[0177] Next, an exemplary implementation of a classifier that may correspond, in some examples, to the zone classifier 1107 of FIG. i Let (n) be the i-th microphone signal at discrete time n (i={1...N}) (i.e., microphone signal x i(n) are the outputs of the N microphones 1105. The N signals x i The processing of (n) produces a “clean” microphone signal e at each discrete time n. i (n), where i={1...N}. The clean signal e, designated 1103A in FIG. i (n), in this example, are fed to a wake word detector 1106. Here, each wake word detector 1106 receives a vector of features w i (j), where j={1...J} is the index corresponding to the jth wake word utterance. In this example, the classifier 1107 takes as input the following aggregate feature set:

number

[0178] According to some implementations, a set of zone labels C k (k={1...K}) may correspond to the number K of different user zones in the environment. For example, the user zones may include a sofa zone, a kitchen zone, a reading chair zone, etc. Some examples may define more than one zone in a kitchen or other room. For example, a kitchen area may include a sink zone, a food preparation zone, a refrigerator zone, and a dining zone. Similarly, a living room area may include a sofa zone, a TV zone, a reading chair zone, one or more entryway zones, etc. Zone labels for these zones may be selectable by the user, for example, during a training phase.

[0179] In some implementations, the classifier 1107 determines the posterior probability p(C k |W(j)) with probability p(C k |W(j)) is the user in zone C k Probability of being in each of the zones (Zone C k, for the jth utterance and the kth zone for each of the utterances) and is an example of the output 1108 of the classifier 1107.

[0180] According to some examples, training data may be collected (e.g., for each user zone) by prompting a user to select or define a zone, e.g., a sofa zone. The training process may include prompting the user to make a training utterance, such as a wake word, near the selected or defined zone. In the sofa zone example, the training process may include prompting the user to make a training utterance in the center and at the very end of the sofa. The training process may include prompting the user to repeat the training utterance several times at each location within the user zone. The user may then be prompted to move to another user zone and continue until all designated user zones are covered.

[0181] 12 is a flow diagram outlining an example of a method that may be performed by an apparatus such as apparatus 200 of FIG. 1A. The blocks of method 1200, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this implementation, method 1200 includes training a classifier to estimate a location of a user within an environment.

[0182] In this example, block 1205 includes prompting the user to perform at least one training utterance at each of a plurality of locations within a first user zone of the environment. The training utterance(s) may, in some examples, be one or more instances of a wake word utterance. According to some implementations, the first user zone may be any user zone selected and / or defined by the user. In some instances, the control system may generate a corresponding zone label (e.g., zone label C described above) for the first user zone. k) and may associate the zone label with the training data obtained for the first user zone.

[0183] To collect these training data, an automated prompt system may be used. As described above, the interface system 205 of the device 200 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. For example, the device 200 may provide the following prompts to the user on the screen of the display system or hear them announced via one or more speakers during the training process: · "Please move to the couch." "Say the wake word 10 times while moving your head." "Please move halfway between the sofa and the reading chair and say the wake word 10 times." · "Stand in the kitchen as if you were cooking and say the wake word 10 times."

[0184] In this example, block 1210 includes receiving a first output signal from each of a plurality of microphones in the environment. In some examples, block 1210 may include receiving the first output signal from all of the active microphones in the environment, while in other examples, block 1210 may include receiving the first output signal from a subset of all of the active microphones in the environment. In some examples, at least some of the microphones in the environment may provide output signals that are asynchronous with respect to the output signals provided by one or more other microphones. For example, a first microphone of the plurality of microphones may sample audio data according to a first sample clock, and a second microphone of the plurality of microphones may sample audio data according to a second sample clock.

[0185] In this example, each microphone of the plurality of microphones is present at a microphone location of the environment. In this example, the first output signal corresponds to an instance of a detected training utterance received from a first user zone. Because block 1205 includes prompting a user to make at least one training utterance at each of a plurality of locations in the first user zone of the environment, in this example, the term "first output signal" refers to a set of all output signals corresponding to the training utterances for the first user zone. In other examples, the term "first output signal" may refer to a subset of all output signals corresponding to the training utterances for the first user zone.

[0186] According to this example, block 1215 includes determining one or more first acoustic features from each of the first output signals. In some examples, the first acoustic features may include a wake word confidence metric and / or a received level metric. For example, the first acoustic features may include a normalized wake word confidence metric, an indication of a normalized average received level, and / or an indication of a maximum received level.

[0187] As described above, since block 1205 includes prompting the user to make at least one training utterance at each of a plurality of locations in a first user zone of the environment, in this example, the term "first output signals" refers to the set of all output signals corresponding to the training utterances for the first user zone. Thus, in this example, the term "first acoustic features" refers to a set of acoustic features derived from the set of all output signals corresponding to the training utterances for the first user zone. Thus, in this example, the set of first acoustic features is at least as large as the set of first output signals. For example, if two acoustic features are determined from each of the output signals, the set of first acoustic features will be twice as large as the set of first output signals.

[0188] In this example, block 1220 includes training a classifier model to make a correlation between the first user zone and the first acoustic feature. The classifier model may be, for example, any of those disclosed herein. According to this implementation, the classifier model is trained without reference to the geometric locations of the multiple microphones. In other words, in this example, data regarding the geometric locations of the multiple microphones (e.g., microphone coordinate data) is not provided to the classifier model during the training process.

[0189] 13 is a flow diagram outlining another example of a method that may be performed by an apparatus such as the apparatus 200 of FIG. 1A. The blocks of the method 1300, as well as other methods described herein, are not necessarily performed in the order shown. For example, in some implementations, at least a portion of the acoustic feature determination process of block 1325 may be performed before block 1315 or block 1320. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this implementation, the method 1300 includes training a classifier to estimate a location of a user within an environment. The method 1300 provides an example of extending the method 1200 to multiple user zones of an environment.

[0190] In this example, block 1305 includes prompting the user to perform at least one training utterance at a location within a user zone of the environment. In some instances, block 1305 may be performed in the manner described above with reference to block 1205 of FIG. 12, except that block 1305 pertains to a single location within the user zone. The training utterance(s) may be one or more instances of a wake word utterance in some examples. According to some implementations, the user zone may be any user zone selected and / or defined by a user. In some instances, the control system may generate a corresponding zone label (e.g., zone label C described above). k , a corresponding instance of a zone label of one of the zone labels, and may associate the zone label with the training data obtained for the user zone.

[0191] According to this example, block 1310 is performed substantially as described above with reference to block 1210 of FIG. 12. However, in this example, the process of block 1310 is generalized to any user zone, not necessarily the first user zone for which training data is obtained. Thus, the output signals received in block 1310 are "output signals from each of a plurality of microphones in the environment, each of the plurality of microphones being at a microphone location in the environment, the output signals corresponding to instances of detected training utterances received from the user zone." In this example, the term "output signals" refers to the set of all output signals corresponding to one or more training utterances at a location of the user zone. In other examples, the term "output signals" may refer to a subset of all output signals corresponding to one or more training utterances at a location of the user zone.

[0192] According to this example, block 1315 includes determining whether sufficient training data has been obtained for the current user zone. In some such examples, block 1315 may include determining whether output signals corresponding to a threshold number of training utterances have been obtained for the current user zone. Alternatively or additionally, block 1315 may include determining whether output signals corresponding to training utterances in a threshold number of locations within the current user zone have been obtained. If not, method 1300 returns to block 1305 in this example and the user is prompted to make at least one additional utterance at a location within the same user zone.

[0193] However, if at block 1315 it is determined that sufficient training data has been obtained for the current user zone, then in this example the process proceeds to block 1320. According to this example, block 1320 includes determining whether to obtain training data for additional user zones. According to some examples, block 1320 may include determining whether training data has been obtained for each user zone previously identified by the user. In other examples, block 1320 may include determining whether training data has been obtained for a minimum number of user zones. The minimum number may have been selected by the user. In other examples, the minimum number may be a recommended minimum number per environment, a recommended minimum number per room of the environment, or the like.

[0194] If at block 1320 it is determined that training data should be obtained for additional user zones, in this example, the process continues to block 1322 which includes prompting the user to move to another user zone of the environment. In some examples, the next user zone may be selectable by the user. According to this example, the process continues to block 1305 after the prompt of block 1322. In some such examples, the user may be prompted to confirm that the user has arrived at the new user zone after the prompt of block 1322. According to some such examples, the user may be required to confirm that the user has arrived at the new user zone before the prompt of block 1305 is provided.

[0195] If at block 1320 it is determined that training data should not be obtained for additional user zones, in this example, the process proceeds to block 1325. In this example, the method 1300 includes obtaining training data for K user zones. In this implementation, block 1325 includes determining first through Gth acoustic features from the first through Hth output signals corresponding to each of the first through Kth user zones for which training data was obtained. In this example, the term "first output signal" refers to the set of all output signals corresponding to training utterances for the first user zone, and the term "Hth output signal" refers to the set of all output signals corresponding to training utterances for the Kth user zone. Similarly, the term "first acoustic feature" refers to the set of acoustic features determined from the first output signal, and the term "Gth acoustic feature" refers to the set of acoustic features determined from the Hth output signal.

[0196] According to these examples, block 1330 includes training a classifier model to make correlations between the first through Kth user zones and the first through Kth acoustic features, respectively. The classifier model may be, for example, any of the classifier models disclosed herein.

[0197] In the previous example, the user zone (e.g., the zone label C explained above) k , ...

number

[0198] Training the classifier model may include determining a best fit to the labeled training data. Without loss of generality, suitable classification techniques for the classifier model may include: Bayesian classifiers with per-class distributions described, for example, by multivariate normal distributions, full-covariance Gaussian mixture models, or diagonal-covariance Gaussian mixture models; Vector quantization; · Nearest neighbor (k-means); A neural network with one output for each class but with a SoftMax output layer; Support Vector Machine (SVM); and / or Boosting techniques such as Gradient Boosting Machines (GBM).

[0199] In one example of implementing the unlabeled case, the data may be automatically partitioned into K clusters, where K may also be unknown. The unlabeled automatic partitioning may be performed, for example, by using classical clustering techniques, such as the k-means algorithm or Gaussian mixture modeling.

[0200] To improve robustness, regularization may be applied to the classifier model training, and model parameters may be updated over time as new utterances are presented.

[0201] Further aspects of the embodiment will now be described.

[0202] An exemplary set of acoustic features (e.g., acoustic features 1106A in FIG. 11) may include a likelihood of wake word confidence, an average received level over the estimated duration of the most confident wake word, and a maximum received level over the duration of the most confident wake word. The features may be normalized to their maximum value for each wake word utterance. The training data may be labeled, and a full covariance Gaussian mixture model (GMM) may be trained to maximize the expectation of the training labels. The estimated zone may be the class that maximizes the posterior probability.

[0203] The above description of some embodiments discusses learning an acoustic zone model from a set of training data collected during a prompted collection process. In that model, training time (or configuration mode) and run-time (or normal mode) can be considered as two separate modes in which a microphone system may be placed. An extension of this scheme is online learning, where some or all of the acoustic zone model is learned or adapted online (e.g., at run-time or in normal mode). In other words, in some implementations, the process of training the classifier may continue even after the classifier has been applied in a "run-time" process to make an estimate of the user zone in which the user is currently located (e.g., according to method 1000 of FIG. 10).

[0204] 14 is a flow diagram outlining another example of a method that may be performed by an apparatus such as apparatus 200 of FIG. 1A. The blocks of method 1400, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this implementation, method 1400 includes ongoing training of a classifier during a "run-time" process that estimates a user's location within an environment. Method 1400 is an example of what is referred to herein as an online learning mode.

[0205] In this example, block 1405 of method 1400 corresponds to blocks 1005-1020 of method 1000. Here, block 1405 includes providing an estimate of a user zone in which the user is currently located based at least in part on the output from the classifier. According to this implementation, block 1410 includes obtaining implicit or explicit feedback regarding the estimate of block 1405. In block 1415, the classifier is updated according to the feedback received in block 1405. Block 1415 may include, for example, one or more reinforcement learning methods. As suggested by the dashed arrow from block 1415 to block 1405, in some implementations, method 1400 may include returning to block 1405. For example, method 1400 may include providing a future estimate of a user zone in which the user will be located at a future time based on applying the updated model.

[0206] Explicit techniques for obtaining feedback may include: Use a voice user interface (UI) to ask the user if the prediction was correct. For example, a sound may be provided to the user indicating: "I think you are sitting on the couch, please say 'right' or 'wrong'." Use a voice UI to inform the user that they can always correct an incorrect prediction (e.g., a sound could be provided to the user indicating: "I can now predict where you will be when you talk to me. If the prediction is wrong, say something like: 'Amanda, I'm not sitting on the couch. I'm sitting in a reading chair.'" Using a voice UI to inform the user that a correct prediction can always be rewarded (e.g., a sound could be provided to the user indicating: "I'm now able to predict where you will be when you talk to me. If my prediction is correct, please help me improve my predictions even further by saying something like, 'Amanda, that's right. I'm sitting on the couch.'"). Includes physical buttons or other UI elements that the user can interact with to provide feedback (e.g., thumbs up and / or thumbs down buttons on a physical device or within a smartphone app).

[0207] The goal of predicting the user zone in which the user is located may be to inform microphone selection or adaptive beamforming schemes that attempt to more effectively pick up sounds from the user's acoustic zone, for example, to better recognize commands following a wake word. In such a scenario, implicit techniques for obtaining feedback on the quality of the zone prediction may include: Penalizing predictions that result in misrecognition of the command following the wake word. Proxies that may indicate misrecognition may include a user interrupting the voice assistant's response to a command by, for example, speaking a counter-command such as "Amanda, stop!"; Penalizing predictions that have low confidence that the speech recognizer successfully recognized the command. Many automatic speech recognition systems have the ability to return a confidence level along with their results that can be used for this purpose; penalizing predictions that result in the second pass wake word detector being unable to retroactively detect the wake word with high confidence; and / or Enhancing predictions that result in highly confident recognition of wake words and / or correct recognition of user commands.

[0208] The following is an example where a second pass wake word detector cannot retroactively detect the wake word with high confidence. Assume that after obtaining output signals corresponding to the current utterance from microphones in the environment and after determining acoustic features based on the output signals (e.g., via multiple first pass wake word detectors configured to communicate with the microphones), the acoustic features are provided to a classifier. In other words, the acoustic features are presumed to correspond to the detected wake word utterance. Further assume that the classifier determines that the person who made the current utterance is most likely in Zone 3, which in this example corresponds to a reading chair. For example, there may be a learned combination of microphones, for example, that are known to be best at hearing a person's voice when the person is in Zone 3, to send to a cloud-based virtual assistant service for voice command recognition.

[0209] Further assume that after determining which microphone(s) will be used for voice recognition, but before the person's voice is actually sent to the virtual assistant service, a second-pass wake word detector operates on the microphone signal corresponding to the voice detected by the selected microphone(s) for Zone 3 that is about to be submitted for command recognition. If that second-pass wake word detector does not match the first-pass wake word detectors where the wake word actually was uttered, it is likely because the classifier mispredicted the zone. Therefore, this classifier should be penalized.

[0210] Techniques for post-updating the zone mapping model after one or more wake words have been spoken may include the following. Maximum a posteriori (MAP) adaptation of Gaussian mixture models (GMM) or nearest neighbor models; and / or Reinforcement learning, for example of neural networks, by associating the appropriate “one-hot” (for correct predictions) or “one-cold” (for incorrect predictions) ground truth labels with the SoftMax output and applying online backpropagation to determine the new network weights.

[0211] Some examples of MAP adaptation in this context may include adjusting the mean in the GMM each time the wake word is spoken. In this way, the mean may more closely match the acoustic features observed when the subsequent wake word is spoken. Alternatively or additionally, such examples may include adjusting the variance / covariance or mixture weight information in the GMM each time the wake word is spoken.

[0212] For example, a MAP adaptation scheme may be as follows:

number

[0213] In the above formula, μ i,old Let x denote the mean of the i-th Gaussian in the mixture, α denotes a parameter that controls how aggressive the MAP adaptation should be (α can be in the range [0.9,0.999]), and x denotes the feature vector of the new wake word utterance. The index "i" corresponds to the mixture element that returns the highest a priori probability of containing the speaker's location at the time of the wake word.

[0214] Alternatively, each of the mixture components may be adjusted according to their a priori probability of containing the wake word, for example as follows:

number

[0215] In the above formula, β i =α*(1-P(i)), where P(i) represents the a priori probability that observation x is due to mixture element i.

[0216] In one reinforcement learning example, there may be three user zones. Suppose for a particular wake word, the model predicts the probabilities for the three user zones to be [0.2, 0.1, 0.7]. If a second source (e.g., a second-pass wake word detector) confirms that zone 3 was correct, then the ground truth label may be [0, 0, 1] ("one hot"). Posterior updates to the zone mapping model may involve backpropagating the error through the neural network, effectively meaning that the neural network will predict zone 3 more strongly if the same input is presented again. Conversely, if the second source indicates that zone 3 was an inaccurate prediction, then the ground truth label may be [0.5, 0.5, 0.0] in one example. Backpropagating the error through the neural network will make the model less likely to predict zone 3 if the same input is presented in the future.

[0217] Further examples of audio processing modifications including cost function optimization As noted elsewhere herein, in various disclosed examples, one or more types of audio processing modifications may be based on optimizing a cost function. Some such examples include flexible rendering.

[0218] Flexible rendering allows spatial audio to be rendered across any number of arbitrarily positioned speakers. Given the widespread deployment of audio devices, including but not limited to smart audio devices (e.g., smart speakers) in the home, there is a need to implement flexible rendering techniques that enable consumer products to perform flexible rendering of audio and playback of the audio so rendered.

[0219] Several techniques have been developed to implement flexible rendering. They cast the rendering problem as one of cost function minimization, which consists of two terms: a first term that models the desired spatial impression the renderer is trying to achieve, and a second term that assigns costs to speaker activations. Until now, this second term has focused on creating a sparse solution, where only speakers in close proximity to the desired spatial location of the audio being rendered are activated.

[0220] Spatial audio playback in consumer environments is typically tied to a given number of loudspeakers arranged in given locations, e.g., 5.1 and 7.1 surround sound. In these cases, content is specifically authored for the relevant loudspeakers and encoded as one discrete channel per loudspeaker (e.g., Dolby Digital, or Dolby Digital Plus, etc.). More recently, immersive object-based spatial audio formats (Dolby Atmos) have been introduced that break this association between content and specific loudspeaker locations. Instead, content may be described as a collection of individual audio objects, each with potentially time-varying metadata that describes the desired perceived location of the audio object in three-dimensional space. At playback, the content is converted to loudspeaker feeds by a renderer that adapts to the number and location of the loudspeakers in the playback system. However, many such renderers still constrain the location of a set of loudspeakers to be one of a set of pre-defined layouts (e.g., for Dolby Atmos, 3.1.2, 5.1.2, 7.1.4, 9.1.6, etc.).

[0221] Going beyond such constrained rendering, methods have been developed that allow object-based audio to be flexibly rendered across a truly arbitrary number of loudspeakers placed at arbitrary locations. These methods require the renderer to have knowledge of the number and physical locations of the loudspeakers in the listening space. For such systems to be practical for the average consumer, an automated method for locating the loudspeakers is desirable. One such method relies on the use of a number of microphones, possibly co-located with the loudspeakers. By playing audio signals through the loudspeakers and recording with the microphones, the distance between each loudspeaker and the microphone is estimated. From these distances, the locations of both the loudspeakers and the microphones are then estimated.

[0222] Concurrent with the introduction of object-based spatial audio in consumer spaces has been the rapid adoption of so-called "smart speakers," such as the Amazon Echo product line. While the immense popularity of these devices can be attributed to the simplicity and convenience offered by their wireless connectivity and integrated voice interface (e.g., Amazon's Alexa), the acoustic capabilities of these devices are generally limited, especially with respect to spatial audio. In most cases, these devices are constrained to mono or stereo playback. However, combining the flexible rendering and automatic localization techniques described above with multiple orchestrated smart speakers can result in a system with very sophisticated spatial playback capabilities that still remains extremely simple for consumers to set up. Wireless connectivity allows consumers to place as many speakers as they need wherever it is convenient, without the need to run speaker wires, and the built-in microphones can be used to automatically localize the speakers for the associated flexible renderer.

[0223] Conventional flexible rendering algorithms are designed to achieve as closely as possible a particular desired perceived spatial impression. In an orchestrated system of smart speakers, sometimes maintaining this spatial impression may not be the most important or desired objective. For example, if someone is trying to simultaneously speak to an integrated voice assistant, it may be desired to momentarily modify the spatial rendering in a way that reduces the relative playback level on speakers near a particular microphone in order to increase the signal-to-noise ratio and / or signal-to-echo ratio (SER) of the microphone signal containing the detected voice. Some embodiments described herein may be implemented as modifications to existing flexible rendering methods to enable such dynamic modifications to the spatial rendering, for example, for the purpose of achieving one or more additional objectives.

[0224] Existing flexible rendering techniques include Center of Mass Amplitude Panning (CMAP) and Flexible Virtualization (FV). Broadly speaking, both of these techniques render a set of one or more audio signals, each with an associated desired perceptual spatial location, for playback on a set of two or more loudspeakers, with the relative activation of the loudspeakers of the set being a function of a model of the perceptual spatial location of said audio signals to be played on the loudspeakers and the proximity of the desired perceptual spatial location of the audio signals to the positions of the loudspeakers. The model ensures that the audio signals are heard by the listener near their intended spatial location, and a proximity term controls which speakers are used to achieve this spatial impression. In particular, the proximity term favors the activation of loudspeakers that are near the desired perceptual spatial location of the audio signals. For both CMAP and FV, this functional relationship is conveniently derived from the following cost function, written as the sum of two terms, one for spatial aspect and one for proximity:

number

number

number

number

[0225] In a particular definition of the cost function, g opts Although the relative levels between the components of are appropriate, it is difficult to control the absolute levels of optimal activation resulting from the above minimization. To address this issue, opt Subsequent normalization of x may be performed to control the absolute level of activation. For example, it may be desirable to normalize the vectors to have unit length, which follows the commonly used low-power panning rule:

number

[0226] The exact behavior of the flexible rendering algorithm depends on two terms in the cost function C spatial and C proximity It is determined by the specific construction of the CMAP. spatial represents the perceived spatial location of audio signals reproduced from a set of loudspeakers in terms of their associated activation gains gi The model is derived by placing the loudspeakers at the center of mass of their positions weighted by (elements of vector g):

number

number

number

number

number

number

number

number

number

number

[0227] For this purpose, the second term of the cost function, C proximity can be defined as the distance-weighted sum of the absolute squared speaker activations, which can be compactly expressed in matrix form as follows:

number

number

number

number

[0228] Combining the two terms of the cost function defined in equations 8 and 9a gives the overall cost function:

number

[0229] Setting the derivative of this cost function with respect to g equal to zero and solving for g gives the optimal speaker activation solution:

number

[0230] In general, an optimal solution of Equation 27 may result in speaker activations with negative values. For the CMAP construction of the flexible renderer, such negative activations may be undesirable, so Equation 27 may be minimized subject to the condition that all activations remain positive.

[0231] 15 and 16 are diagrams illustrating an example set of speaker activation and object rendering positions. In these examples, the speaker activation and object rendering positions correspond to speaker positions of 4, 64, 165, -87, and -4 degrees. FIG. 15 shows speaker activations 1505a, 1510a, 1515a, 1520a, and 1525a, which contain the optimal solution of Equation 11 for these particular speaker positions. FIG. 16 plots the individual speaker positions as dots 1605, 1610, 1615, 1620, and 1625, which correspond to speaker activations 1505a, 1510a, 1515a, 1520a, and 1525a, respectively. FIG. 16 also shows ideal object positions (in other words, the positions where the audio objects should be rendered) for a number of possible object angles as dots 1630a, and the corresponding actual rendering positions for those objects as dots 1635a connected to the ideal object positions by dotted lines 1640a.

[0232] One class of embodiments includes a method for rendering audio for playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) smart audio devices. For example, a set of smart audio devices present in a user's home (in a system) may be orchestrated to handle a variety of simultaneous use cases, including (in one embodiment) flexible rendering of audio for playback by all or some of the smart audio devices (i.e., by speaker(s) of all or some of the smart audio devices). Many interactions with the system are possible that require dynamic modifications to the rendering. Such modifications may, but are not necessarily, focused on spatial fidelity.

[0233] Some embodiments are methods for rendering audio for playback by at least one (e.g., all or some) of a set of smart audio devices (or for playback by at least one (e.g., all or some) of the speakers of another set of speakers). The rendering may include minimization of a cost function, the cost function including at least one dynamic speaker activation term. Examples of such dynamic speaker activation terms include (but are not limited to): · The proximity of the speaker to one or more listeners; · Proximity of the speaker to attractive or repulsive forces; · Audibility of the loudspeakers relative to a certain location (e.g. the listener's position or the baby's room); · Speaker capabilities (e.g. frequency response and distortion); · Synchronicity of speakers relative to other speakers; Wake word performance; Echo cancellation performance.

[0234] The dynamic speaker activation term(s) may enable at least one of a variety of behaviors, including warping the spatial presentation of audio away from a particular smart audio device so that its microphone can better hear the talker, or so that the smart audio device's speaker(s) can better hear a secondary audio stream.

[0235] Some embodiments implement the rendering for playback over a coordinated (orchestrated) speaker(s) of multiple smart audio devices, while other embodiments implement the rendering for playback over a speaker(s) of a separate set of speakers.

[0236] Pairing the flexible rendering method (implemented according to some embodiments) with a set of wireless smart speakers (or other smart audio devices) can result in an extremely capable and easy-to-use spatial audio rendering system. Considering the interaction with such a system, it becomes apparent that dynamic modifications to the spatial rendering may be desirable in order to optimize for other objectives that may arise during the use of the system. To achieve this goal, one class of embodiments augments the existing flexible rendering algorithm (where speaker activation is a function of the spatial and proximity terms disclosed previously) with one or more additional dynamically configurable features that depend on one or more properties of the audio signal being rendered, the set of speakers, and / or other external inputs. According to some embodiments, the existing flexible rendering cost function given in Equation 1 is augmented with these one or more additional dependencies according to the following equation:

number

[0237] In Equation 28, the term

number

number

number

number

number

number

number

number

[0238]

number

[0239]

number

[0240]

number

[0241] Using the new cost function defined in Equation 28, the optimal set of activations can be found by minimization with respect to g and possible a posteriori regularization as specified previously in Equations 28a and 28b.

[0242] In some cases, the cost term C j by a ducking module, such as ducking module 400 of FIG.

number

[0243] FIG. 17 is a flow diagram outlining one example of a method that may be performed by an apparatus or system such as that shown in FIG. 1A. The blocks of method 1700, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. The blocks of method 1700 may be performed by one or more devices that may be (or may include) a control system, such as control system 160 shown in FIG. 1A.

[0244] In this implementation, block 1705 includes receiving audio data by the control system via the interface system. In this example, the audio data includes one or more audio signals and associated spatial data. According to this implementation, the spatial data indicates an intended perceived spatial location corresponding to the audio signal. In some cases, the intended perceived spatial location may be explicit, as indicated by position metadata, such as Dolby Atmos position metadata. In other examples, the intended perceived spatial location may be implicit, for example, the intended perceived spatial location may be an assumed location associated with a channel according to Dolby 5.1, Dolby 7.1, or another channel-based audio format. In some examples, block 1705 includes a rendering module of the control system receiving the audio data via the interface system.

[0245] According to this example, block 1710 includes rendering, by the control system, the audio data for playback through a set of loudspeakers of the environment to generate a rendered audio signal. In this example, rendering each of the one or more audio signals included in the audio data includes determining a relative activation of a set of loudspeakers in the environment by optimizing a cost function. According to this example, the cost is a function of a model of a perceived spatial location of the audio signal when played on the set of loudspeakers in the environment. In this example, the cost is also a function of a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker of the set of loudspeakers. In this implementation, the cost is also a function of one or more additional dynamically configurable features. In this example, the dynamically configurable features are based on one or more of the following: the proximity of a loudspeaker to one or more listeners, the proximity of a loudspeaker to an attractive force location (where attractive force is a factor that favors relatively higher loudspeaker activation closer to an attractive force location), the proximity of a loudspeaker to a repulsive force location (where repulsive force is a factor that favors relatively lower loudspeaker activation closer to a repulsive force location), the performance of each loudspeaker relative to other loudspeakers in the environment, the synchronicity of the loudspeaker relative to other loudspeakers, wake word performance, or echo canceller performance.

[0246] In this example, block 1715 includes providing the rendered audio signals via an interface system to at least some of the loudspeakers of the set of loudspeakers of the environment.

[0247] According to some examples, the model of perceptual spatial location may generate binaural responses corresponding to audio object positions at the left and right ears of a listener. Alternatively or additionally, the model of perceptual spatial location may place the perceptual spatial location of an audio signal reproduced from a set of loudspeakers at the center of gravity of the positions of the set of loudspeakers weighted by the associated activation gains of the loudspeakers.

[0248] In some examples, the one or more additional dynamically configurable features may be based at least in part on a level of the one or more audio signals. In some instances, the one or more additional dynamically configurable features may be based at least in part on a spectrum of the one or more audio signals.

[0249] Some examples of the method 1700 include receiving loudspeaker layout information. In some examples, the one or more additional dynamically configurable features may be based at least in part on the location of each of the loudspeakers in the environment.

[0250] Some examples of method 1700 include receiving loudspeaker specification information. In some examples, the one or more additional dynamically configurable features may be based at least in part on the capabilities of each loudspeaker, which may include one or more of a frequency response, playback level limits, or parameters of one or more loudspeaker dynamics processing algorithms.

[0251] According to some examples, the one or more additional dynamically configurable features may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the other loudspeakers. Alternatively or additionally, the one or more additional dynamically configurable features may be based at least in part on listener or speaker locations of one or more people in the environment. Alternatively or additionally, the one or more additional dynamically configurable features may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the listener or speaker locations. The estimates of acoustic transmission may be based at least in part on walls, furniture or other objects that may be present between each loudspeaker and the listener or speaker locations, for example.

[0252] Alternatively or additionally, the one or more additional dynamically configurable features may be based at least in part on object locations of one or more non-loudspeaker objects or landmarks in the environment. In some such implementations, the one or more additional dynamically configurable features may be based at least in part on measurements or estimates of acoustic transmission from each loudspeaker to the object or landmark locations.

[0253] By employing one or more well-defined additional cost terms to implement flexible rendering, a large number of new and useful behaviors can be achieved. All the example behaviors listed below are cast in terms of penalizing certain loudspeakers under certain conditions that are considered undesirable. As a result, these loudspeakers are less activated in the spatial rendering of a set of audio signals. In many of these cases, it may be considered to simply lower the volume of the undesirable loudspeakers independently of any modifications to the spatial rendering, but such a strategy may significantly reduce the overall balance of the audio content. For example, certain components of the mix may become completely inaudible. On the other hand, in the disclosed embodiment, integrating these penalties into the core optimization of the rendering allows the rendering to adapt and perform the best possible spatial rendering with the remaining speakers that are less penalized. This is a much more elegant, adaptable, and effective solution.

[0254] Example use cases include, but are not limited to: Providing a more balanced spatial presentation around the listening area - It has been found that spatial audio is best presented across loudspeakers that are approximately the same distance from the intended listening area, and costs can be structured such that loudspeakers that are significantly closer or farther than the average distance from the loudspeakers to the listening area are penalized, reducing their activation; Moving the audio away from or towards the listener or talker - when a user of the system intends to speak to a smart voice assistant of or associated with the system, it may be beneficial to create a cost that penalizes loudspeakers that are closer to the talker, in this way these loudspeakers are less activated and thus their associated microphones can better hear the talker; - in order to provide a single listener with a more intimate experience that minimizes playback levels for others in the listening space, speakers that are far from the listener's location may be heavily penalized so that only the speakers closest to the listener are activated most significantly; Moving the audio away from or towards a landmark, zone, or area - Certain locations near the listening space may be considered sensitive, such as baby rooms, infant beds, offices, reading areas, study areas, etc. In such cases, a cost may be established that penalizes the use of speakers close to this location, zone or area. Alternatively, for the same (or similar) case as above, a system of speakers could have generated measurements of the acoustic transmission from each speaker into the baby room, especially if one of the speakers (with attached or associated microphone) is in the baby room itself. In this case, rather than using the physical proximity of the speakers to the baby room, a cost could be constructed that penalizes the use of speakers with a high measured acoustic transmission into this room; and / or Optimal use of the speaker's capabilities - The capabilities of different loudspeakers may vary significantly. For example, one common smart speaker may only include a single 1.6" full-range driver with limited low-frequency capabilities, while another smart speaker may include a much more capable 3" woofer. These capabilities are generally reflected in the frequency response of the speaker, and thus the set of responses associated with the speaker may be utilized in cost terms. Speakers with lower capabilities at certain frequencies, as measured by their frequency response, compared to other speakers may be penalized and therefore activated less. In some implementations, such frequency response values ​​may be stored with the smart loudspeaker and then reported to a computation unit responsible for optimizing the flexible rendering; - Many speakers include two or more drivers, each responsible for reproducing a different frequency range. For example, one common smart speaker is a bidirectional design that includes a woofer for lower frequencies and a tweeter for higher frequencies. Typically, such speakers include crossover circuitry to split the full-range reproduction audio signal into appropriate frequency ranges and send them to the respective drivers. Alternatively, such speakers may provide the flexible renderer playback access to each individual driver, as well as information about the capabilities of each individual driver, such as frequency response. By applying cost terms as described immediately above, in some examples, the flexible renderer may automatically construct a crossover between the two drivers based on their relative capabilities at different frequencies; - The above examples using frequency response focus on the inherent capabilities of the speaker, but may not accurately reflect the capabilities of the speaker as placed in the listening environment. In certain cases, the frequency response of the speaker measured at the intended listening position may be available through some calibration procedure. Such measurements may be used instead of pre-calculated responses to better optimize the use of the speaker. For example, a particular speaker may be inherently very competent at certain frequencies, but because of its placement (e.g. behind a wall or furniture) may produce a very limited response at the intended listening position. Capturing this response and feeding the measurements into the appropriate cost terms may prevent significant activation of such speakers; - Frequency response is only one aspect of a loudspeaker's reproduction capabilities. Many small loudspeakers begin to distort and then reach their excursion limits as the reproduction level increases, especially for low frequencies. To reduce such distortion, many loudspeakers implement dynamics processing that constrains the reproduction level below some limiting thresholds that may be variable over frequency. If a speaker is near or at these thresholds and other speakers participating in flexible rendering are not, it makes sense to reduce the signal level in the limiting speaker and redirect this energy to other less burdened speakers. Such behavior may be achieved automatically according to some embodiments by appropriately configuring the associated cost terms. Such cost terms may include one or more of the following: Monitoring the global playback volume in relation to a loudspeaker's limit threshold. For example, loudspeakers whose volume levels are closer to their limit threshold may be penalized more; Monitoring dynamic signal levels, possibly varying over frequency, in conjunction with loudspeaker limiting thresholds, also possibly varying over frequency. For example, loudspeakers whose monitored signal levels are closer to their limiting thresholds may be penalized more; Directly monitoring loudspeaker dynamics processing parameters, such as limiting gain. In some such instances, loudspeakers whose parameters indicate more limiting may be penalized more; Monitoring the actual instantaneous voltage, current, and power being delivered by the amplifier to the loudspeaker to determine if the loudspeaker is operating within its linear range. For example, loudspeakers that are operating less linearly may be penalized more; - Smart speakers with integrated microphones and interactive voice assistants typically employ some type of echo cancellation to reduce the level of the audio signal played from the speaker that is picked up by the recording microphone. The greater this reduction, the more likely the speaker is to hear and understand talkers in the space. If the echo canceller residuals are consistently high, this may be an indication that the speaker is being driven into a non-linear region where the echo path becomes difficult to predict. In such cases, it may make sense to redirect signal energy away from the speaker, and therefore a cost term that takes echo canceller performance into account may be beneficial. Such a cost term may assign a higher cost to speakers whose associated echo cancellers have poor performance; - To achieve predictable imaging when rendering spatial audio across multiple loudspeakers, it is generally required that the playback across a set of loudspeakers be reasonably synchronized over time. While this is of course the case for wired loudspeakers, with a large number of wireless loudspeakers, synchronization can be difficult and the end result can be variable. In such cases, each loudspeaker may be able to report its relative degree of synchronization with the target, which may then be reflected in the synchronization cost term. In some such examples, loudspeakers with a lesser degree of synchronization may be penalized more and therefore excluded from rendering. Additionally, for certain types of audio signals, e.g., components of an audio mix that are intended to be diffuse or omnidirectional, strict synchronization may not be required. In some implementations, the components may be tagged as such in metadata and the synchronization cost term may be modified so that the penalties are reduced.

[0255] Further examples of embodiments are now described. Similar to the proximity costs defined in Equations 25a and 25b, the new cost function term

number

number

number

number

[0256] Combining Equations 29a and 29b with a matrix-quadratic version of the CMAP and FV cost functions given in Equation 26 gives a potentially useful implementation of the general extended cost function (for some embodiments) given in Equation 28:

number

[0257] With this definition of the new cost function terms, the overall cost function remains a matrix quadratic equation, and the optimal set of activations g opt , we can obtain:

number

[0258] Weight term w ij Each of the loudspeakers is assigned a given consecutive penalty value

number

number

number

number

[0259] If all loudspeakers are penalized, it is often convenient to subtract a minimum penalty from all weight terms in post-processing, so that at least one of the speakers is not penalized.

number

[0260] As mentioned above, there are many possible use cases that can be realized using the new cost function terms described herein (and similar new cost function terms employed according to other embodiments). More specific details will now be provided with three examples: moving audio towards a listener or talker, moving audio away from a listener or talker, and moving audio away from a landmark.

[0261] In a first example, what is referred to herein as a "force of attraction" is used to attract audio to a location, which in some examples may be a listener or talker location, a landmark location, a furniture location, etc. This location may be referred to herein as a "force of attraction" or "attractor location." As used herein, a "force of attraction" is a factor that favors relatively higher loudspeaker activation closer to the force of attraction. According to this example, a weight w ij is the fixed attractor location

number

number

[0262] To demonstrate the use case of "pulling" audio to a listener or talker, j = 20, β j Set =3,

number

number

number

[0263] In the second and third examples, a "repulsion force" is used to "keep" audio away from a location, which may be a person's location (e.g., a listener location, a talker location, etc.) or another location, such as a landmark location, a furniture location, etc. In some examples, the repulsion force may be used to keep audio away from an area or zone of a listening environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or a bedroom). According to some such examples, a particular location may be used to represent a zone or area. For example, a location representing a crib may be an estimated location of an infant's head, an estimated sound source location corresponding to an infant, etc. This location may be referred to herein as a "repulsion force location" or "repulsion location." As used herein, a "repulsion force" is a factor that favors relatively low loudspeaker activation in proximity to a repulsion force location. According to this example, a fixed repulsion location, similar to the attraction forces of Equations 35a and 35b, may be used to keep audio away from an area or zone of a listening environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or a bedroom).

number

number

[0264] To demonstrate the use case of moving audio away from the listener or talker, an example is j = 5, β j Set =2,

number

number

number

[0265] A third example use case is to "steer" audio away from acoustically sensitive landmarks, such as the door to a sleeping infant's room. Similar to the last example,

number

[0266] In a further example of method 1700 of FIG. 17, a use case is to respond to a selection of two or more audio devices in an audio environment for ducking or other audio processing corresponding to a ducking solution, and to apply a penalty to the two or more audio devices corresponding to the ducking solution. Following the previous example, the selection of the two or more audio devices is, in some embodiments, determined by a value f i , which is a unitless parameter that controls the degree to which audio processing changes occur on audio device i. i Many combinations are possible. In one simple example, the penalty assigned to audio device i for purposes of ducking is w ij =f i6. In some examples, one or more of such weights may be determined by a ducking module, such as ducking module 400 of FIG. 6. In some such examples, ducking solution 480 provided to a renderer may include one or more of such weights. In other examples, these weights may be determined by the renderer. In some such examples, one or more of such weights may be determined by the renderer in response to ducking solution 480. According to some examples, one or more of such weights may be determined according to an iterative process, such as method 800 of FIG. 8.

[0267] In addition to the previous examples of determining weights, in some implementations the weights may be determined as follows:

number

[0268] In the above formula, α j , β j , τ j represent adjustable parameters indicating the overall strength of the penalty, the abruptness of the onset of the penalty, and the extent of the penalty, respectively, as explained above with reference to Equation 33.

[0269] In the previous example, we used a unit-free parameter f i Alternatively, we can use s, which is the improvement in the speech-to-echo ratio at audio device i, expressed directly in decibels. i was also introduced. With the solution represented in this way, the penalty can also be determined as follows:

number

[0270] where f i is the dB term s i is replaced by converting it to a value in the range 0 to infinity. iAs becomes more negative, the penalty increases, thereby moving more audio away from device i.

[0271] In some examples (such as the previous two equations), the tunable parameter α j , β j , τ j One or more of may be determined by a ducking module, such as ducking module 400 of FIG. 6. In some such examples, ducking solution 480 provided to the renderer may include one or more of such adjustable parameters. In other examples, the one or more adjustable parameters may be determined by the renderer. In some such examples, the one or more adjustable parameters may be determined by the renderer in response to ducking solution 480. According to some examples, one or more of such adjustable parameters may be determined according to an iterative process, such as method 800 of FIG. 8.

[0272] The ducking penalty described above may be understood as part of a combination of multiple penalty terms resulting from multiple simultaneous use cases. For example, audio may be "kept away" from sensitive landmarks using penalties such as those described in Equations 35c-d, while the term f as determined by the decision aspect may be "kept away" from sensitive landmarks using penalties such as those described in Equations 35c-d. i Or, one can still "move away" from microphone locations where it is desired to improve the SER using s1.

[0273] Aspects of some disclosed implementations include a system or device configured (e.g., programmed) to perform one or more of the disclosed methods, and a tangible computer-readable medium (e.g., disc) storing code for implementing one or more of the disclosed methods, or steps thereof. For example, the system may be or include a programmable general-purpose processor, digital signal processor, or microprocessor programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including one or more of the disclosed methods, or steps thereof. Such a general-purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem programmed (and / or otherwise configured) to perform one or more of the disclosed methods (or steps thereof) in response to data asserted thereto.

[0274] Some disclosed embodiments are implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform the required processing on the audio signal(s), including implementing one or more of the disclosed methods. Alternatively, some embodiments (or elements thereof) may be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include input devices and memory) programmed and / or otherwise configured with software or firmware to perform any of the various operations, including one or more of the disclosed methods or steps thereof. Alternatively, elements of some disclosed embodiments are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more of the disclosed methods or steps thereof, the system also including other elements (e.g., one or more loudspeakers and / or one or more microphones). A general-purpose processor configured to perform one or more of the disclosed methods or steps thereof would typically be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0275] Another aspect of some disclosed implementations is a computer-readable medium (e.g., a disc or other tangible storage medium) storing code (e.g., executable code) for performing any embodiment of one or more of the disclosed methods or steps thereof.

[0276] While specific embodiments and applications are described herein, it will be apparent to those skilled in the art that many variations to the embodiments and applications described herein are possible without departing from the scope of the material described and claimed herein. Although specific implementations have been shown and described, it should be understood that the disclosure should not be limited to the specific embodiments described and shown or to the specific methods described.

[0277] Various aspects of the invention can be understood from the following enumerated exemplary embodiments (EEE).

[0278] <eee1>1. A method for audio processing, comprising: receiving, by a control system, output signals from one or more microphones in the audio environment, the output signals including a signal corresponding to a current speech of the person; determining, by the control system, in response to the output signals, based at least in part on the audio device location information and the echo management system information, one or more audio processing modifications to apply to audio data being rendered into loudspeaker feed signals for two or more audio devices in the audio environment, the audio processing modifications including a reduction in loudspeaker playback levels for one or more loudspeakers in the audio environment; applying, by a control system, one or more types of audio processing modifications; The method includes:

[0279] <eee2>The method of claim 8, wherein at least one of the one or more types of audio processing modifications corresponds to an increase in signal-to-echo ratio.

[0280] <eee3>The method according to any one of claims 8 to 10, wherein the echo management system information includes a model of echo management system performance.

[0281] <eee4>The method of EEE3, wherein the model of the echo management system performance includes an acoustic echo canceller (AEC) performance matrix.

[0282] <eee5>The method according to EEE3 or EEE4, wherein the model of echo management system performance includes a measure of expected echo return loss enhancement provided by the echo management system.

[0283] <eee6>The method of any one of EEE1 to 5, wherein the one or more types of audio processing modifications are based at least in part on acoustic models of inter-device echo and intra-device echo.

[0284] <eee7>The method of any one of EEE1 to 6, wherein the one or more types of audio processing modifications are based at least in part on an interaudibility matrix.

[0285] <eee8>The method of any one of EEE1 to 7, wherein the one or more types of audio processing modifications are based at least in part on the estimated location of the person.

[0286] <eee9>The method according to EEE8, wherein the estimated location of the person is based at least in part on output signals from multiple microphones in the audio environment.

[0287] <eee10>The method of EEE8 or EEE9, wherein the one or more types of audio processing modification include modifying a rendering process to warp the rendering of the audio signal away from the estimated location of the person.

[0288] <eee11>The method of any one of EEE1 to 10, wherein the one or more types of audio processing modifications are based at least in part on listening objectives.

[0289] <eee12>The method of claim 8, wherein the listening objective includes at least one of a spatial component or a frequency component.

[0290] <eee13>The method of any one of EEE1 to 12, wherein the one or more types of audio processing modifications are based at least in part on one or more constraints.

[0291] <eee14>The method according to claim 8, wherein the one or more constraints are based on a perceptual model.

[0292] <eee15>The method of any one of EEE13 to EEE14, wherein the one or more constraints include one or more of an audio content energy preservation, an audio spatiality preservation, an audio energy vector, or a regularization constraint.

[0293] <eee16>The method of any one of EEE1 to 15, further comprising updating at least one of an acoustic model of the audio environment or a model of echo management system performance after applying one or more types of audio processing modifications.

[0294] <eee17>The method of any one of EEE1 to 16, wherein determining one or more types of audio processing modifications is based on optimizing a cost function.

[0295] <eee18>The method of any one of EEE1 to 17, wherein the one or more types of audio processing modifications include spectral modifications.

[0296] <eee19>The method according to claim 8, wherein the spectral modification comprises reducing the level of the audio data in a frequency band from 500 Hz to 3 KHz.

[0297] <eee20>The method of any one of EEE1-19, wherein the current utterance comprises a wake word utterance.

[0298] <eee21>23. An apparatus configured to carry out the method according to any one of claims 1 to 20.

[0299] <eee22>A system configured to carry out the method according to any one of claims 1 to 20.

[0300] <eee23> One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform a method according to any one of EEE1 to 20.< / eee23>

Claims

1. 1. A method for audio processing, comprising: receiving, by a control system, output signals from one or more microphones in an audio environment, the output signals including a signal corresponding to a current speech of a person; determining, by the control system, in response to the output signal, based at least in part on audio device location information and echo management system information, one or more audio processing modifications to apply to audio data being rendered into loudspeaker feed signals for two or more audio devices in the audio environment, the audio processing modifications including a reduction in loudspeaker playback levels for one or more loudspeakers in the audio environment; causing the control system to apply the one or more types of audio processing modifications; The method includes:

2. The method of claim 1 , wherein at least one of the one or more types of audio processing modifications corresponds to an increase in a signal-to-echo ratio.

3. The method of claim 1 , wherein the echo management system information includes a model of echo management system performance.

4. The method of claim 3 , wherein the model of echo management system performance includes an acoustic echo canceller (AEC) performance matrix.

5. The method of claim 3 , wherein the model of echo management system performance includes a measure of expected echo return loss enhancement provided by the echo management system.

6. The method of claim 1 , wherein the one or more types of audio processing modifications are based at least in part on acoustic models of inter-device and intra-device echoes.

7. The method of claim 1 , wherein the one or more types of audio processing modifications are based at least in part on a mutual audibility matrix, the mutual audibility matrix representing energy of echo paths between the audio devices.

8. The method of claim 1 , wherein the one or more types of audio processing modifications are based at least in part on an estimated location of the person.

9. The method of claim 8 , wherein the estimated location of the person is based at least in part on output signals from a plurality of microphones in the audio environment.

10. The method of claim 8 , wherein the one or more types of audio processing modifications include modifying a rendering process to warp a rendering of an audio signal away from the estimated location of the person.

11. The method of claim 1 , wherein the one or more types of audio processing modifications are based at least in part on listening intent.

12. The method of claim 11 , wherein the listening objective comprises at least one of a spatial component or a frequency component.

13. The method of claim 1 , wherein the one or more types of audio processing modifications are based at least in part on one or more constraints.

14. The method of claim 13 , wherein the one or more constraints are based on a perceptual model.

15. The method of claim 13 , wherein the one or more constraints include one or more of an audio content energy preservation, an audio spatiality preservation, an audio energy vector, or a regularization constraint.

16. 2. The method of claim 1, further comprising updating at least one of an acoustic model of the audio environment or a model of echo management system performance after applying the one or more types of audio processing modifications.

17. The method of claim 1 , wherein determining the one or more types of audio processing modifications is based on optimizing a cost function.

18. The method of claim 1 , wherein the one or more types of audio processing modifications include spectral modifications.

19. 20. The method of claim 18, wherein the spectral modification comprises reducing a level of audio data in a frequency band from 500 Hz to 3 KHz.

20. The method of claim 1 , wherein the current utterance comprises a wake word utterance.

21. Apparatus configured to carry out the method of any one of claims 1 to 20.

22. A system configured to carry out the method of any one of claims 1 to 20.

23. 21. One or more non-transitory media having software stored thereon, the software including instructions for controlling one or more devices to perform the method of any one of claims 1 to 20.

Images