Managing playback of multiple streams of audio over multiple speakers

Abstract

To provide managing playback of multiple streams of audio over multiple speakers.SOLUTION: A multi-stream rendering system and method may render and play simultaneously a plurality of audio program streams over a plurality of arbitrarily placed loudspeakers. At least one of the program streams may be a spatial mix. The rendering of the spatial mix may be dynamically modified according to the simultaneous rendering of one or more additional program streams. The rendering of one or more additional program streams may be dynamically modified according to the simultaneous rendering of the spatial mix.SELECTED DRAWING: Figure 2A

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application is a joint venture between U.S. Provisional Patent Application No. 62 / 992,068 (filed March 19, 2020), U.S. Provisional Patent Application No. 62 / 949,998 (filed December 18, 2019), European Patent Application No. 19217580.0 (filed December 18, 2019), Spanish Patent Application No. P201930702 (filed July 30, 2019), U.S. Provisional Patent Application No. 62 / 971,421 (filed February 7, 2020), U.S. Provisional Patent Application No. 62 / 705,410 (filed June 25, 2020), and U.S. Provisional Patent Application No. 62 / 880,111 (filed June 25, 2020). This application claims priority to U.S. Provisional Patent Application No. 62 / 704,754 (filed July 30, 2019), U.S. Provisional Patent Application No. 62 / 705,896 (filed July 21, 2020), U.S. Provisional Patent Application No. 62 / 880,114 (filed July 30, 2019), U.S. Provisional Patent Application No. 62 / 705,351 (filed June 23, 2020), U.S. Provisional Patent Application No. 62 / 880,115 (filed July 30, 2019), and U.S. Provisional Patent Application No. 62 / 705,143 (filed June 12, 2020), each of which is incorporated herein by reference in its entirety.

[0002] Technical Field This disclosure relates to systems and methods for playing and rendering for playback audio through some or all speakers (e.g., each activated speaker) of a set of speakers. [Background technology]

[0003] Audio devices, including but not limited to smart audio devices, are widely deployed and are becoming a common feature in many homes. While existing systems and methods for controlling audio devices offer advantages, improved systems and methods would be desirable.

[0004] Notation and Name Throughout this disclosure, including the claims, "speaker" and "loudspeaker" are used interchangeably to refer to any sound emitting transducer (or collection of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., woofers and tweeters) that may be driven by a single common speaker feed or multiple speaker feeds. In some examples, the speaker feeds may receive different processing in different circuit branches coupled to different transducers.

[0005] Throughout this disclosure, including the claims, the expression performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) is used broadly to refer to performing the operation directly on the signal or data, or performing the operation on a processed version of the signal or data (e.g., on a version of the signal that has been subjected to preliminary filtering or preprocessing before performing the operation).

[0006] Throughout this disclosure, including the claims, the term "system" is used in a broad sense to denote an apparatus, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0007] Throughout this disclosure, including the claims, the term "processor" is used broadly to denote a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing on audio or other audio data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0008] Throughout this disclosure, including the claims, the terms "couple" or "coupled" are used to mean a direct or indirect connection. Thus, when a first device couples to a second device, the connection may be through a direct connection or through an indirect connection via other devices and connections.

[0009] As used herein, a "smart device" is an electronic device that is generally configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near-field communications, Wi-Fi, light fidelity (Li-Fi), 3G, 4G, 5G, etc., and that can operate interactively and / or autonomously to some degree. Some prominent types of smart devices are smartphones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bands, smart keychains, and smart audio devices. The term "smart device" may also refer to devices that exhibit certain characteristics of ubiquitous computing, such as artificial intelligence.

[0010] As used herein, the expression "smart audio device" refers to a smart device that is either a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device (e.g., a television (TV) or a mobile phone) that includes or is coupled to at least one microphone (and optionally includes or is coupled to at least one speaker and / or at least one camera) and is designed to accomplish a large or primary purpose. For example, a television can typically play (and is considered to be able to play) audio from program material, but in most cases modern televisions run some operating system on which applications run locally, including television viewing applications. Similarly, a mobile phone's audio input and output may do many things, but these are serviced by applications running on the phone. In this sense, a single-purpose audio device with a speaker and microphone is often configured to run local applications and / or services that directly use the speaker and microphone. Several single-purpose audio devices may be configured to be grouped together to accomplish audio playback over a zone or user-configured area.

[0011] One common type of multipurpose audio device is an audio device that implements at least some aspects of the virtual assistant functionality, although other aspects of the virtual assistant functionality may be implemented by one or more other devices, such as one or more servers with which the multipurpose audio device is configured to communicate. Such multipurpose audio devices may be referred to herein as "virtual assistants." A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and, optionally, includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to utilize multiple devices (different from the virtual assistant) for applications that are in some way cloud-enabled or otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant functionality, e.g., speech recognition functionality, may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate over a network such as the Internet. Virtual assistants may sometimes cooperate with each other, e.g., in a discrete, conditionally defined manner. For example, two or more virtual assistants can cooperate in the sense that one of them, e.g., the virtual assistant that is most confident that it heard the wake word, will respond to that word. Connected virtual assistants can in some implementations form a kind of constellation, which may be managed by one main application that may be (or may implement) a virtual assistant.

[0012] Here, a "wake word" is used broadly to mean any sound (e.g., a word spoken by a human being, or any other sound) that the smart audio device is configured to wake up in response to detecting ("listening") to that sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "awakening" refers to the device entering a state in which it waits for (i.e., listens for) a voice command. In some instances, what may be referred to herein as a "wake word" may include multiple words, e.g., a phrase.

[0013] Here, the expression "wake word detector" refers to a device (or software including instructions for configuring a device) configured to continuously look for alignment between real-time audio (e.g., speech) features and a trained model. Typically, a wake word event is triggered whenever the wake word detector determines that the probability that the wake word has been detected exceeds a predetermined threshold. For example, the threshold may be a predetermined threshold that is adjusted to provide a reasonable compromise between false accept and false reject rates. Following the wake word event, the device may enter a state (which may be referred to as an "awake" or "attention" state) in which it listens for commands and passes received commands to a larger, more computationally intensive recognizer. Summary of the Invention [Means for solving the problem]

[0014] Some embodiments relate to a method for managing playback of multiple streams of audio by at least one (e.g., all or part) of a smart audio device of a collection of smart audio devices and / or by at least one (e.g., all or part) of a speaker of another collection of speakers.

[0015] One class of embodiments includes a method for managing playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) smart audio devices. For example, a set of smart audio devices present in a user's home (in a system) may be orchestrated to handle a variety of concurrent use cases, including flexible rendering of audio for playback by all or some of the smart audio devices (i.e., by speakers of all or some of the smart audio devices).

[0016] Orchestrating smart audio devices (e.g., in a home to handle a variety of concurrent use cases) may involve the simultaneous playback of one or more audio program streams through a set of interconnected speakers. For example, a user may be listening to an Atmos soundtrack of a movie (or other object-based audio program) through a set of speakers (e.g., included in or controlled by a set of smart audio devices) when the user issues a command (e.g., a wake word followed by a command) to an associated smart audio device (e.g., a smart assistant). In this case, audio playback by the system may be modified (according to some embodiments) to warp the spatial presentation of the program (e.g., the Atmos mix) away from the location of the speaker (the user who is speaking) and direct playback of the corresponding response of the smart audio device (e.g., of a voice assistant) to the speaker closer to the speaker. This may provide a significant advantage over simply decreasing the volume of playback of the audio program content in response to detection of the command (or the corresponding wake word). Similarly, a user may wish to use speakers in a kitchen to get cooking tips while the same program (e.g., an Atmos soundtrack) is being played in an adjacent open living space. In this case, according to some embodiments, the playback of the program (e.g., an Atmos soundtrack) can be warped away from the kitchen and the cooking tips can be played from speakers near or in the kitchen. Furthermore, the cooking tips played in the kitchen can be dynamically adjusted (according to some embodiments) to be louder than any of the program (e.g., an Atmos soundtrack) that may be seeping out of the living space and heard by those in the kitchen.

[0017] Some embodiments are multiple stream rendering systems configured to implement the above-mentioned exemplary use cases and many other possible ones. In one class of embodiments, an audio rendering system may be configured to render multiple audio program streams (and / or simultaneously play those streams) for simultaneous playback through multiple arbitrarily positioned loudspeakers, at least one of said program streams being a spatial mix, and the rendering (or rendering and playback) of said spatial mix being dynamically modified in response to (or in association with) the simultaneous playback (or rendering and playback) of one or more additional program streams.

[0018] Aspects of some embodiments include a system configured (e.g., programmed) to perform any of the disclosed method embodiments or steps thereof, and a tangible, non-transitory, computer-readable medium (e.g., a disk or other tangible storage medium) implementing non-transitory storage of data that stores code (e.g., executable code to perform) for performing any of the disclosed method embodiments or steps thereof. For example, some embodiments may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including the disclosed method embodiments or steps thereof. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform the disclosed method embodiments (or steps thereof) in response to data presented thereto.

[0019] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more apparatus may be capable of performing, at least in part, the methods disclosed herein. In some implementations, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general purpose single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof.

[0020] In some implementations, the control system includes or implements at least two rendering modules. According to some examples, the control system may include or implement N rendering modules, where N is an integer greater than 2.

[0021] In some examples, the first rendering module is configured to receive the first audio program stream via the interface system. In some cases, the first audio program stream includes a first audio signal scheduled for playback by at least some speakers of the environment. In some examples, the first audio program stream includes first spatial data, including channel data and / or spatial metadata. According to some examples, the first rendering module is configured to render the first audio signal for playback via the speakers of the environment and generate a first rendered audio signal.

[0022] In some implementations, the second rendering module is configured to receive the second audio program stream via the interface system. In some examples, the second audio program stream includes a second audio signal scheduled for playback by at least some speakers of the environment. In some examples, the second audio program stream includes second spatial data, including channel data and / or spatial metadata. According to some examples, the second rendering module is configured to render the second audio signal for playback via the speakers of the environment to generate a second rendered audio signal.

[0023] According to some examples, the first rendering module is configured to modify the rendering process for the first audio signal based at least in part on the second audio signal, the second rendered audio signal, or at least one of the characteristics thereof to generate a modified first rendered audio signal. According to some examples, the second rendering module is further configured to modify the rendering process for the second audio signal based at least in part on the first audio signal, the first rendered audio signal, or at least one of the characteristics thereof to generate a modified second rendered audio signal.

[0024] In some implementations, the audio processing system includes a mixing module configured to mix the modified first rendered audio signal and the modified second rendered audio signal to generate a mixed audio signal. In some examples, the control system is further configured to provide the mixed audio signal to at least some speakers of the environment.

[0025] According to some examples, the audio processing system may include one or more additional rendering modules. In some examples, each of the one or more additional rendering modules may be configured to receive an additional audio program stream via an interface system. The additional audio program stream may include an additional audio signal scheduled for playback by at least one speaker of the environment. In some examples, each of the one or more additional rendering modules may be configured to render the additional audio signal for playback through at least one speaker of the environment to generate an additional rendered audio signal. In some examples, each of the one or more additional rendering modules may be configured to modify a rendering process for the additional audio signal based, at least in part, on at least one of the first audio signal, the first rendered audio signal, the second audio signal, the second rendered audio signal, or characteristics thereof to generate a modified additional rendered audio signal. In some such examples, the mixing module may be further configured to mix a modified additional rendered audio signal with at least the modified first rendered audio signal and the modified second rendered audio signal to generate the mixed audio signal.

[0026] In some implementations, modifying the rendering process for the first audio signal may include warping the rendering of the first audio signal away from a rendering location of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may include modifying a loudness of one or more of the first rendered audio signals in response to a loudness of the second audio signal or one or more of the second rendered audio signals.

[0027] According to some examples, modifying the rendering process of the second audio signal may include warping the rendering of the second audio signal away from a rendering location of the first rendered audio signal. Alternatively or additionally, modifying the rendering process of the second audio signal may include modifying a loudness of one or more of the second rendered audio signals in response to a loudness of the first audio signal or one or more of the first rendered audio signals. According to some implementations, modifying the rendering process for the first audio signal and / or the second audio signal may include performing a spectral modification, an audibility-based modification, and / or a dynamic range modification.

[0028] In some examples, the audio processing system may include a microphone system including one or more microphones. In some such examples, the first rendering module may be configured to modify a rendering process for the first audio signal based at least in part on a first microphone signal from the microphone system. In some such examples, the second rendering module may be configured to modify a rendering process for the second audio signal based at least in part on the first microphone signal.

[0029] According to some examples, the control system may be further configured to estimate a first sound source position based on the first microphone signal and modify a rendering process for at least one of the first audio signal or the second audio signal based at least in part on the first sound source position. In some examples, the control system may be further configured to determine whether the first microphone signal corresponds to environmental noise and modify a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to environmental noise.

[0030] In some examples, the control system may be configured to determine whether the first microphone signal corresponds to a human voice and modify a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to a human voice. According to some such examples, modifying the rendering process for the first audio signal may include reducing a loudness of a first rendered audio signal reproduced by a speaker near the first sound source position compared to a loudness of a first rendered audio signal reproduced by a speaker farther from the first sound source position.

[0031] According to some examples, the control system may be configured to determine that the first microphone signal corresponds to a wake word, determine a response to the wake word, and control at least one speaker near the first audio source location to play the response. In some examples, the control system may be configured to determine that the first microphone signal corresponds to a command, determine a response to the command, control at least one speaker near the first audio source location to play the response, and execute the command. According to some examples, the control system may be further configured to return to the rendering process for the first audio signal unmodified after controlling the at least one speaker near the first audio source location to play the response.

[0032] In some implementations, the control system may be configured to derive a loudness estimate for the played first audio program stream and / or the played second audio program stream based at least in part on the first microphone signal. According to some examples, the control system may be further configured to modify a rendering process for at least one of the first audio signal or the second audio signal based at least in part on the loudness estimate. In some cases, the loudness estimate may be a perceived loudness estimate. According to some such examples, modifying the rendering process may include altering at least one of the first audio signal or the second audio signal to maintain a perceived loudness of the first audio signal and / or the second audio signal in the presence of an interfering signal.

[0033] In some examples, the control system may be configured to determine that the first microphone signal corresponds to a human voice and to play the first microphone signal at one or more speakers proximate a location in the environment different from the first sound source location. According to some such examples, the control system may be further configured to determine whether the first microphone signal corresponds to a child's cry. In some such examples, the location in the environment may correspond to an estimated location of a caregiver.

[0034] According to some examples, the control system may be configured to derive a loudness estimate for the first audio program stream being played and / or the second audio program stream being played. In some such examples, the control system may be further configured to modify a rendering process for the first audio signal and / or the second audio signal based at least in part on the loudness estimate. According to some examples, the loudness estimate may be a perceived loudness estimate. Modifying the rendering process may include altering at least one of the first audio signal or the second audio signal to maintain a perceived loudness in the presence of an interfering signal.

[0035] In some implementations, rendering the first audio signal and / or rendering the second audio signal may include flexible rendering to arbitrarily positioned speakers. In some such examples, the flexible rendering may include Center of Mass Amplitude Panning or Flexible Virtualization.

[0036] At least some aspects of the present disclosure may be implemented by one or more audio processing methods. In some examples, the methods may be implemented, at least in part, by a control system such as those disclosed herein. Some such methods include receiving, by a first rendering module, a first audio program stream, the first audio program stream including a first audio signal scheduled for playback by at least some speakers of the environment. In some examples, the first audio program stream includes first spatial data, including channel data and / or spatial metadata. Some such methods include rendering, by the first rendering module, the first audio signal for playback through speakers of the environment to generate a first rendered audio signal.

[0037] Some such methods include receiving, by a second rendering module, a second audio program stream. In some examples, the second audio program stream includes a second audio signal scheduled for playback by at least one speaker of the environment. Some such methods include rendering, by the second rendering module, the second audio signal for playback through the at least one speaker of the environment to generate a second rendered audio signal.

[0038] Some such methods include modifying, by a first rendering module, a rendering process for the first audio signal based at least in part on the second audio signal, the second rendered audio signal, or at least one of their characteristics to generate a modified first rendered audio signal. Some such methods include modifying, by a second rendering module, a rendering process for the second audio signal based at least in part on the first audio signal, the first rendered audio signal, or at least one of their characteristics to generate a modified second rendered audio signal. Some such methods include mixing the modified first rendered audio signal and the modified second rendered audio signal to generate a mixed audio signal, and providing the mixed audio signal to at least some speakers of the environment.

[0039] According to some examples, modifying the rendering process for the first audio signal may include warping the rendering of the first audio signal away from a rendering location of the second rendered audio signal and / or modifying a loudness of one or more of the first rendered audio signals in response to a loudness of the second audio signal or one or more of the second rendered audio signals.

[0040] In some examples, modifying the rendering process of the second audio signal may include warping the rendering of the second audio signal away from a rendering location of the first rendered audio signal and / or modifying a loudness of one or more of the second rendered audio signals in response to a loudness of the first audio signal or one or more of the first rendered audio signals.

[0041] According to some examples, modifying the rendering process for the first audio signal may include performing a spectral modification, an audibility-based modification, and / or a dynamic range modification.

[0042] Some methods may include modifying, by a first rendering module, a rendering process for the first audio signal based at least in part on a first microphone signal from the microphone system. Some methods may include modifying, by a second rendering module, a rendering process for the second audio signal based at least in part on the first microphone signal.

[0043] Some methods may include estimating a first sound source position based on the first microphone signal and modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on the first sound source position.

[0044] Some or all of the operations, functions and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including, but not limited to, random access memory (RAM) devices, read-only memory (ROM) devices, and the like. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in a non-transitory medium having software stored thereon. The method may include receiving, by a first rendering module, a first audio program stream, the first audio program stream including a first audio signal scheduled for playback by at least one speaker of the environment. In some examples, the first audio program stream includes first spatial data, including channel data and / or spatial metadata. Some such methods include rendering, by the first rendering module, the first audio signal for playback through the speakers of the environment, generating a first rendered audio signal.

[0045] Some such methods include receiving, by a second rendering module, a second audio program stream. In some examples, the second audio program stream includes a second audio signal scheduled for playback by at least one speaker of the environment. Some such methods include rendering, by the second rendering module, the second audio signal for playback through the at least one speaker of the environment to generate a second rendered audio signal.

[0046] Some such methods include modifying, by a first rendering module, a rendering process for the first audio signal based at least in part on the second audio signal, the second rendered audio signal, or at least one of their characteristics to generate a modified first rendered audio signal. Some such methods include modifying, by a second rendering module, a rendering process for the second audio signal based at least in part on the first audio signal, the first rendered audio signal, or at least one of their characteristics to generate a modified second rendered audio signal. Some such methods include mixing the modified first rendered audio signal and the modified second rendered audio signal to generate a mixed audio signal, and providing the mixed audio signal to at least some speakers of the environment.

[0047] According to some examples, modifying the rendering process for the first audio signal may include warping the rendering of the first audio signal away from a rendering location of the second rendered audio signal and / or modifying a loudness of one or more of the first rendered audio signals in response to a loudness of the second audio signal or one or more of the second rendered audio signals.

[0048] In some examples, modifying the rendering process for the second audio signal may include warping the rendering of the second audio signal away from a rendering location of the first rendered audio signal and / or modifying a loudness of one or more of the second rendered audio signals in response to a loudness of the first audio signal or one or more of the first rendered audio signals.

[0049] According to some examples, modifying the rendering process for the first audio signal may include performing a spectral modification, an audibility-based modification, and / or a dynamic range modification.

[0050] Some methods may include modifying, by a first rendering module, a rendering process for the first audio signal based at least in part on a first microphone signal from the microphone system. Some methods may include modifying, by a second rendering module, a rendering process for the second audio signal based at least in part on the first microphone signal.

[0051] Some methods may include estimating a first sound source position based on the first microphone signal and modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on the first sound source position.

[0052] Details of one or more implementations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. It should be noted that the relative dimensions of the following figures may not be drawn to scale. [Brief description of the drawings]

[0053] [Figure 1A] FIG. 1 is a block diagram illustrating example components of an apparatus capable of implementing various aspects of the present disclosure. [Figure 1B] FIG. 2 is a block diagram of a minimal version of an embodiment. [Figure 2A] Another (more advanced) embodiment having additional features is shown. [Figure 2B]FIG. 1 is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as that shown in FIG. 1A, FIG. 1B, or FIG. 2A. [Figure 2C] FIG. 1 illustrates an exemplary set of speaker activations. [Figure 2D] FIG. 13 illustrates an example set of speaker activation and object rendering positions. [Figure 2E] 1B is a flow diagram outlining an example of a method that may be performed by a device or system such as that shown in FIG. 1A. [Figure 2F] 1 is a graph of speaker activation in an example embodiment. [Figure 2G] 1 is a graph of object rendering positions in an example embodiment. [Figure 2H] 1 is a graph of speaker activation in an example embodiment. [Figure 2I] 1 is a graph of object rendering positions in an example embodiment. [Figure 2J] 1 is a graph of speaker activation in an example embodiment. [Figure 2K] 1 is a graph of object rendering positions in an example embodiment. [Figure 3A] 1 shows an example of a connected living space floor plan. [Figure 3B] 1 shows an example of a connected living space floor plan. [Figure 4A] We present an example of a multi-stream renderer that provides simultaneous playback of a spatial music mix and voice assistant responses. [Figure 4B] We present an example of a multi-stream renderer that provides simultaneous playback of a spatial music mix and voice assistant responses. [Figure 5A] 13 illustrates a third example use case of the disclosed multi-stream renderer. [Figure 5B] 13 illustrates a third example use case of the disclosed multi-stream renderer. [Figure 5C] 13 illustrates a third example use case of the disclosed multi-stream renderer. [Figure 6] Figure 1B shows an example of the frequency / transform domain of the multi-stream renderer. [Figure 7] Figure 2A shows an example of the frequency / transform domain of the multi-stream renderer. [Figure 8] We present an implementation of a multi-stream rendering system with an audio stream loudness estimator. [Figure 9A] An example of a multi-stream rendering system configured for cross-fading of multiple rendered streams is shown. [Figure 9B] 1 is a point graph illustrating speaker activation in an exemplary embodiment. [Figure 10] 1 is a graph of tri-linear interpolation between points illustrating speaker activation according to an example. [Figure 11] 1 shows the floor plan of the listening environment, which in this example is a residential space. [Figure 12A] 12A-12C are diagrams illustrating an example of flexible rendering of spatial audio in the reference spatial mode for different listening positions and orientations in the living space shown in FIG. 11. [Figure 12B] 12A-12C are diagrams illustrating an example of flexible rendering of spatial audio in the reference spatial mode for different listening positions and orientations in the living space shown in FIG. 11. [Figure 12C] 12A-12C are diagrams illustrating an example of flexible rendering of spatial audio in the reference spatial mode for different listening positions and orientations in the living space shown in FIG. 11. [Figure 12D] 12A-12C are diagrams illustrating an example of flexible rendering of spatial audio in the reference spatial mode for different listening positions and orientations in the living space shown in FIG. 11. [Figure 12E]An example of reference spatial mode rendering for two listeners in different positions in the listening environment is shown. [Figure 13A] 1 illustrates an example of a graphical user interface (GUI) for receiving user input regarding a listener's position and orientation. [Figure 13B] 1 illustrates a distributed spatial rendering mode in accordance with an example embodiment. [Figure 14A] 13 illustrates a partially distributed spatial rendering mode according to an example. [Figure 14B] An example shows a fully distributed spatial rendering mode. [Figure 15] Exemplary rendering positions are shown for Center of Mass Amplitude Pan (CMAP) and Flexible Virtualization (FV) rendering systems on a 2D plane. [Figure 16A] 16D and 16E show various examples of intermediate distributed spatial modes between the distributed spatial modes shown in FIG. 15 and FIG. 16D. [Figure 16B] 16D and 16E show various examples of intermediate distributed spatial modes between the distributed spatial modes shown in FIG. 15 and FIG. 16D. [Figure 16C] 16D and 16E show various examples of intermediate distributed spatial modes between the distributed spatial modes shown in FIG. 15 and FIG. 16D. [Figure 16D] To achieve a fully distributed rendering mode, Figure 15 shows an example of warping applied to all of the rendering points. [Figure 17] Here is an example of a GUI that allows the user to select a rendering mode. [Figure 18] FIG. 1 is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as those disclosed herein. [Figure 19] 1 shows an example of the geometric relationships between three audio devices in an environment. [Figure 20]20 illustrates another example of the geometric relationships between three audio devices in the environment shown in FIG. 19. [Figure 21A] Both the triangles shown in Figures 19 and 20 are shown without the corresponding audio equipment and other features of the environment. [Figure 21B] Here is an example of estimating the interior angles of a triangle formed by three audio devices. [Figure 22] FIG. 1B is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. [Figure 23] An example is shown in which each audio device in the environment is a vertex of multiple triangles. [Figure 24] An example of part of the sorting process is shown below. [Diagram 25] 4 shows an example of multiple estimates of audio device positions generated during the forward alignment process. [Figure 26] An example of part of the reverse alignment process is given below. [Figure 27] 4 shows an example of multiple estimates of audio device position generated during the de-alignment process. [Figure 28] 4 shows a comparison of estimated and actual audio device positions. [Figure 29] FIG. 1B is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. [Figure 30A] An example of some of the blocks in FIG. 29 is shown. [Figure 30B] 13 illustrates a further example of determining listener angular orientation data. [Figure 30C] 13 illustrates a further example of determining listener angular orientation data. [Figure 30D] 30C shows an example of determining an appropriate rotation for audio device coordinates according to the method described with reference to FIG. [Diagram 31] FIG. 1 is a block diagram illustrating example components of a system in which various aspects of the disclosure can be implemented. [Diagram 32] A, B, and C show examples of playback limit thresholds and corresponding frequencies. [Diagram 33] Graphs A and B show examples of dynamic range compressed data. [Diagram 34] 1 illustrates examples of spatial zones in a listening environment. [Diagram 35] 35 shows an example of loudspeakers within the spatial zones of FIG. 34. [Diagram 36] 36 shows an example of nominal spatial positions superimposed on the spatial zones and speakers of FIG. [Figure 37] FIG. 1 is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as those disclosed herein. [Figure 38A] 2C and 2D show examples of loudspeaker participation values ​​corresponding to the examples of FIG. [Figure 38B] 2C and 2D show examples of loudspeaker participation values ​​corresponding to the examples of FIG. [Figure 38C] 2C and 2D show examples of loudspeaker participation values ​​corresponding to the examples of FIG. [Figure 39A] 2F and 2G show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2F and 2G. [Figure 39B] 2F and 2G show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2F and 2G. [Figure 39C] 2F and 2G show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2F and 2G. [Figure 40A] 2H and 2I show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2I and 2H. [Figure 40B] 2H and 2I show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2I and 2H. [Figure 40C] 2H and 2I show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2I and 2H. [Figure 41A] 2J and 2K show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2J and 2K. [Figure 41B] 2J and 2K show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2J and 2K. [Figure 41C] 2J and 2K show examples of loudspeaker participation values ​​corresponding to the examples of FIGS. 2J and 2K. [Diagram 42] FIG. 2 is a diagram of an environment, which in this example is a residential space.

[0054] Like reference numbers and designations in the various drawings indicate like elements. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0055] Flexible rendering is a technique for rendering spatial audio through any number of arbitrarily positioned speakers. With the proliferation of smart audio devices (e.g., smart speakers) in the home, there is a need to provide flexible rendering techniques that allow consumers to flexibly render audio and play the rendered audio using their smart audio devices.

[0056] Several techniques have been developed to achieve flexible rendering, including Center of Mass Amplitude Panning (CEAP) and Flexible Virtualization (FV). Both of these techniques cast the rendering problem as a problem of cost function minimization. The cost function consists of two terms: a first term that models the desired spatial impression the renderer is trying to achieve, and a second term that assigns a cost to activating speakers. To date, this second term has focused on producing a sparse solution where only speakers close to the desired spatial location of the audio being rendered are activated.

[0057] Some embodiments relate to a method for managing playback of multiple streams of audio by at least one (e.g., in whole or in part) of a smart audio device of a collection of smart audio devices (or by at least one (e.g., in whole or in part) of a speaker of another collection of speakers).

[0058] One class of embodiments includes a method for managing playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) smart audio devices. For example, a set of smart audio devices present in a user's home (in a system) may be orchestrated to handle a variety of concurrent use cases, including flexible rendering of audio for playback by all or some of the smart audio devices (i.e., by speakers of all or some of the smart audio devices).

[0059] Orchestrating smart audio devices (e.g., in a home to handle a variety of concurrent use cases) may involve the simultaneous playback of one or more audio program streams through a set of interconnected speakers. For example, a user may be listening to an Atmos soundtrack (or other object-based audio program) from a movie through a set of speakers, but then the user may issue a command to an associated smart assistant (or other smart audio device). In this case, audio playback by the system may be modified (according to some embodiments) to warp the spatial presentation of the Atmos mix away from the location of the speaker (the user who is speaking) and away from the nearest smart audio device, while simultaneously warping playback of the corresponding response (of the voice assistant) of the smart audio device towards the location of the speaker. This may provide a significant advantage over simply decreasing the volume of playback of audio program content in response to detection of a command (or a corresponding wake word). Similarly, a user may wish to use a speaker in the kitchen to get cooking tips while the same Atmos soundtrack is being played in an adjacent open living space. In this case, according to some examples, the Atmos soundtrack can be warped away from the kitchen and / or the loudness of one or more rendered signals of the Atmos soundtrack can be modified in response to the loudness of one or more rendered signals of the cooking tips soundtrack. Further, in some implementations, the cooking tips played in the kitchen can be dynamically adjusted to be heard by the person in the kitchen louder than any of the Atmos soundtrack that may be seeping out of the living space.

[0060] Some embodiments involve a multiple stream rendering system configured to implement the above-mentioned exemplary use cases and many other possible ones. In one class of embodiments, the audio rendering system may be configured to simultaneously play multiple audio program streams through multiple arbitrarily positioned loudspeakers, at least one of the program streams being a spatial mix, and the rendering of the spatial mix being dynamically modified in response to (or in association with) the simultaneous playback of one or more additional program streams.

[0061] In some embodiments, a Multi-Stream Renderer may be configured to implement the scenarios described above and many other cases in which the simultaneous playback of multiple audio program streams must be managed. Some implementations of a Multi-Stream Renderer may be configured to perform the following operations: Simultaneously rendering and playing multiple audio program streams through multiple arbitrarily positioned loudspeakers, at least one of said program streams being a spatial mix. The term "program stream" refers to a collection of one or more audio signals intended to be listened to together as a whole. Examples include a music selection, a movie soundtrack, a podcast, a live voice call, a synthesized voice response from a smart assistant, etc. A spatial mix is ​​a program stream that is intended to deliver different (more than mono) signals at the listener's left and right ears. Examples of spatial mix audio formats include stereo, 5.1 and 7.1 surround sound, object audio formats such as Dolby Atmos, and Ambisonics. Rendering a program stream refers to the process of actively distributing one or more associated audio signals across multiple loudspeakers to achieve a particular perceptual impression. Dynamically modifying the rendering of the at least one spatial mix as a function of the rendering of one or more of the additional program streams. Examples of such modifications to the rendering of the spatial mix include, but are not limited to, the following: Modifying the relative activation of a plurality of loudspeakers as a function of the relative activation of a loudspeaker associated with the rendering of at least one of the one or more additional program streams. Warping the intended spatial balance of the spatial mix as a function of spatial characteristics of a rendering of at least one of said one or more additional program streams. Modifying the loudness or audibility of the spatial mix as a function of the loudness or audibility of at least one of the one or more additional program streams.

[0062] FIG. 1A is a block diagram illustrating example components of a device that can implement various aspects of the disclosure. According to some examples, device 100 may be or include a smart audio device configured to perform at least some of the methods disclosed herein. In other implementations, device 100 may be or include another device configured to perform at least some of the methods disclosed herein, such as a laptop computer, a cellular phone, a tablet device, a smart home hub, etc. In some such implementations, device 100 may be or include a server. In some implementations, device 100 may be configured to implement what may be referred to herein as an "audio session manager."

[0063] In this example, device 100 includes interface system 105 and control system 110. Interface system 105 may be configured to communicate with one or more devices running or configured to run software applications, in some implementations. Such software applications may be referred to herein as “applications” or simply “apps.” Interface system 105 may be configured to exchange control information and associated data relating to the applications, in some implementations. Interface system 105 may be configured to communicate with one or more other devices of an audio environment, in some examples, a home audio environment. Interface system 105 may be configured to exchange control information and associated data with audio devices of the audio environment, in some implementations. The control information and associated data may, in some examples, relate to one or more applications with which device 100 is configured to communicate.

[0064] The interface system 105, in some implementations, may be configured to receive an audio program stream. The audio program stream may include audio signals scheduled to be played by at least some speakers in the environment. The audio program stream may include spatial data, such as channel data and / or spatial metadata. The interface system 105, in some implementations, may be configured to receive input from one or more microphones in the environment.

[0065] The interface system 105 may include one or more network interfaces and / or one or more external device interfaces (such as one or more universal serial bus interfaces). According to some implementations, the interface system 105 may include one or more wireless interfaces. The interface system 105 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 105 may include one or more interfaces between the control system 110 and a memory system, such as the optional memory system 115 shown in FIG. 1A, although in some cases the control system 110 may include the memory system.

[0066] The control system 110 may include, for example, a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0067] In some implementations, the control system 110 may reside in more than one device. For example, a portion of the control system 110 may reside in a device in one of the environments illustrated herein, and another portion of the control system 110 may reside in a device outside the environment, such as a server, a mobile device (e.g., a smartphone or tablet computer). In other examples, a portion of the control system 110 may reside in a device in one of the environments illustrated herein, and another portion of the control system 110 may reside in one or more other devices of the environment. For example, the functionality of the control system may be distributed across multiple smart audio devices of the environment, or shared by an orchestration device (e.g., what may be referred to herein as a smart home hub) and one or more other devices of the environment. The interface system 105 may also reside in more than one device in some such examples.

[0068] In some implementations, control system 110 may be configured, at least in part, to perform the methods disclosed herein. According to some examples, control system 110 may be configured to implement a method for managing the playback of multiple streams of audio through multiple speakers.

[0069] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored in one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. The one or more non-transitory media may reside, for example, in optional memory system 115 and / or control system 110 shown in FIG. 1A. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media storing software. The software may include, for example, instructions for controlling at least one device to process audio data. The software may be executable by one or more components of a control system, such as, for example, control system 110 of FIG. 1A.

[0070] 1A . Optional microphone system 120 may include one or more microphones. In some implementations, one or more of the microphones may be part of or associated with another device, such as a speaker of a speaker system, a smart audio device, etc. In some examples, device 100 may not include microphone system 120, although in some such implementations device 100 may still be configured to receive microphone data for one or more microphones in an audio environment via interface system 110.

[0071] According to some implementations, the device 100 may include an optional loudspeaker system 125 as shown in FIG. 1A. The optional speaker system 125 may include one or more speakers. A loudspeaker is also referred to herein as a "speaker." In some examples, at least some speakers of the optional speaker system 125 may be arranged arbitrarily. For example, at least some speakers of the optional speaker system 125 may be arranged in positions that do not correspond to any standard-defined loudspeaker layout, such as Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc. In some such examples, at least some loudspeakers of the optional speaker system 125 may be arranged in positions that are convenient for the space (e.g., positions where there is space to accommodate the loudspeakers), but may be positions that are not in any standard-defined loudspeaker layout. In some examples, the device 100 may not include the loudspeaker system 125.

[0072] In some implementations, the device 100 may include an optional sensor system 129 as shown in FIG. 1A. The optional sensor system 129 may include one or more cameras, touch sensors, gesture sensors, motion detectors, and the like. According to some implementations, the optional sensor system 129 may include one or more cameras. In some implementations, the camera may be a freestanding camera. In some examples, the one or more cameras of the optional sensor system 129 may be present in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, the one or more cameras of the optional sensor system 129 may be present in a television, a cell phone, or a smart speaker. In some examples, the device 100 may not include the sensor system 129. However, in some such implementations, the device 100 may still be configured to receive sensor data for one or more sensors in the audio environment via the interface system 110.

[0073] In some implementations, device 100 may include an optional display system 135, as shown in FIG. 1A. Optional display system 135 may include one or more displays, such as one or more light emitting diode (LED) displays. In some cases, optional display system 135 may include one or more organic light emitting diode (OLED) displays. In some examples in which device 100 includes display system 135, sensor system 129 may include a touch sensor system and / or a gesture sensor system proximate one or more displays of display system 135. According to some such implementations, control system 110 may be configured to control display system 135 to present one or more graphical user interfaces (GUIs).

[0074] According to some such examples, device 100 may be or include a smart audio device. In some such implementations, device 100 may be or include a wake word detector. For example, device 100 may be or include a virtual assistant.

[0075] FIG. 1B is a block diagram of a minimal version of one embodiment. Depicted are N program streams (N≧2), with the first explicitly labeled as spatial. A collection of corresponding audio signals is fed through a corresponding renderer, each of which is individually configured for playback of the corresponding program stream through a common set of M arbitrarily spaced loudspeakers (M≧2). The renderers may be referred to as "rendering modules." The rendering modules and mixer 130a may be implemented via software, hardware, firmware, or some combination thereof. In this example, the rendering modules and mixer 130a are implemented via a control system 110a, which is an example of the control system 110 described above with reference to FIG. 1A. Each of the N renderers outputs a set of M loudspeaker feeds. The loudspeaker feeds are summed across all N renderers for simultaneous playback through the M loudspeakers. According to this implementation, information regarding the layout of the M loudspeakers in the listening environment is provided to all renderers, as indicated by the dashed lines feeding back from the loudspeaker block, so that the renderers can be properly configured for playback through those speakers. This layout information may or may not be transmitted from the speaker or speakers themselves, depending on the specific implementation. According to some examples, the layout information may be provided by one or more smart speakers configured to determine the relative positions of each of the M loudspeakers in the listening environment. Some of such automatic location methods may be based on direction of arrival (DOA) or time of arrival (TOA) methods. In other examples, this layout information may be determined by another device and / or input by a user.In some examples, loudspeaker specification information regarding the capabilities of at least some of the M loudspeakers in the listening environment may be provided to all renderers. Such loudspeaker specification information may include impedance, frequency response, sensitivity, power rating, number and location of individual drivers, etc. According to this example, information from one or more renderings of additional program streams may be fed to the renderers of the primary spatial stream, such that said renderings may be dynamically modified as a function of said information. This information is represented by the dashed lines leading from rendering blocks 2 through N back up to rendering block 1.

[0076] FIG. 2A illustrates another (more capable) embodiment with additional features. In this example, the rendering module and mixer 130b is implemented via a control system 110b, which is an example of the control system 110 described above with reference to FIG. 1A. In this version, the dashed lines moving up and down among all N renderers represent the idea that any of the N renderers may contribute to the dynamic modification of any of the remaining N-1 renderers. In other words, the rendering of any one of the N program streams may be dynamically modified as a function of the combination of one or more renderings of any of the remaining N-1 program streams. Additionally, any one or more of the program streams may be a spatial mix, and the rendering of any program stream, whether it is spatial or not, may be dynamically modified as a function of any of the other program streams. Loudspeaker layout information may be provided to the N renderers, for example, as described above. In some examples, loudspeaker specification information may be provided to the N renderers. In some implementations, microphone system 120a may include a set of K microphones in a listening environment, where K≧1. In some examples, the microphones may be attached to or associated with the one or more of the loudspeakers. These microphones may feed back both captured audio signals, represented by solid lines, and additional configuration information (e.g., their positions), represented by dashed lines, to a set of N renderers. Any of the N renderers may then be dynamically modified as a function of this additional microphone input. Various examples are provided herein.

[0077] Examples of information derived from microphone input and then used to dynamically modify any of the N renderers include, but are not limited to: · Detecting the utterance of a particular word or phrase by a user of the system. · Estimating the location of one or more users of the System. An estimate of the loudness of any of the N program stream combinations at a particular location in the listening space. An estimate of the loudness of other environmental sounds, such as background noise in your listening environment.

[0078] FIG. 2B is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as that shown in FIG. 1A, FIG. 1B, or FIG. 2A. The blocks of method 200, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. The blocks of method 200 may be performed by one or more apparatuses that may be (or may include) a control system such as control system 110, control system 110a, or control system 110b shown in FIG. 1A, FIG. 1B, and FIG. 2A, or one of the other disclosed control system examples.

[0079] In this implementation, block 205 includes receiving a first audio program stream via an interface system. In this example, the first audio program stream includes a first audio signal scheduled to be played by at least some speakers of the environment. Here, the first audio program stream includes first spatial data. According to this example, the first spatial data includes channel data and / or spatial metadata. In some examples, block 205 includes a first rendering module of the control system receiving the first audio program stream via the interface system.

[0080] According to this example, block 210 includes rendering the first audio signal for playback through speakers of the environment to generate a first rendered audio signal. Some examples of method 200 include receiving loudspeaker layout information, for example, as described above. Some examples of method 200 include receiving loudspeaker specification information, for example, as described above. In some examples, the first rendering module may generate the first rendered audio signal based, at least in part, on the loudspeaker layout information and / or the loudspeaker specification information.

[0081] In this example, block 215 includes receiving the second audio program stream via the interface system. In this implementation, the second audio program stream includes a second audio signal scheduled to be played by at least some speakers of the environment. According to this example, the second audio program stream includes second spatial data. The second spatial data includes channel data and / or spatial metadata. In some examples, block 215 includes a second rendering module of the control system receiving the second audio program stream via the interface system.

[0082] According to this implementation, block 220 includes rendering the second audio signal for playback through speakers of the environment to generate a second rendered audio signal. In some examples, the second rendering module can generate the second rendered audio signal based at least in part on the received loudspeaker layout information and / or the received speaker specification information.

[0083] In some cases, some or all of the speakers in an environment may be positioned arbitrarily. For example, at least some of the speakers in an environment may be positioned in locations that do not correspond to a standard-defined speaker layout, such as Dolby 5.1, Dolby 7.1, Hamasaki 22.2, etc. In some such examples, at least some of the speakers in an environment may be positioned in convenient locations with respect to the environment's furniture, walls, etc. (e.g., locations where there is space to accommodate the speakers), but in locations that are not in any standard-defined speaker layout.

[0084] Thus, some implementation blocks 210 or 220 may involve flexible rendering to arbitrarily placed speakers. Some such implementations may involve Center of Mass Amplitude Panning (CMAP), Flexible Virtualization (FV), or a combination of both. From a high level, these techniques render a set of one or more audio signals, each with an associated desired perceived spatial location, for playback through a set of two or more speakers, where the relative activation of the speakers of the set is a function of a model of the perceived spatial location of the audio signals played through those speakers and the proximity of the desired perceived spatial location of the audio signals to the location of the speakers. The model ensures that the audio signals are heard by the listener near their intended spatial location, and the proximity term controls which speakers are used to achieve this spatial impression. In particular, the proximity term favors the activation of speakers close to the desired perceived spatial location of the audio signals. For both CMAP and FV, this functional relationship is conveniently derived from a cost function written as a sum of two terms, one for the spatial aspect and one for the proximity:

number

[0085] Here, the set

number

number

[0086] In some definitions of the cost function, g opt While the relative levels between the components of are appropriate, it is difficult to control the absolute levels of optimal activations resulting from the above minimization. To address this issue, g is adjusted so that the absolute levels of activations are controlled. opt A subsequent normalization of may be performed. For example, normalizing vectors to have unit length may be desirable, which aligns with the commonly used constant power pan rule:

number

[0087] The exact behavior of the flexible rendering algorithm is determined by the two terms of the cost function C spatial and C proximity For CMAP, C spatial is the perceived spatial location of an audio signal reproduced from a set of loudspeakers, weighted by the locations of those loudspeakers and their associated activation gains (elements of vector g), given by the center of mass:

number

[0088] Equation 3 is then manipulated into a spatial cost that represents the square of the error between the desired audio position and the audio position produced by the active loudspeakers:

number

[0089] In FV, the spatial terms of the cost function are defined differently. Here, the goal is to generate a binaural response b corresponding to an audio object position →o at the listener's left and right ear. Conceptually, b is a 2×1 vector of filters (one filter for each ear), but is more conveniently treated as a 2×1 vector of complex values ​​at a particular frequency. Continuing with this representation at a particular frequency, the desired binaural response can be obtained from a set of HRTF indices according to the object position:

number

[0090] Together, the 2 × 1 binaural response e produced by the loudspeakers at the listener's ears is modeled as a 2 × M acoustic transfer matrix H multiplied by an M × 1 vector g of complex speaker activation values:

number

[0091] The acoustic transfer matrix H is the set of loudspeaker positions relative to the listener positions, {→s i Finally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation 5) and the binaural response produced by the loudspeaker (Equation 6):

number

[0092] Conveniently, the spatial terms of the CMAP and FV cost functions defined in Equation 4 and Equation 7 can both be reconstructed into a quadratic form of the matrix as a function of the speaker activations g:

number

[0093] To this end, the second term of the cost function, C proximity can be defined as the distance-weighted sum of the squared absolute values ​​of the speaker activations, which can be succinctly expressed in matrix form as follows:

number

number

number

[0094] The distance penalty function can take many forms, but the following is a useful parameterization:

number

number

[0095] Combining the two terms of the cost function defined in Equations 8 and 9a gives the overall cost function.

number

number

number

number

[0096] In general, the optimal solution of Equation 11 may result in speaker activations that are negative in value. For the CMAP construction of a flexible renderer, such negative activations may be undesirable, and thus Equation 11 may be minimized under the condition that all positive activations remain positive.

[0097] 2C and 2D are diagrams illustrating an exemplary collection of an exemplary set of speaker activations and object rendering positions. In these examples, the speaker activations and object rendering positions correspond to speaker positions of 4, 64, 165, -87, and -4 degrees. FIG. 2C shows speaker activations 245a, 250a, 255a, 260a, and 265a that constitute the optimal solution to Equation 11 for these particular speaker positions. FIG. 2D plots the individual speaker positions as boxes 267, 270, 272, 274, and 275 that correspond to speaker activations 245a, 250a, 255a, 260a, and 265a, respectively. FIG. 2D also shows ideal object positions (in other words, positions where audio objects should be rendered) for a number of possible object angles as dots 276a, and the corresponding actual rendering positions for those objects as dots 278a connected to the ideal object positions by dotted lines 279a.

[0098] One class of embodiments involves a method of rendering audio for playback by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) smart audio devices. For example, a set of smart audio devices in a user's home (a system in a user's home) may be orchestrated to handle a variety of simultaneous use cases. Such use cases include rendering (according to an embodiment) audio for playback by all or some of the smart audio devices (i.e., by all or some speakers). Many interactions with the system are contemplated that require dynamic modifications to the rendering. Such modifications may, but do not necessarily, focus on spatial fidelity.

[0099] Some embodiments are methods for rendering audio for playback by at least one (e.g., all or a portion) of a set of smart audio devices (or for playback by at least one (e.g., all or a portion) of a different set of speakers). The rendering may include minimization of a cost function, the cost function including at least one dynamic speaker activation term. Examples of such dynamic speaker activation terms include (but are not limited to): · Proximity of the speaker to the listener or listeners; ·The proximity of the speaker to the forces of attraction or repulsion; · Audibility of the loudspeaker with respect to some position (e.g. the listener's position or the baby's room); · Speaker capabilities (frequency response, distortion); · Synchronization of speakers relative to other speakers; Wake word performance; and Echo cancellation performance.

[0100] The dynamic speaker activation term may enable at least one of a variety of behaviors, including distorting the spatial presentation of audio away from a particular smart audio device to allow its microphone to better hear a speaker or to allow a secondary audio stream to better be heard from a speaker of the smart audio device.

[0101] Some embodiments include Implement rendering for playback over speakers of multiple orchestrated smart audio devices. Other embodiments implement rendering for playback over a speaker or speakers of a separate collection of speakers.

[0102] Pairing the flexible rendering method (implemented according to some embodiments) with a collection of wireless smart speakers (or other smart audio devices) can provide an extremely capable and easy-to-use spatial audio rendering system. When considering interactions with such a system, it becomes apparent that dynamic modifications to the spatial rendering may be desirable to optimize for other objectives that may arise during use of the system. To this end, one class of embodiments augments existing flexible rendering algorithms with one or more additional dynamically configurable features that depend on one or more attributes of the audio signal being rendered, the collection of speakers, and / or other external inputs. According to some embodiments, the existing flexible rendering cost function given in Equation 1 is augmented with these one or more additional dependencies, as follows:

number

[0103] In Equation 12, the term

number

number

number

number

number

number

number

number

[0104]

number

[0105]

number

[0106]

number

[0107] Using the new cost function defined in Equation 12, we can find the optimal set of activations through minimization with respect to g and possible a posteriori regularization, as previously described in Equations 2a and 2b.

[0108] FIG 2E is a flow diagram outlining one example of a method that may be performed by a device or system such as that shown in FIG 1A. The blocks of method 280, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. The blocks of method 280 may be performed by one or more devices that may be (or may include) a control system, such as control system 110 shown in FIG 1A.

[0109] In this implementation, block 285 involves receiving audio data by a control system via an interface system. In this example, the audio data includes one or more audio signals and associated spatial data. According to this implementation, the spatial data indicates an intended perceived spatial location corresponding to the audio signal. In some cases, the intended perceived spatial location may be explicit, e.g., indicated by positional metadata such as Dolby Atmos positional metadata. In other cases, the intended perceived spatial location may be implicit, e.g., the intended perceived spatial location may be an assumed location associated with a channel according to Dolby 5.1, Dolby 7.1, or other channel-based audio format. In some examples, block 285 involves a rendering module of the control system receiving the audio data via an interface system.

[0110] According to this example, block 290 involves rendering the audio data by the control system to generate rendered audio signals for playback through a set of loudspeakers in the environment. In this example, rendering each of one or more audio signals included in the audio data involves determining a relative activation of a set of loudspeakers in the environment by optimizing a cost function. According to this example, the cost is a function of a model of a perceived spatial location of the audio signal when played on a set of loudspeakers in the environment. In this example, the cost is also a function of a measure of the proximity of the intended perceived spatial location of the audio signal to the location of each loudspeaker in the set of loudspeakers. In this implementation, the cost is also a function of one or more additional dynamically configurable features. In this example, the dynamically configurable features are based on one or more of the following: the proximity of a loudspeaker to one or more listeners; the proximity of a loudspeaker to an attraction position, where attraction is a factor that favors relatively higher activation of loudspeakers closer to an attraction position; the proximity of a loudspeaker to a repulsion position, where repulsion is a factor that favors relatively lower activation of loudspeakers closer to a repulsion position; the capabilities of each loudspeaker relative to other loudspeakers in the environment; synchronization of a loudspeaker relative to other loudspeakers; wake word performance; or echo cancellation performance.

[0111] In this example, block 295 involves providing the rendered audio signals via an interface system to at least some loudspeakers of the set of loudspeakers in the environment.

[0112] According to some examples, the model of perceived spatial location can generate a binaural response corresponding to audio object positions at the left and right ears of a listener. Alternatively or additionally, the model of perceived spatial location can place the perceived spatial location of an audio signal reproduced from a set of loudspeakers at the center of mass of the positions of said set of loudspeakers weighted by the associated activation gains of the loudspeakers.

[0113] In some examples, the one or more additional dynamically configurable functions may be based, at least in part, on a level of the one or more audio signals, hi some instances, the one or more additional dynamically configurable functions may be based, at least in part, on a spectrum of the one or more audio signals.

[0114] Some examples of method 280 involve receiving speaker layout information. In some examples, the one or more additional dynamically configurable features may be based, at least in part, on the location of each loudspeaker in the environment.

[0115] Some examples of method 280 involve receiving loudspeaker specification information. In some examples, the one or more additional dynamically configurable features may be based, at least in part, on the capabilities of each loudspeaker, which may include one or more of a frequency response, playback level limits, or parameters of one or more loudspeaker dynamics processing algorithms.

[0116] According to some examples, the one or more additional dynamically configurable functions may be based, at least in part, on measurements or estimates of acoustic transmission from each loudspeaker to the other loudspeakers. Alternatively or additionally, the one or more additional dynamically configurable functions may be based, at least in part, on the positions of one or more human listeners or speakers in the environment. Alternatively or additionally, the one or more additional dynamically configurable functions may be based, at least in part, on measurements or estimates of acoustic transmission from each loudspeaker to the listener or speaker positions. The estimates of acoustic transmission may be based, for example, at least in part, on walls, furniture or other objects that may be between each loudspeaker and the listener or speaker positions.

[0117] Alternatively or additionally, the one or more additional dynamically configurable features may be based, at least in part, on object positions of one or more non-loudspeaker objects or landmarks in the environment. In some such implementations, the one or more additional dynamically configurable features may be based, at least in part, on measurements or estimates of acoustic transmission from each loudspeaker to object or landmark positions.

[0118] By employing one or more well-defined additional cost terms to achieve flexible rendering, many new useful behaviors can be achieved. All the example behaviors listed below are created in the form of penalizing certain loudspeakers under certain conditions that are considered undesirable. The end result is that these loudspeakers are activated less in the spatial rendering of said set of audio signals. In many of these cases, one may consider simply making the undesirable loudspeakers smaller, independent of any modification of the spatial rendering, but such a strategy may seriously degrade the overall balance of the audio content. Certain components of the mix may, for example, become completely inaudible. On the other hand, in the disclosed embodiment, by integrating these penalizations into the core optimization of the rendering, the rendering can adapt and perform the best possible spatial rendering with the less penalized remaining speakers. This is a much more elegant, adaptive and effective solution.

[0119] Exemplary use cases include, but are not limited to, the following:

[0120] Provides a more balanced spatial presentation around the listening area It has been found that spatial audio is best presented through loudspeakers that are approximately the same distance from the intended listening area. Costs may be constructed such that loudspeakers that are significantly closer or farther away than the average distance of the loudspeakers to the listening area are penalized, thereby reducing their activation.

[0121] ● Moving the audio away from or towards the listener or speaker o If a user of the system is going to talk to the system's or an associated smart voice assistant, it is beneficial to make the cost penalize loudspeakers that are closer to the speaker, so that these loudspeakers are activated less and the associated microphones can better hear the speaker. o To provide a more private experience for a single listener that minimizes playback levels for other listeners in the listening space, speakers farther from the listener's position may be heavily penalized, so that only the speakers closest to the listener are activated most prominently.

[0122] Move the audio away from or closer to a landmark, zone, or area o Certain locations in the vicinity of the listening space may be considered sensitive, e.g. baby rooms, cribs, offices, reading areas, study areas, etc. In such cases, a cost may be constructed that penalizes the use of speakers close to this location, zone or area. Alternatively, for the same (or similar) case as above, the system of speakers may generate measurements of the sound transmission from each speaker to the baby room, especially if one of the speakers (with an attached or associated microphone) is within the baby room itself. In this case, rather than using the physical proximity of the speakers to the baby room, a cost may be constructed that penalizes the use of speakers with high measured sound transmission to the baby room. And / or

[0123] ● Optimal use of speaker capabilities ○ The capabilities of different loudspeakers can vary significantly. For example, one popular smart speaker only contains a single 1.6 inch full-range driver with limited bass capabilities. Another smart speaker contains a much more capable 3 inch woofer. These capabilities are generally reflected in the frequency response of the speaker, and thus the set of responses associated with the speaker can be utilized in cost terms. At certain frequencies, speakers that are less capable than other speakers as measured by their frequency response are penalized and therefore activated to a lesser extent. In some implementations, such frequency response values ​​may be stored in the smart loudspeaker and then reported to a computational unit responsible for optimizing the flexible rendering.

[0124] Many speakers contain multiple drivers, each responsible for reproducing a different frequency range. For example, some popular smart speakers are two-way designs that include a woofer for low frequencies and a tweeter for high frequencies. Typically, such speakers contain crossover circuitry to split the full-range playback audio signal into appropriate frequency ranges and send them to the respective drivers. Alternatively, such speakers can provide the Flexible Renderer playback access to each individual driver, as well as information about each individual driver's capabilities, such as frequency response. By applying cost terms such as those described above, in some instances the Flexible Renderer can automatically construct a crossover between the two drivers based on their relative capabilities at different frequencies.

[0125] o The above use cases of frequency response focus on the inherent capabilities of the speaker, but may not accurately reflect the capabilities of the speaker placed in the listening environment. In some cases, the frequency response of the speaker measured at the intended listening position may be available through some calibration procedure. Such measurements may be used instead of pre-calculated responses to better optimize the use of the speaker. For example, some speakers may be inherently very capable at certain frequencies, but because of their placement (e.g., behind a wall or furniture) may produce a very limited response at the intended listening position. Capturing this response and inputting the measurements into the appropriate cost terms can prevent significant activation of such speakers.

[0126] o Frequency response is only one aspect of a loudspeaker's reproduction capabilities. Many small loudspeakers start to distort as the reproduction level gets higher and then reach an excursion limit, especially for low frequencies. To reduce such distortion, many loudspeakers implement dynamics processing that constrains the reproduction level below some limiting thresholds that may be variable over frequency. If a speaker is close to or at these thresholds and other speakers participating in the flexible rendering are not, it makes sense to reduce the signal level of the limiting speaker and direct this energy to other, less burdened speakers. Such behavior can be achieved automatically according to some embodiments by appropriately configuring the associated cost terms. Such cost terms may involve one or more of the following: Monitoring of the global playback volume relative to a loudspeaker's limiting threshold. For example, loudspeakers whose volume levels are closer to their limiting threshold may be penalized more heavily; monitoring of dynamic signal levels, possibly varying over frequency, in relation to a loudspeaker limiting threshold, also possibly varying over frequency. For example, loudspeakers whose monitored signal levels are closer to their limiting thresholds may be penalized more; Direct monitoring of loudspeaker dynamics processing parameters, such as limiting gain. In some such instances, loudspeakers whose parameters indicate stronger limiting may be penalized more heavily; and / or Monitoring the actual instantaneous voltage, current and power being delivered by the amplifier to the loudspeaker to determine if the loudspeaker is operating in its linear range. For example, loudspeakers operating with less linearity may be penalized more.

[0127] Smart speakers with integrated microphones and interactive voice assistants typically use some type of echo cancellation to reduce the level of the audio signal played from the speaker that is picked up by the recording microphone. The greater this reduction, the more likely the speaker is to hear and understand the talker in the space. If the echo canceller residual is consistently high, this may be an indication that the speaker is being driven into a nonlinear region where the echo path becomes difficult to predict. In such cases, it makes sense to divert signal energy away from that speaker, and thus a cost term that takes echo canceller performance into account may be beneficial. Such a cost term may assign a higher cost to speakers whose associated echo cancellers perform poorly.

[0128] o To achieve predictable imaging when rendering spatial audio with multiple loudspeakers, it is generally necessary that the playback on a set of loudspeakers be reasonably synchronized over time. For wired loudspeakers, this is of course the case, but when there are many wireless loudspeakers, synchronization can be difficult and the end result can be variable. In such cases, it may be possible for each loudspeaker to report its relative degree of synchronization with the target, and this degree may be input into the synchronization cost term. In some such instances, loudspeakers with lesser degrees of synchronization may be penalized more heavily and thus excluded from the rendering. Furthermore, strict synchronization may not be required for certain types of audio signals, e.g., components of an audio mix that are intended to be diffuse or non-directional. In some implementations, components may be tagged as such using metadata, and the synchronization cost term may be modified such that the penalty is reduced.

[0129] Next, example embodiments will be described.

[0130] Similar to the proximity costs defined in Equations 9a and 9b, the terms in the new cost function

number

number

number

number

[0131] Combining Equations 13a and 13b with a matrix-quadratic version of the CMAP and FV cost functions given in Equation 10 yields a potentially useful implementation of the general extended cost function (of some embodiments) given in Equation 12:

number

[0132] With this definition of the new cost function term, the overall cost function remains a matrix quadratic form, and the optimal set of activations g opt can be found through differentiation of Equation 14, resulting in:

number

[0133] Weight term w ij for each of the loudspeakers, given successive penalty values

number

number

number

number

[0134] If all loudspeakers are penalized, then in post-processing it is often convenient to subtract a minimum penalty from all weight terms to ensure that at least one of the speakers is not penalized:

number

[0135] As mentioned above, there are many possible use cases that can be realized using the new cost function terms described herein (and similar new cost function terms used according to other embodiments). Three examples are now provided to provide more specific details: moving audio towards a listener or speaker, moving audio away from a listener or speaker, and moving audio away from a landmark.

[0136] In a first example, what is referred to herein as a "gravitational force" is used to pull audio towards a location. That location may be, in some examples, a listener or speaker location, a landmark location, a furniture location, etc. In the present specification, this location may be referred to as an "attraction position" or an "attractor position." As used herein, "attraction" is a factor that favors relatively higher loudspeaker activation in the vicinity closer to the attraction position. According to this example, a weight w ij takes the form of Equation 17, and the continuous penalty value p ij is the fixed attractor position of the i-th speaker

number

number

[0137] To illustrate the use case of "pulling" audio towards a listener or speaker, specifically, j = 20, β j Set =3,

number

number

[0138] In the second and third examples, a "repulsion force" is used to "push" the audio away from a location, which may be a listener's location, a speaker's location, or other location, such as a landmark location, a piece of furniture, etc. In some examples, a repulsion force may be used to push the audio away from an area or zone of an auditory environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or bedroom). According to some such examples, a particular location may be used as representative of a zone or area. For example, a location representative of an infant's bed may be an estimated location of the infant's head, an estimated sound source location corresponding to the infant, etc. This location may be referred to herein as a "repulsion force location" or "repulsion location." As used herein, a "repulsion force" is a factor that promotes relatively lower speaker activation closer to the repulsion force location. According to this example, a fixed repulsion location may be used to push the audio away from an area or zone of an auditory environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or bedroom). According to this example, a fixed repulsion location may be used to push the audio away from an area or zone of an auditory environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or bedroom), an estimated sound source location corresponding to the infant, etc. This location may be referred to herein as a "repulsion force location" or "repulsion location." As used herein, a "repulsion force" is a factor that promotes relatively lower speaker activation closer to the repulsion force location. According to this example, a fixed repulsion location may be used to push the audio away from an area or zone of an auditory environment, such as an office area, a reading area, a bed or sleeping area (e.g., a crib or bedroom), an estimated sound source location corresponding to the infant, etc. According to this example, a fixed repulsion location may be used to push the audio away from an area or zone of an auditory environment, such as an office area, a reading area

number

number

[0139] To illustrate the use case of moving audio away from the listener or speaker, specifically, j = 5, β j Set =2,

number

number

[0140] A third example use case is to "push" audio away from an acoustically sensitive landmark, such as a door to a sleeping baby's room. Similar to the previous example, →l j to the vector corresponding to the 180 degree door position (bottom center of the plot). To achieve a stronger repulsion force and to completely bias the sound field towards the front part of the primary listening space, we set α j = 20, β j =5. FIG. 2J is a graph of speaker activations in an example embodiment. Again, in this example, FIG. 2J shows speaker activations 245d, 250d, 255d, 260d, and 265d that constitute an optimal solution to the same set of speaker positions, but with stronger repulsion forces. FIG. 2K is a graph of object rendering positions in an example embodiment. Again, in this example, FIG. 2K shows ideal object positions 276d for a number of possible object angles, and the corresponding actual rendering positions 278d for those objects connected to the ideal object positions 276d by dotted lines 279d. The skewed orientation of the actual rendering positions 278d shows the effect of stronger repulsion weighting on the optimal solution to the cost function.

[0141] Returning now to FIG. 2B, in this example, block 225 involves modifying the rendering process for the first audio signal based, at least in part, on at least one of the second audio signal, the second rendered audio signal, or the characteristics thereof to generate a modified first rendered audio signal. Various examples of modifying the rendering process are disclosed herein. The "characteristics" of the rendered signal may include, for example, the estimated or measured loudness or audibility at the intended listening position in a silent environment or in the presence of one or more additional rendered signals. Other examples of characteristics include parameters related to the rendering of the signal, such as the intended spatial location of the constituent signals of the associated program stream, the location of the loudspeakers at which the signal is rendered, the relative activation of the loudspeakers as a function of the intended spatial location of the constituent signals, and any other parameters or conditions related to the rendering algorithm utilized to generate the rendered signal. In some examples, block 225 may be executed by a first rendering module.

[0142] According to this example, block 230 involves modifying a rendering process for the second audio signal based, at least in part, on at least one of the first audio signal, the first rendered audio signal, or characteristics thereof to generate a modified second rendered audio signal. In some examples, block 230 may be performed by a second rendering module.

[0143] In some implementations, modifying the rendering process for the first audio signal may include warping the rendering of the first audio signal away from a rendering location of the second rendered audio signal and / or modifying a loudness of one or more of the first rendered audio signals in response to a loudness of the second audio signal or one or more of the second rendered audio signals. Alternatively or additionally, modifying the rendering process for the second audio signal may include warping the rendering of the second audio signal away from a rendering location of the first rendered audio signal and / or modifying a loudness of one or more of the second rendered audio signals in response to a loudness of the first audio signal or one or more of the first rendered audio signals. Some examples are provided below with reference to FIG. 3 et seq.

[0144] However, other types of rendering process modifications are within the scope of this disclosure. For example, in some examples, modifying the rendering process for the first audio signal or the second audio signal may involve performing spectral modifications, audibility-based modifications, or dynamic range modifications. These modifications may or may not involve loudness-based rendering modifications, depending on the specific example. For example, in the above-mentioned case where a primary spatial stream is rendered in an open plan living area and a secondary stream consisting of cooking tips is rendered in an adjacent kitchen, it may be desirable to ensure that the cooking tips remain audible in the kitchen. This can be achieved by estimating what the loudness would be for the rendered cooking tips stream in the kitchen in the absence of an interfering first signal, then estimating said loudness in the presence of the first signal in the kitchen, and finally dynamically modifying the loudness and dynamic range of both streams across multiple frequencies to ensure audibility of the second signal in the kitchen.

[0145] In the example shown in Figure 2B, block 235 involves mixing at least the modified first rendered audio signal and the modified second rendered audio signal to generate a mixed audio signal, which may be performed, for example, by mixer 130b shown in Figure 2A.

[0146] According to this example, block 240 involves providing the mixed audio signal to at least some speakers in the environment. Some examples of method 200 involve playing the mixed audio signal through speakers.

[0147] As shown in FIG. 2B, some implementations may provide more than two rendering modules. Some such implementations may provide N rendering modules, where N is an integer greater than two. Thus, some such implementations may include one or more additional rendering modules. In some such examples, each of the one or more additional rendering modules may be configured to receive an additional audio program stream via an interface system. The additional audio program stream may include an additional audio signal scheduled for playback by at least one speaker of the environment. Some such implementations may involve rendering the additional audio signal for playback via at least one speaker of the environment to generate an additional rendered audio signal, and modifying a rendering process for the additional audio signal based, at least in part, on at least one of the first audio signal, the first rendered audio signal, the second audio signal, the second rendered audio signal, or characteristics thereof to generate a modified additional rendered audio signal. According to some such examples, the mixing module may be further configured to mix a modified additional rendered audio signal with at least the modified first rendered audio signal and the modified second rendered audio signal to generate the mixed audio signal.

[0148] As described above with reference to FIGS. 1A and 2A, some implementations may include a microphone system including one or more microphones in a listening environment. In some such examples, the first rendering module may be configured to modify a rendering process for the first audio signal based at least in part on a first microphone signal from the microphone system. The "first microphone signal" may be received from a single microphone or from two or more microphones, depending on the specific implementation. In some such implementations, the second rendering module may be configured to modify a rendering process for the second audio signal based at least in part on the first microphone signal.

[0149] As described above with reference to FIG. 2A, in some examples, the location of one or more microphones may be known and provided to the control system. According to some such implementations, the control system may be further configured to estimate a first sound source location based on the first microphone signal and modify a rendering process for at least one of the first audio signal or the second audio signal based at least in part on the first sound source location. The first sound source location may be estimated according to a triangulation process, for example, based on DOA data from each of three or more microphones or groups of microphones having known locations. Alternatively or additionally, the first sound source location may be estimated according to the amplitude of received signals from two or more microphones. The microphone generating the highest amplitude signal may be assumed to be closest to the first sound source location. In some such examples, the first sound source location may be set to the location of the closest microphone. In some such examples, the first sound source location may be associated with a location of a zone, where the zone is selected by processing signals from two or more microphones through a pre-trained classifier, such as a Gaussian mixer model.

[0150] In some such implementations, the control system may be configured to determine whether the first microphone signal corresponds to environmental noise. Some such implementations may involve modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to environmental noise. For example, if the control system determines that the first microphone signal corresponds to environmental noise, modifying the rendering process for the first audio signal or the second audio signal may involve increasing a level of the rendered audio signal such that a perceived loudness of the signal in the presence of noise at the intended listening position is substantially equal to a perceived loudness of the signal in the absence of noise.

[0151] In some examples, the control system may be configured to determine whether the first microphone signal corresponds to a human voice. Some such implementations may involve modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to a human voice. For example, if the control system determines that the first microphone signal corresponds to a human sound, such as a wake word, modifying the rendering process for the first audio signal or the second audio signal may involve decreasing a loudness of a rendered audio signal reproduced by a speaker near the first audio source position compared to a loudness of a rendered audio signal reproduced by a speaker farther from the first audio source position. Modifying the rendering process for the first audio signal or the second audio signal may alternatively include warping an intended location of a constituent signal of the associated program stream away from the first audio source position and / or modifying the rendering process to penalize the use of speakers near the first audio source position compared to speakers further from the first audio source position.

[0152] In some implementations, if the control system determines that the first microphone signal corresponds to a human voice, the control system may be configured to play the first microphone signal on one or more speakers proximate a location in the environment that is different from the first sound source location. In some such examples, the control system may be configured to determine whether the first microphone signal corresponds to a child's crying. According to some such implementations, the control system may be configured to play the first microphone signal on one or more speakers proximate a location in the environment that corresponds to an estimated location of a caregiver, such as a parent, relative, guardian, child care provider, teacher, nurse, etc. In some examples, the process of estimating the estimated location of the caregiver may be triggered by a voice command such as "<wake word>, don't wake the baby". The control system may estimate the location of the speaker (caregiver) according to the location of the nearest smart audio device implementing a virtual assistant by triangulation based on DOA information provided by three or more local microphones, etc. According to some implementations, the control system has prior knowledge of the location of the baby's room (and / or listening devices therein) and can therefore perform appropriate processing.

[0153] According to some such examples, the control system may be configured to determine whether the first microphone signal corresponds to a command. If the control system determines that the first microphone signal corresponds to the command, in some cases, the control system may be configured to determine a response to the command and control at least one speaker proximate the first audio source location to play the response. In some such examples, the control system may be configured to return to an unmodified rendering process for the first audio signal or the second audio signal after controlling the at least one speaker proximate the first audio source location to play the response.

[0154] In some implementations, the control system may be configured to execute the commands. For example, the control system may be or include a virtual assistant configured to control audio devices, televisions, home appliances, etc. according to the commands.

[0155] With this definition of the minimal and more capable multi-stream rendering systems shown in Figures 1A, 1B, and 2A, dynamic management of the simultaneous playback of multiple program streams can be achieved for a number of useful scenarios. Some examples will now be described with reference to Figures 3A and 3B.

[0156] Let us first examine the example discussed above involving the simultaneous playback of a spatial movie soundtrack in a living room and cooking tips in a connected kitchen. The spatial movie soundtrack is an example of the "first audio program stream" mentioned above, and the cooking tips audio is an example of the "second audio program stream" mentioned above. Figures 3A and 3B show an example of a floor plan of a connected living space. In this example, living space 300 includes a living room at the top left, a kitchen at the bottom center, and a bedroom at the bottom right. The squares and circles 305a-305h distributed throughout the living space represent a set of eight loudspeakers conveniently placed in the space but not conforming to a standard prescribed layout (arbitrarily placed). In Figure 3A, only the spatial movie soundtrack is being played, and all loudspeakers in the living room 310 and kitchen 315 are utilized to generate an optimized spatial playback around listener 320a seated on couch 325 facing television 330, taking into account the capabilities and layout of the loudspeakers. Optimal playback of this movie soundtrack is visually represented by a cloud 335a within range of the active loudspeakers.

[0157] In FIG. 3B, cooking tips are simultaneously rendered and played for a second listener 320b through a single loudspeaker 305g in the kitchen 315. This playback of the second program stream is visually represented by a cloud 340 emanating from the loudspeaker 305g. If these cooking tips were played simultaneously without modifications to the movie soundtrack rendering, as shown in FIG. 3A, audio from the movie soundtrack emanating from speakers in or near the kitchen 315 would interfere with the second listener's ability to understand the cooking tips. Instead, in this example, the spatial movie soundtrack rendering is dynamically modified as a function of the cooking tips rendering. Specifically, the movie soundtrack rendering is shifted away from the speakers near the cooking tips rendering location (the kitchen 315), and this shift is visually represented by a smaller cloud 335b in FIG. 3B, which is pushed away from the speakers near the kitchen. If the cooking tips stop playing while the movie soundtrack is still playing, in some implementations the movie soundtrack rendering can dynamically shift back to the original optimal configuration seen in FIG. 3A. Such dynamic shifts in the spatial movie soundtrack rendering can be achieved through a number of disclosed methods. Many spatial audio mixes contain multiple component audio signals designed to be played at specific locations in the listening space. For example, Dolby 5.1 and 7.1 surround sound mixes consist of 6 and 8 signals, respectively, that are intended to be played on speakers at defined canonical positions around the listener. Object-based audio formats, such as Dolby Atmos, consist of component audio signals and associated metadata that describe the potentially time-varying 3D locations in the listening space where the audio is intended to be rendered.Given that a spatial movie soundtrack renderer can render individual audio signals at any position with respect to any set of loudspeakers, a dynamic shift to the rendering shown in Figures 3A and 3B can be achieved by warping the intended positions of the audio signals in the spatial mix. For example, the 2D or 3D coordinates associated with the audio signals are pushed away from the positions of the speakers in the kitchen or pulled towards the upper left corner of the living room. The result of such a warp is that the speakers near the kitchen are used less because the warped positions of the audio signals in the spatial mix are now further away from this position. This method achieves the goal of making the second audio stream more intelligible to the second listener, but at the cost of significantly altering the intended spatial balance of the movie soundtrack for the first listener.

[0158] A second method for achieving a dynamic shift to spatial rendering may be realized by using a flexible rendering system. In some such implementations, the flexible rendering system may be a CMAP, FV, or a hybrid of both, as described above. Some such flexible rendering systems attempt to reproduce a spatial mix such that all component signals are perceived to come from their intended locations. While doing so for each signal in the mix, in some examples activation of loudspeakers close to the desired location of that signal is prioritized. In some implementations, additional terms may be dynamically added to the rendering optimization, which penalizes the use of certain loudspeakers based on other criteria. In the present example, what may be called a "repulsion force" may be dynamically placed at a kitchen location to highly penalize the use of loudspeakers near this location, effectively distancing the rendering of a spatial movie soundtrack. As used herein, the term "repulsion force" may refer to a factor that corresponds to relatively low speaker activation in a particular location or area of ​​a listening environment. In other words, the phrase "repulsion" may refer to a factor that favors activation of speakers that are relatively farther from a particular location or area corresponding to the "repulsion," but according to some such implementations, the renderer may still attempt to recreate the intended spatial balance of the mix using the remaining, lower-penalized speakers. As such, this technique may be considered a superior way to achieve dynamic shifts in rendering compared to methods that simply warp the intended locations of the component signals of the mix.

[0159] The above-mentioned scenario of shifting the spatial movie soundtrack rendering away from the cooking hints in the kitchen can be achieved using a minimal version of the multi-stream renderer shown in FIG. 1B. However, improvements to the scenario can be realized by employing a more capable system, shown in FIG. 2A. Although shifting the spatial movie soundtrack rendering improves the intelligibility of the cooking hints in the kitchen, the movie soundtrack may still be prominently audible in the kitchen. Depending on the instantaneous state of both streams, the cooking hints may end up being masked by the movie soundtrack. For example, loud moments in the movie soundtrack mask quieter moments in the cooking hints. To address this issue, dynamic modifications to the cooking hint rendering as a function of the spatial movie soundtrack rendering may be added. For example, a method may be implemented to dynamically modify the audio signal over frequency and time to maintain its perceived loudness in the presence of an interfering signal. In this scenario, an estimate of the perceived loudness of the shifted movie soundtrack at the kitchen location may be generated and fed into such a process as an interfering signal. The time- and frequency-varying levels of the cooking tips may then be dynamically modified to maintain their perceived loudness above this interference, thereby better maintaining intelligibility to the second listener. The required loudness estimate of the movie soundtrack in the kitchen may be generated from a speaker feed of the soundtrack rendering, a signal from a microphone in or near the kitchen, or a combination thereof. The process of maintaining the perceived loudness of the cooking tips generally increases the level of the cooking tips, and in some cases the overall loudness may become uncomfortably high. To address this issue, further rendering modifications may be employed. The interfering spatial movie soundtrack may be dynamically turned down in response to the loudness of the loudness-modified cooking tips in the kitchen becoming too loud.Finally, some external noise source may interfere with the audibility of both program streams simultaneously. For example, a blender may be used in the kitchen while cooking. An estimate of the loudness of this environmental noise source in both the living room and the kitchen may be generated from a microphone connected to the rendering system. This estimate may for example be added to the estimate of the loudness of the soundtrack in the kitchen, which affects the loudness modification of the cooking tips. At the same time, the rendering of the soundtrack in the living room may be additionally modified as a function of the environmental noise estimate in order to maintain the perceived loudness of the soundtrack in the living room in the presence of this environmental noise, thereby better preserving audibility for the listener in the living room.

[0160] As can be seen, this exemplary use case of the disclosed multi-stream renderer employs a number of interconnected modifications to two program streams in order to optimize the simultaneous playback of the two program streams. In summary, these modifications to the streams can be enumerated as follows: Spatial film soundtracks ○ As a function of cooking cues rendered in the kitchen, the spatial rendering is shifted away from the kitchen Dynamic reduction in loudness as a function of the loudness of cooking cues rendered in the kitchen Dynamically increase loudness as a function of the estimated loudness in the living room of interfering blender noise from the kitchen ●Cooking tips Dynamically increase the loudness as a function of the combined loudness estimates of both the movie soundtrack and the blender noise in the kitchen.

[0161] A second exemplary use case of the disclosed multi-stream renderer involves the simultaneous playback of a spatial program stream, such as music, along with the smart voice assistant's response to some query by the user. In existing smart speakers, playback is generally constrained to mono or stereo playback through a single device, and an interaction with a voice assistant typically consists of the following stages: 1) Music playback 2) The user speaks the wake word for the voice assistant. 3) The smart speaker recognizes the wake word and significantly reduces the volume of the music (ducks). 4) The user speaks a command to the smart assistant (i.e. "play next song"). 5) The smart speaker recognises the command, confirms this by mixing some voice response (i.e. "Got it, play the next song") into the ducked music and playing it through the speaker, then carries out the command. 6) Your smart speaker will return the music to its original loudness.

[0162] 4A and 4B show an example of a multi-stream renderer that provides simultaneous playback of a spatial music mix and a voice assistant response. When playing spatial audio on multiple orchestrated smart speakers, some embodiments provide an improvement to the above sequence of events. Specifically, the spatial mix may be shifted away from one or more of the speakers appropriately selected to relay the response from the voice assistant. Creating this space for the voice assistant response means that the spatial mix does not have to be reduced as much, or at all, compared to the current state of the art listed above. FIG. 4A and FIG. 4B illustrate this scenario. In this example, the modified sequence of events occurs as follows: 1) A spatial music program stream is played through a number of orchestrated smart speakers for user cloud 335c in FIG. 4A. 2) User 320c speaks the wake word for the voice assistant. 3) One or more smart speakers (e.g., speaker 305d and / or speaker 305f) recognize the wake word using associated recordings from microphones associated with the one or more smart speakers and determine the location of user 320c or which speaker(s) user 320c is closest to. 4) The rendering of the spatial music mix is ​​shifted away from the location determined in the previous step in anticipation of the voice assistant response program stream being rendered near that location (cloud 335d in Figure 4B). 5) A user speaks a command to a smart assistant (e.g., a smart speaker running smart assistant / virtual assistant software). 6) The smart speaker recognizes the command, synthesizes a corresponding response program stream, and renders the response near the user's location (cloud 440 in FIG. 4B). 7) The rendering of the spatial music program stream shifts back to its original state (cloud 335c in Figure 4A) once the voice assistant response is complete.

[0163] In addition to optimizing the simultaneous playback of the spatial music mix and voice assistant responses, shifting the spatial music mix can also improve the ability of the set of speakers to understand the listener in step 5. This is because the music is shifted away from speakers closer to the listener, thereby improving the voice-to-other ratio of the associated microphones.

[0164] Similar to what was described for the previous scenario with the spatial movie mix and cooking tips, the current scenario can be further optimized beyond what is provided by shifting the rendering of the spatial mix as a function of the voice assistant response. By itself, shifting the spatial mix may not be enough to make the voice assistant response fully understandable to the user. A simple solution is to make the spatial mix quieter by a certain amount, but not as loud as is required by the current state of the art. Alternatively, the loudness of the voice assistant response program stream can be dynamically increased as a function of the loudness of the spatial music mix program stream to maintain audibility of the response. As an extension, the loudness of the spatial music mix may also be dynamically cut if this boosting process for the response stream becomes too loud.

[0165] 5A, 5B, and 5C show a third example of the disclosed multiple stream renderer. This example involves managing the simultaneous playback of a spatial music mix program stream and a comfort noise program stream while trying to ensure that a baby remains asleep in the next room, yet be heard if the baby cries. FIG. 5A shows a starting point where the spatial music mix (represented by cloud 335e) is optimally played across all speakers in the living room 310 and kitchen 315 for a number of people at a party. In FIG. 5B, the baby 510 is trying to sleep in the adjacent bedroom 505 depicted at the bottom right. To ensure this, the spatial music mix is ​​dynamically shifted out of the bedroom to minimize leakage into the bedroom, as shown by cloud 335f, while still maintaining a reasonable experience for the people at the party. At the same time, a second program stream (represented by cloud 540) containing soothing white noise is played from speakers 305h in the baby's room to mask any remaining leakage from the music in the adjacent room. To ensure complete masking, in some examples, the loudness of this white noise stream may be dynamically modified as a function of an estimate of the loudness of the spatial music leaking into the baby's room. This estimate may be generated from a speaker feed of the spatial music rendering, a signal from a microphone in the baby's room, or a combination thereof. Also, the loudness of the spatial music mix may be dynamically attenuated as a function of the loudness modified noise if it becomes too loud. This is similar to the loudness processing between the spatial movie mix and the cooking tips in the first scenario. Finally, a microphone in the baby's room (e.g., a microphone associated with speaker 305h, which in some implementations may be a smart speaker) may be configured to record audio from the baby (canceling sounds that may be picked up from the spatial music and white noise), and the combination of these processed microphone signals may then serve as a third program stream.The third program stream may be simultaneously played near a listener 320d, who may be a parent or other caregiver, in the living room 310 if crying is detected (through machine learning, via a pattern matching algorithm, etc.). FIG. 5C illustrates the playback of this additional stream with cloud 550. In this case, the spatial music mix may be further shifted away from the speaker near the parent playing the baby's cry, as shown by the modified shape of cloud 335g relative to the shape of cloud 335f in FIG. 5B. The baby's cry program stream may be loudness modified as a function of the spatial music stream such that the baby's cry remains audible to listener 320d. The interconnected modifications that optimize the simultaneous playback of three program streams considered in this example can be summarized as follows:

[0166] Spatial music mix in the living room Spatial rendering is shifted away from the baby's room to reduce propagation into the baby's room. Dynamic reduction of loudness as a function of the loudness of the white noise rendered in the baby's room In response to the baby's cry being rendered to a speaker closer to the parent, the spatial rendering is shifted away from the parent. ●White noise Dynamic increase in loudness as a function of the estimated loudness of the music stream permeating the baby's room Recording of a baby crying Dynamic increase in loudness as a function of the loudness estimate of the music mix at the parent's or other caregiver's location.

[0167] Next, examples of how some of the above-described embodiments may be implemented are described.

[0168] In Figure 1B, each rendering block 1...N may be implemented as identical instances of any single-stream renderer, such as the CMAP, FV, or hybrid renderers mentioned above. Configuring a multi-stream renderer in this way has several convenient and useful properties.

[0169] First, when rendering is done in this hierarchical arrangement, and each of the single-stream renderer instances is configured to operate in the frequency / transform domain (e.g. QMF), the mixing of streams can also happen in the frequency / transform domain, and the inverse transform only needs to be performed once for the M channels - a significant efficiency improvement over performing NxM inverse transforms to mix in the time domain.

[0170] FIG. 6 shows a frequency / transform domain example of the multi-stream renderer shown in FIG. 1B. In this example, a quadrature mirror analysis filter bank (QMF) is applied to each of the program streams 1-N before each program stream is received by a corresponding one of the rendering modules 1-N. According to this example, the rendering modules 1-N operate in the frequency domain. After the mixer 630a mixes the outputs of the rendering modules 1-N, the inverse synthesis filter bank 635a transforms the mix to the time domain and provides a time-domain mixed speaker feed signal to the loudspeakers 1-M. In this example, the quadrature mirror filter bank, the rendering modules 1-N, the mixer 630a, and the inverse filter bank 635a are components of the control system 110c.

[0171] FIG. 7 shows an example of the frequency / transform domain of the multi-stream renderer shown in FIG. 2A. As in FIG. 6, a quadrature mirror analysis filter bank (QMF) is applied to each of the program streams 1-N before each program stream is received by a corresponding one of the rendering modules 1-N. According to this example, the rendering modules 1-N operate in the frequency domain. In this implementation, the time-domain microphone signals from the microphone system 120b are also provided to the quadrature mirror filter bank, and the rendering modules 1-N receive the microphone signals in the frequency domain. After the mixer 630b mixes the outputs of the rendering modules 1-N, the inverse filter bank 635b transforms the mix to the time domain and provides a time-domain mixed speaker feed signal to the loudspeakers 1-M. In this example, the quadrature mirror filter bank, the rendering modules 1-N, the mixer 630b, and the inverse filter bank 635b are components of the control system 110d.

[0172] Another advantage of the hierarchical approach in the frequency domain is the computation of the perceived loudness of each audio stream and using this information in dynamically modifying one or more of the other audio streams. To illustrate this embodiment, consider the example described above with reference to Figures 3A and 3B. In this case, there are two audio streams (N=2), a spatial movie soundtrack, and cooking tips. There may also be environmental noise generated by a blender in the kitchen, which is picked up by one or more of the K microphones.

[0173] Each audio stream s is rendered separately, captured by each microphone i, and transformed into the frequency domain, after which the source excitation signal E s or E iThis serves as a time-varying estimate of the perceived loudness of each audio stream s or microphone signal i. In this example, these source excitation signals are derived from the rendered stream or captured microphone, and the conversion coefficients X s , or X for microphone signals i is calculated for b frequency bands over time t for c loudspeakers via b is smoothed using;

number

[0174] The raw source excitation is the perceived loudness estimate of each stream at a particular location. For the spatial stream, that location is at the center of cloud 335b in FIG. 3B, while for the cooking tips stream, it is at the center of cloud 340. The location for the blender noise picked up by the microphones may be based, for example, on the particular location of the microphone(s) closest to the source of the blender noise.

[0175] The raw source excitation must be transformed to the listening positions of the audio stream(s) that it modifies in order to estimate how perceptible it is as noise at each listening position of the target audio stream. For example, if audio stream 1 is a movie soundtrack and audio stream 2 is cooking tips,

number

number

[0176] In formula 13a, ^E xs represents the raw noise excitation calculated on the source audio stream without reference to the microphone input. xi represents the raw noise excitation calculated with reference to the microphone input. According to this example, the raw noise excitation^E xs or ^E xi is then summed over streams 1 through N, microphones 1 through K, and output channels 1 through M to obtain a total noise estimate for the target stream x:

number

[0177] According to some alternative implementations, the term in Eq.

number

[0178] In this example, the entire raw noise estimate is smoothed to avoid perceptible artifacts that may be caused by modifying the target stream too abruptly. In this implementation, the smoothing is based on the concept of using a fast attack and slow release, similar to audio compressors. The smoothed noise estimate for target stream x

number

number

[0179] Once we have a perfect noise estimate for the stream x, x Once we have (b,t), we use the previously calculated source excitation signal E x (b,t,c) can be reused to determine a set of time-varying gains Gx(b,t,c) to apply to the target audio stream x to ensure that the target audio stream x remains audible above the noise. These gains can be computed using any of a variety of techniques.

[0180] In one embodiment, a loudness function L{·,·} can be applied to the excitation to model various nonlinearities in human loudness perception and compute a specific loudness signal that describes the time-varying frequency distribution of perceived loudness. Applying L{·,·} to the noise estimates and the excitation for the rendered audio stream x gives an estimate for the specific loudness of each signal:

number

[0181] In formula 17a, L xn represents an estimate of the specific loudness of the noise, and in Equation 17b, L xrepresents an estimate for the specific loudness of the rendered audio stream x. These specific loudness signals represent the perceived loudness when the signals are heard in isolation. However, when two signals are mixed, masking can occur. For example, if the noise signal is much louder than the stream x signal, it will mask the stream x signal, thereby reducing the perceived loudness of that signal relative to the perceived loudness of that signal heard in isolation. This phenomenon can be modeled with a partial loudness function PL{·,·} that takes two inputs. The first input is the excitation of the signal of interest, and the second input is the excitation of the competing (noise) signal. This function returns a partial specific loudness signal PL that represents the perceived loudness of the signal of interest in the presence of the competing signal. The partial specific loudness of the stream x signal in the presence of the noise signal can then be calculated directly from the excitation signal over frequency band b, time t, and loudspeaker c:

number

[0182] To maintain the audibility of the audio stream x signal in the presence of noise, a gain G is applied to audio stream x to increase its loudness until audio stream x is audible above the noise, as shown in Equations 8a and 8b. x (b,t,c). Alternatively, if the noise is from a separate audio stream s, two sets of gains can be calculated. In one such example, the first one G x (b,t,c) is applied to audio stream x to increase its loudness and the second one G s (b,t) are applied to the competing audio stream s to reduce its loudness, so that their combination of gains ensures the audibility of the audio stream x. This is shown in Equations 9a and 9b. In both sets of equations, ￣PL x(b,t,c) represent the partial specific loudness of the source signal in the presence of noise after application of the compensation gain.

number

[0183] In practice, and again to avoid audible artifacts, the raw gain is further smoothed over frequency using a smoothing function S{·} before being applied to the audio stream. x (b,t,c) and ￣G s (b,t) represents the final compensation gain for the target audio stream x and the competing audio stream s.

number

[0184] In one embodiment, these gains may be applied directly to all rendered output channels of the audio stream. In another embodiment, they may instead be applied to the objects of the audio stream before they are rendered. This may be done, for example, using the methods described in U.S. Patent Application Publication No. 2019 / 0037333 A1, which is incorporated herein by reference. These methods include calculating a pan coefficient for each audio object associated with each of a plurality of predefined channel coverage zones based on the spatial metadata of the audio object. The audio signal may be transformed into submixes with respect to the predefined channel coverage zones based on the calculated pan coefficients and the audio object. Each of the submixes may represent a sum of the components of the plurality of audio objects with respect to one of the predefined channel coverage zones. The submix gains may be generated by applying audio processing to each of the submixes and may control the object gains applied to each audio object. The object gains may be a function of the pan coefficient for each audio object and the submix gains associated with each of the predefined channel coverage zones. Applying gain to objects has several advantages, especially when combined with other processing of the stream.

[0185] FIG. 8 illustrates an implementation of a multi-stream rendering system with an audio stream loudness estimator. According to this example, the multi-stream rendering system of FIG. 8 is configured to implement loudness processing, e.g., as described in Equations 12a-21b, and also compensation gain application in each single stream renderer. In this example, a quadrature mirror filter bank (QMF) is applied to each of program streams 1 and 2 before each program stream is received by a corresponding one of rendering modules 1 and 2. In an alternative example, a quadrature mirror filter bank (QMF) may be applied to each of program streams 1-N before each program stream is received by a corresponding one of rendering modules 1-N. According to this example, rendering modules 1 and 2 operate in the frequency domain. In this implementation, loudness estimation module 805a calculates a loudness estimate for program stream 1, e.g., as described above with reference to Equations 12a-17b. Similarly, in this example, loudness estimation module 805b calculates a loudness estimate for program stream 2.

[0186] In this implementation, the time-domain microphone signals from microphone system 120c are also provided to a quadrature mirror filter bank such that loudness estimation module 805c receives the microphone signals in the frequency domain. In this implementation, loudness estimation module 805c calculates loudness estimates for the microphone signals, for example, as described above with reference to Equations 12b-17a. In this example, loudness processing module 810 is configured to implement loudness processing, for example, as described in Equations 18-21b, and compensation gain application for each single stream rendering module. In this implementation, loudness processing module 810 is configured to modify the audio signals of program stream 1 and program stream 2 to maintain their perceived loudness in the presence of one or more interfering signals. In some instances, the control system can determine that the microphone signals correspond to environmental noise and that the program streams should be raised above it. However, in some examples, the control system may determine that the microphone signals correspond to a wake word, a command, a child crying, or other such audio that may need to be heard by a smart audio device and / or one or more listeners. In some such implementations, the loudness processing module 810 may be configured to modify the microphone signals to maintain their perceived loudness in the presence of interfering audio signals of program stream 1 and / or audio signals of program stream 2. Here, the loudness processing module 810 is configured to provide appropriate gains to the rendering modules 1 and 2. After the mixer 630c mixes the outputs of the rendering modules 1-N, the inverse filter bank 635c transforms the mix to the time domain and provides a mixed speaker feed signal in the time domain to the loudspeakers 1-M. In this example, the quadrature mirror filter bank, the rendering modules 1-N, the mixer 630c, and the inverse filter bank 635c are components of the control system 110e.

[0187] FIG. 9A illustrates an example of a multi-stream rendering system configured for cross-fading of multiple rendered streams. In some such embodiments, cross-fading of multiple rendered streams is used to provide a smooth experience when the rendering configuration is dynamically changed. One example is the aforementioned use case of simultaneous playback of a spatial program stream, such as music, with a smart voice assistant's response to some query by the listener, as described above with reference to FIGS. 4A and 4B. In this case, it is useful to instantiate extra single-stream renderers with alternative spatial rendering configurations and cross-fade between them simultaneously, as shown in FIG. 9A.

[0188] In this example, QMF is applied to program stream 1 before program stream 1 is received by rendering modules 1a and 1b. Similarly, QMF is applied to program stream 2 before program stream 2 is received by rendering modules 2a and 2b. In some instances, the output of rendering module 1a may correspond to a desired playback of program stream 1 before detection of the wake word, while the output of rendering module 1b may correspond to a desired playback of program stream 1 after detection of the wake word. Similarly, the output of rendering module 2a may correspond to a desired playback of program stream 2 before detection of the wake word, while the output of rendering module 2b may correspond to a desired playback of program stream 2 after detection of the wake word. In this implementation, the outputs of rendering modules 1a and 1b are provided to crossfade module 910a, and the outputs of rendering modules 2a and 2b are provided to crossfade module 910b. Crossfade times may range, for example, from hundreds of milliseconds to several seconds.

[0189] After mixer 630d mixes the outputs of crossfade modules 910a and 910b, inverse filter bank 635d transforms the mix to the time domain and provides time domain mixed speaker feed signals to loudspeakers 1 through M. In this example, the quadrature mirror filter bank, the rendering module, the crossfade module, mixer 630d and inverse filter bank 635d are components of control system 110f.

[0190] In some embodiments, it may be possible to pre-calculate the rendering configuration to be used in each of the single-stream renderers 1a, 1b, 2a, and 2b. This is particularly convenient and efficient for use cases such as smart voice assistants, since the spatial configuration is often known a priori and does not depend on other dynamic aspects of the system. In other embodiments, it may not be possible or desirable to pre-calculate the rendering configuration, in which case the complete configuration for each single-stream renderer must be dynamically calculated while the system is running.

[0191] Some implementation aspects include: 1. An audio rendering system that simultaneously plays multiple audio program streams through multiple arbitrarily positioned loudspeakers, at least one of the program streams being a spatial mix, and wherein the rendering of the spatial mix is ​​dynamically modified in response to the simultaneous playback of one or more additional program streams. 2. A system as described in claim 1, wherein the rendering of any of the plurality of audio program streams may be dynamically modified as a function of any one or more combinations of the remaining plurality of audio program streams. 3. The system of claim 1 or 2, wherein the modification includes one or more of the following: modifying a relative activation of the plurality of loudspeakers as a function of a relative activation of a loudspeaker associated with the rendering of at least one of the one or more additional program streams; warping the intended spatial balance of the spatial mix as a function of spatial characteristics of a rendering of at least one of the one or more additional program streams; or · Modifying the loudness or audibility of the spatial mix as a function of the loudness or audibility of at least one of the one or more additional program streams. 4. The system of claim 1 or 2, further comprising dynamically modifying the rendering as a function of one or more microphone inputs. 5. The system of claim 4, wherein information derived from the microphone input used to modify the rendering includes one or more of the following: -Detection of the utterance of certain phrases by the user of the system; · an estimate of the location of one or more users of the system; An estimate of the loudness of any of the N program stream combinations at a particular location in the listening space; or An estimate of the loudness of other environmental sounds in the listening environment, e.g. background noise.

[0192] Other example embodiments of the systems and methods of the present invention for managing the playback of multiple audio streams through multiple speakers (e.g., a set of speakers of an orchestrated smart audio device) include: 1. An audio system (e.g., an audio rendering system) that simultaneously plays multiple audio program streams through multiple, arbitrarily positioned loudspeakers (e.g., a set of speakers of an orchestrated smart audio device), where at least one of the program streams is a spatial mix, and where the rendering of the spatial mix is ​​dynamically modified in response to (or in conjunction with) the simultaneous playback of one or more additional program streams. 2. The system of claim 1, wherein the modifications to the spatial mix include one or more of the following: warping the rendering of the spatial mix away from the rendering location of said one or more additional streams; or Modifying the loudness of the spatial mix depending on the loudness of said one or more additional streams. 3. The system of claim 1, further comprising the step of dynamically modifying the rendering of the spatial mix as a function of one or more microphone inputs (i.e., signals captured by one or more microphones of one or more smart audio devices, e.g., a set of orchestrated smart audio devices). 4. The system of claim 3, wherein at least one of the one or more microphone inputs includes a human voice. Optionally, the rendering is dynamically modified in response to a determined location of a source of the voice (human). 5. The system of claim 3, wherein at least one of the one or more microphone inputs includes environmental noise. 6. The system of claim 3, wherein an estimate of loudness of the spatial stream or the one or more additional streams is derived from at least one of the one or more microphone inputs.

[0193] One practical consideration when implementing dynamic cost flexible rendering (according to some embodiments) is the amount of computation. In some cases, it may not be feasible to solve a unique cost function for each frequency band for each audio object in real time, given that object positions (which may be indicated by metadata, for each audio object being rendered) may change many times per second. An alternative approach that reduces computational effort at the expense of memory is to use a lookup table that samples the three-dimensional space of all possible object positions. The sampling does not need to be the same in all dimensions. FIG. 9B is a graph of points showing speaker activations in one exemplary embodiment. In this example, the x and y dimensions are sampled with 15 points, and the z dimension is sampled with 5 points. Other implementations may include more or fewer samples. According to this example, each point represents M speaker activations for a CMAP or FV solution.

[0194] At run time, to determine the actual activation for each speaker, in some examples, tri-linear interpolation between the last eight speaker activation points may be used. Figure 10 is a graph of tri-linear interpolation between points showing speaker activations according to one example. In this example, the process of successive linear interpolation includes interpolating each pair of points in the top surface to determine first and second interpolated points 1005a and 1005b, interpolating each pair of points in the bottom surface to determine third and fourth interpolated points 1010a and 1010b, interpolating the first and second interpolated points 1005a and 1005b to determine a fifth interpolated point 1015 in the top surface, interpolating the third and fourth interpolated points 1010a and 1010b to determine a sixth interpolated point 1020 in the bottom surface, and interpolating the fifth and sixth interpolated points 1015 and 1020 to determine a seventh interpolated point 1025 between the top and bottom surfaces. While trilinear interpolation is a valid interpolation method, those skilled in the art will understand that trilinear interpolation is only one possible interpolation method that may be used in implementing aspects of the present disclosure and that other examples may include other interpolation methods.

[0195] In the first example above, where repulsion forces are used to create an acoustic space for a voice assistant, for example, another key concept is the transition from a rendering scene without repulsion forces to one with repulsion forces: both the previous set of speaker activations without repulsion forces and the new set of speaker activations with repulsion forces are computed and interpolated over a period of time to create a smooth transition and give the impression that the sound field is dynamically distorted.

[0196] An example of an audio rendering method implemented according to an embodiment includes an audio rendering method comprising: 1. A method comprising: rendering a collection of one or more audio signals, each having an associated desired perceived spatial location, through a collection of two or more loudspeakers, wherein relative activation of the collections of loudspeakers is a function of a model of the perceived spatial location of the audio signals played through those loudspeakers, a proximity of the desired perceived spatial location of the audio object to the position of the loudspeakers, and one or more additional dynamically configurable features that depend on at least one or more attributes of the collection of audio signals, one or more attributes of the collection of loudspeakers, or one or more external inputs.

[0197] Further examples of embodiments are described with reference to FIG. 11. As with other figures provided herein, the types and numbers of elements shown in FIG. 11 are given merely as examples. Other implementations may include more, fewer, and / or different types and numbers of elements. FIG. 11 shows a floor plan of a listening environment, which in this example is a residential space. According to this example, the environment 1100 includes a living room 1110 at the top left, a kitchen 1115 at the bottom center, and a bedroom 1122 at the bottom right. The squares and circles distributed throughout the residential space represent a set of loudspeakers 1105a-1105h that are conveniently located in the space but do not conform to a standardized defined layout (arbitrarily placed), some of which may be smart speakers in some implementations. In some examples, the loudspeakers 1105a-1105h may be coordinated to implement one or more disclosed embodiments. In this example, the environment 1100 includes cameras 1111a-1111e distributed throughout the environment. In some implementations, one or more smart audio devices in the environment 1100 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, one or more cameras of the optional sensor system 130 may be present in or on the surface of the television 1130, in a mobile phone, or in a smart speaker, such as one or more of the loudspeakers 1105b, 1105d, 1105e, or 1105h. Although the cameras 1111a-1111e are not shown in all figures of the environments 1100 presented in this disclosure, each of the environments 1100 may still include one or more cameras in some implementations.

[0198] Figures 12A, 12B, 12C, and 12D show examples of flexible rendering of spatial audio in the reference spatial mode for several different listening positions and orientations in the living space shown in Figure 11. Figures 12A-12D show this capability in four example listening positions. In each example, arrow 1205 pointing towards person 1220a represents the location of the forward sound stage (where person 1220a is facing). In each example, arrow 1210a represents the left surround field and arrow 1210b represents the right surround field.

[0199] In FIG. 12A, a reference spatial mode is determined and spatial audio is flexibly rendered for a person 1220a sitting on a living room couch 1225. According to some implementations, a control system (such as control system 110 of FIG. 1A) may be configured to determine an assumed listening position and / or an assumed orientation of the reference spatial mode according to reference spatial mode data received via an interface system, such as interface system 105 of FIG. 1A. Some examples are described below. In some such examples, the reference spatial mode data may include microphone data from a microphone system (such as microphone system 120 of FIG. 1A).

[0200] In some such examples, the reference spatial mode data may include microphone data corresponding to a wake word and a voice command, e.g., “[wake word], put the TV in the forward sound stage.” Alternatively or additionally, the microphone data may be used to triangulate a user's position in response to the sound of the user's voice, e.g., via direction of arrival (DOA) data. For example, three or more loudspeakers 1105a-1105e may use the microphone data to triangulate a position of a person 1220a sitting on a couch 1225 in a living room in response to the sound of the person's voice, via DOA data. An orientation of the person 1220a may be assumed according to the position of the person 1220a: if the person 1220a is in the position shown in FIG. 12A, the person 1220a may be assumed to be facing the television 1130.

[0201] Alternatively or additionally, the position and orientation of the person 1220a may be determined according to image data from a camera system (such as sensor system 130 in FIG. 1A).

[0202] In some examples, the position and orientation of person 1220a may be determined according to user input obtained via a graphical user interface (GUI). According to some such examples, the control system may be configured to control a display device (e.g., a display device of a cellular telephone) to present a GUI that allows person 1220a to input the position and orientation of person 1220a.

[0203] FIG. 13A shows an example of a GUI for receiving user input regarding the position and orientation of a listener. According to this example, the user has previously identified several possible listening positions and corresponding orientations. The loudspeaker positions corresponding to each position and the corresponding orientation have already been entered and stored during the setup process. Several examples are described below. For example, a listening environment layout GUI may be provided, and the user may be prompted to touch positions corresponding to possible listening positions and speaker positions and to name the possible listening positions. In this example, at the time shown in FIG. 13A, the user has already given user input regarding the user's position to the GUI 1300 by touching the virtual button "Living Room Couch". Because of the L-shaped couch 1225, there are two possible forward-facing positions, so the user is prompted to indicate which direction the user is facing.

[0204] In Fig. 12B, a reference spatial mode is determined and spatial audio is flexibly rendered for a person 1220a sitting in a reading chair 1215 in the living room. In Fig. 12C, a reference spatial mode is determined and spatial audio is flexibly rendered for a person 1220a standing next to a kitchen counter 1230. In Fig. 12D, a reference spatial mode is determined and spatial audio is flexibly rendered for a person 1220a sitting at a breakfast table 1240. It can be observed that the forward sound stage orientation does not necessarily correspond to a particular loudspeaker in the environment 1100, as indicated by the arrow 1205. As the position and orientation of the listener changes, the role of the speakers for rendering the various components of the spatial mix also changes.

[0205] For a person 1220a in any of Figures 12A-12D, that person hears the spatial mix as intended for each of the positions and orientations shown. However, this experience may not be optimal for additional listeners in the space. Figure 12E shows an example of a reference spatial mode rendering when two listeners are in different positions in the listening environment. Figure 12E shows a reference spatial mode rendering for a person 1220a on a couch and a person 1220b standing in the kitchen. In this example, the rendering may be optimal for person 1220a, but person 1220b, given his position, hears mostly signals from the surround field and very little of the front soundstage.

[0206] In this case, and in other cases where multiple people are in a space and may move around in unpredictable ways (e.g., parties), a rendering mode that is more appropriate for such a distributed audience is needed. FIG. 13B illustrates a distributed spatial rendering mode according to an example embodiment. In this example of a distributed spatial mode, the front sound stage is rendered uniformly throughout the entire listening space, rather than only from a living room or a position in front of a listener on a couch. This distribution of the front sound stage is represented by multiple arrows 1305d circling the cloud 1335, all of which have the same or nearly the same length. The intended meaning of the arrows 1305d is that the multiple listeners shown (people 1220a-1220f) can hear this part of the mix equally well, regardless of their location. However, if this uniform distribution were applied to all components of the mix, all spatial aspects of the mix would be lost, and people 1220a-1220f would essentially hear mono audio. To maintain some degree of spatiality, the left and right surround components of the mix, represented by arrows 1210a and 1210b, respectively, are still rendered spatially. (Often there may be left and right surrounds, left and right rear surrounds, overheads, and dynamic audio objects with spatial positions within this space. Arrows 1210a and 1210b are intended to represent the left and right portions of all these possibilities.) To maximize the perceived spatiality, the area in which these components are spatialized is expanded to more fully cover the entire auditory space, including the space previously occupied only by the forward soundstage. This expanded area in which the surround components are rendered can be understood by comparing the relatively long arrows 1210a and 1210b shown in FIG. 13B with the relatively short arrows 1210a and 1210b shown in FIG. 12A. Additionally, arrows 1210a and 1210b shown in FIG. 12A represent the surround components in the reference spatial mode, which extend from approximately the sides of person 1220a to the rear of the listening environment, and do not extend into the front stage area of ​​the listening environment.

[0207] In this example, care is taken in implementing the uniform distribution of the front sound stage and the expanded spatialization of the surround components so that the perceived loudness of these components is largely maintained compared to the rendering for the reference spatial mode. The goal is to shift the spatial impression of these components to optimize for multiple people while maintaining the relative levels of each component in the mix. For example, it would be undesirable if the front sound stage became twice as large relative to the surround components as a result of its uniform distribution.

[0208] To switch between the various reference and distributed rendering modes of the exemplary embodiments, in some examples, a user may interact with a voice assistant associated with the orchestrated speaker system. For example, to play audio in the reference spatial mode, the user may utter a wake word for the voice assistant (e.g., "Listen Dolby") followed by the command, "Play [insert content name] for me" or "Play [insert content name] in personal mode." Based on recordings from various microphones associated with the system, the system may then automatically determine the user's location and orientation, or one of several predefined zones that is closest to the user, and begin playing audio in the reference mode that corresponds to this determined location. To play audio in the distributed spatial mode, the user may utter a different command, for example, "Play [insert content name] in distributed mode."

[0209] Alternatively or additionally, the system may be configured to automatically switch between the reference and distributed modes based on other inputs. For example, the system may have a means to automatically determine the number of listeners in the space and their locations. This may be achieved, for example, by monitoring audio activity in the space from associated microphones and / or through the use of other relevant sensors such as one or more cameras. In this case, the system may be configured with a mechanism to continuously change the rendering between the reference spatial mode as shown in FIG. 12E and a fully distributed spatial mode as shown in FIG. 13B. The point on this continuum at which the rendering is set may be calculated, for example, as a function of the number of people reported in the space.

[0210] Figures 12A, 14A, and 14B illustrate this behavior. In Figure 12A, the system detects only a single listener (person 1220a) on a couch facing the TV, so the rendering mode is set to the reference spatial mode for this listener position and orientation. Figure 14A illustrates a partially distributed spatial rendering mode according to an example. In Figure 14A, two additional listeners (persons 1220e and 1220f) are detected behind person 1220a, and the rendering mode is set to a point between the reference spatial mode and the fully distributed spatial mode. Here, some of the front sound stage (arrows 1305a, 1305b, and 1305c) has been pulled back toward the additional listeners (persons 1220e and 1220f), but there is still more emphasis on the front sound stage position of the reference spatial mode. This emphasis is indicated in Figure 14A by arrow 1205 and the relatively long length of arrow 1305a compared to the lengths of arrows 1305b and 1305c, and the surround field is only partially extended towards the position of the forward sound stage of the reference spatial mode, as indicated by the length and position of arrows 1210a and 1210b.

[0211] FIG. 14B illustrates a fully distributed spatial rendering mode according to one example. In some examples, the system may detect multiple listeners (people 1220a, 1220e, 1220f, 1220g, 1220h, and 1220i) across the space, and the system may automatically set the rendering mode to the fully distributed spatial mode. In other examples, the rendering mode may be set according to user input. The fully distributed spatial mode is indicated in FIG. 14B by the uniform or substantially uniform length of arrow 1305d, and the lengths and positions of arrows 1210a and 1210b.

[0212] In the preceding examples, the parts of the spatial mix that are rendered with a more uniform distribution in distributed rendering mode are designated as the forward soundstage. In many spatial mix contexts, this makes sense because traditional mix implementations typically place the most important parts of the mix, such as dialogue for movies or lead vocals, drums, and bass for music, in the forward soundstage. This is true for most 5.1 and 7.1 surround sound mixes, as well as stereo content that is upmixed to 5.1 or 7.1 using algorithms such as Dolby Pro Logic or Dolby Surround. Here, the forward soundstage is given by the left, right, and center channels. This is also true for many object-based audio mixes, such as Dolby Atmos, where audio data may be designated as the forward soundstage according to spatial metadata that indicates an (x,y) spatial location with y<0.5. However, with object-based audio, the mix engineer has the freedom to place the audio anywhere in 3D space. In particular, with object-based music, mixing engineers are moving away from traditional mixing norms and placing what are considered important parts of the mix, such as lead vocals, in non-traditional positions, such as overhead. In such cases, it becomes difficult to construct simple rules to determine which components of the mix are suitable for rendering in a more distributed spatial manner for the distributed rendering mode. Object-based audio already contains metadata associated with each of its constituent audio signals that describes where in 3D space the signal should be rendered. To address the above issues, in some implementations, additional metadata may be added that allows content creators to flag certain signals as suitable for more distributed spatial rendering in the distributed rendering mode. During rendering, the system may use this metadata to select which components of the mix to which the more distributed rendering will be applied.This gives content creators control over how the distributed rendering mode will sound for their particular content.

[0213] In some alternative implementations, the control system may be configured to implement a content type classifier to identify one or more elements of the audio data that should be rendered in a more spatially distributed manner. In some examples, the content type classifier may reference content type metadata (e.g., metadata indicating that the audio data is dialogue, vocals, percussion, bass, etc.) to determine whether the audio data should be rendered in a more spatially distributed manner. According to some such implementations, the content type metadata to be rendered in a more spatially distributed manner may be selectable by a user, for example, according to user input via a GUI displayed on a display device.

[0214] The exact mechanism used to render one or more elements of a spatial audio mix in a more spatially distributed manner than in the reference spatial mode may vary between different embodiments, and this disclosure is intended to cover all such mechanisms. One exemplary mechanism involves generating multiple copies of each such element with multiple associated rendering positions that are more evenly distributed throughout the listening space. In some implementations, the rendering positions and / or the number of rendering positions for the distributed spatial mode may be user-selectable, while in other implementations, the rendering positions and / or the number of rendering positions for the distributed spatial mode may be pre-set. In some such implementations, a user may select several rendering positions for the distributed spatial mode, which may be pre-set, e.g., evenly spaced throughout the listening environment. The system then renders all of these copies at a set of distributed positions rather than the original single element in its original intended position. According to some implementations, the levels of the copies may be modified so that the perceived level associated with the combined rendering of all copies is the same or substantially the same (e.g., within a threshold number of decibels, such as 2 dB, 3 dB, 4 dB, 5 dB, 6 dB, etc.) as the level of the original single elements in the reference rendering mode.

[0215] A more elegant mechanism may be implemented in the context of either a CMAP or FV flexible rendering system, or a hybrid of both systems, where each element of a spatial mix is ​​rendered at a specific location in space, and each element may have an associated assumed fixed position, for example the canonical position of a channel in a 5.1 or 7.1 surround sound mix, or a time-varying position as in the case of object-based audio like Dolby Atmos.

[0216] FIG. 15 shows example rendering positions for the CMAP and FV rendering systems on a 2D plane. Each small, numbered circle represents an example rendering position, where the rendering system can render elements of the spatial mix anywhere on or within the circle 1500. The positions on the circle 1500 labeled L, R, C, Lss, Rss, Lrs, and Rrs represent the fixed canonical rendering positions of the seven full-range channels of the 7.1 surround mix in this example: left (L), right (R), center (C), left side surround (Lss), right side surround (Rss), left rear surround (Lrs), and right rear surround (Rrs). In this context, the rendering positions near L, R, and C are considered the forward sound stage. For the reference rendering mode (also referred to herein as the "reference spatial mode"), the listener is assumed to be located at the center of a large circle opposite the C rendering location. For any of the FIGS. 12A-12D that show reference renderings for various listening positions and orientations, one can conceptualize the center of FIG. 15 being superimposed over the listener. Now, FIG. 15 is additionally rotated and scaled so that the C location is aligned with the location of the forward sound stage (arrow 1205) and the circle 1500 of FIG. 15 surrounds the cloud 1235. The resulting alignment describes any speaker in FIG. 12A-12D being relatively close to any of the rendering locations in FIG. 15. In some implementations, for both CMAP and FV rendering systems, it is this proximity that largely governs the relative activation of speakers when rendering elements of a spatial mix at a particular location.

[0217] When spatial audio is mixed in a studio, speakers are typically placed at uniform distances around the listening position. In most cases, no speakers are within the resulting circle or hemisphere. When the audio is placed "inside the room" (e.g., the center of FIG. 15), the rendering tends toward firing all speakers on the periphery to achieve a "nowhere sound." In CMAP and FV rendering systems, a similar effect can be achieved by modifying the proximity penalty term in the cost function governing speaker activation. In particular, for rendering positions on the periphery of the circle 1500 in FIG. 15, the proximity penalty term fully penalizes the use of speakers far from the desired rendering position. Thus, only speakers near the intended rendering position are effectively activated. As the desired rendering position moves toward the center of the circle (radius zero), the proximity penalty term reduces to zero, so that no speakers are prioritized at the center. The corresponding result for a rendering position with a radius of zero is a perfectly uniform perceived distribution of the audio across the listening space, which is exactly the desired result for some elements of the mix in the most dispersed spatial rendering modes.

[0218] Given this behavior of the CMAP and FV systems at zero radius, a more spatially distributed rendering of any element of the spatial mix may be achieved by warping its intended spatial location towards the zero radius point. This warping may be made continuous between the original intended location and zero radius, thereby providing a natural, continuous control between the reference spatial mode and the various distributed spatial modes. Figures 16A, 16B, 16C, and 16D show examples of warping applied to all of the rendering points of Figure 15 to achieve various distributed spatial rendering modes. Figure 16D shows an example of such warping applied to all of the rendering points of Figure 15 to achieve a fully distributed rendering mode. It can be seen that the L, R, and C points (the front sound stage) are collapsed to zero radius, thereby ensuring rendering in a completely uniform manner. Additionally, the Lss and Rss rendering points are pulled back toward the front soundstage along the circumference of the circle so that the spatialized surround field (Lss, Rss, Lbs, and Rbs) encompasses the entire listening area. This warping is applied to the entire rendering space, and it can be seen that all of the rendering points in FIG. 15 are warped to new positions in FIG. 16D that correspond to the warping of the 7.1 canonical positions. The spatial mode referenced in FIG. 16D is an example of what is referred to herein as the "most distributed spatial mode" or "fully distributed spatial mode."

[0219] Figures 16A, 16B and 16C show various examples of intermediate distributed spatial modes between the distributed spatial modes depicted in Figures 15 and 16D. Figure 16B represents a midpoint between the distributed spatial modes depicted in Figures 15 and 16D. Figure 16A represents a midpoint between the distributed spatial modes depicted in Figures 15 and 16B. Figure 16C represents a midpoint between the distributed spatial modes depicted in Figures 16B and 16D.

[0220] FIG. 17 illustrates an example of a GUI through which a user may select a rendering mode. According to some implementations, a control system may control a display device (e.g., a cellular phone) to display GUI 1700 or a similar GUI on the display. The display device may include a sensor system (e.g., a touch sensor system or a gesture sensor system proximate to the display (e.g., above or below the display). The control system may be configured to receive user input via GUI 1700 in the form of a sensor signal from the sensor system. The sensor signal may correspond to a user touch or gesture that corresponds to an element of GUI 1700.

[0221] According to this example, the GUI includes a virtual slider 1701 with which the user can interact to select a rendering mode. The user can move the slider in either direction along a track 1707, as indicated by arrow 1703. In this example, line 1705 indicates a position of the virtual slider 1701 that corresponds to a reference spatial mode, such as one of the reference spatial modes disclosed herein. Other implementations may provide other features on the GUI with which the user can interact, such as a virtual knob or dial. According to some implementations, after selecting the reference spatial mode, the control system may present a GUI as shown in FIG. 13A, or another such GUI that allows the user to select a listener position and orientation for the reference spatial mode.

[0222] In this example, line 1725 indicates a position of virtual slider 1701 that corresponds to the most dispersed spatial mode, such as the dispersed spatial mode shown in FIG. 13B. According to this implementation, lines 1710, 1715, and 1720 indicate positions of virtual slider 1701 that correspond to intermediate spatial modes. In this example, the position of line 1710 corresponds to the intermediate spatial mode, such as that of FIG. 16A. Here, the position of line 1715 corresponds to the intermediate spatial mode, such as that of FIG. 16B. In this implementation, the position of line 1720 corresponds to the intermediate spatial mode, such as that of FIG. 16C. According to this example, a user can interact with (e.g., touch) an “apply” button to instruct the control system to implement the selected rendering mode.

[0223] However, other implementations may provide other ways for a user to select one of the aforementioned distributed spatial modes. According to some examples, a user may utter a voice command, for example, "Play [insert content name] in semi-distributed mode." The "semi-distributed mode" may correspond to the distributed mode indicated by the location of line 1715 in GUI 1700 of FIG. 17. According to some such examples, a user may utter a voice command, for example, "Play [insert content name] in 1 / 4 distributed mode." The "1 / 4 distributed mode" may correspond to the distributed mode indicated by the location of line 1710.

[0224] FIG. 18 is a flow diagram outlining an example of a method that may be performed by an apparatus or system such as those disclosed herein. The blocks of method 1800, as well as other methods described herein, are not necessarily performed in the order shown. In some implementations, one or more blocks of method 1800 may be performed in parallel. Additionally, some implementations of method 1800 may include more or fewer blocks than shown and / or described. The blocks of method 1800 may be performed by one or more apparatuses (which may be or include a control system, such as control system 110 shown in FIG. 1A and described above, or one of the other disclosed control system examples).

[0225] In this implementation, block 1805 involves receiving, by the control system, via the interface system, audio data including one or more audio signals and associated spatial data, in this example, the spatial data indicating intended perceived spatial locations corresponding to the audio signals, where the spatial data includes channel data and / or spatial metadata.

[0226] In this example, block 1810 includes determining a rendering mode by a control system. Determining the rendering mode may, in some cases, include receiving a rendering mode instruction via an interface system. Receiving the rendering mode instruction may involve, for example, receiving a microphone signal corresponding to a voice command. In some examples, receiving the rendering mode instruction may include receiving a sensor signal corresponding to a user input via a graphical user interface. The sensor signal may, for example, be a touch sensor signal and / or a gesture sensor signal.

[0227] In some implementations, receiving the rendering mode instruction may include receiving an instruction of a number of people in the listening area. According to some such examples, the control system may be configured to determine the rendering mode based, at least in part, on the number of people in the listening area. In some such examples, the instruction of the number of people in the listening area may be based on microphone data from a microphone system and / or image data from a camera system.

[0228] According to the example shown in FIG. 18, block 1815 includes rendering, by the control system, the audio data for playback through a set of loudspeakers of the environment according to the rendering mode determined in block 1810 to generate a rendered audio signal. In this example, rendering the audio data includes determining a relative activation of a set of loudspeakers in the environment, where the rendering mode is variable between a reference spatial mode and one or more distributed spatial modes. In this implementation, the reference spatial mode has an assumed listening position and orientation. According to this example, in the one or more distributed spatial modes, one or more elements of the audio data are rendered in a more spatially distributed manner than in the reference spatial mode, respectively. In this example, in the one or more distributed spatial modes, the spatial positions of the remaining elements of the audio data are warped such that they span the rendering space of the environment more completely than in the reference spatial mode.

[0229] In some implementations, rendering the one or more elements of audio data in a more spatially distributed manner than in the reference spatial mode may include generating copies of the one or more elements, and some such implementations may include rendering all the copies simultaneously at a set of locations distributed across the environment.

[0230] According to some implementations, the rendering may be based on a CMAP, an FV, or a combination thereof. Rendering the one or more elements of audio data in a more spatially distributed manner than in a reference spatial mode may include warping a rendering position of each of the one or more elements towards a zero radius.

[0231] In this example, block 1820 includes providing, by the control system, via the interface system, the rendered audio signals to at least some of the set of loudspeakers in the environment.

[0232] According to some implementations, the rendering mode may be selectable from a continuum of rendering modes ranging from a reference spatial mode to a most dispersed spatial mode. In some such implementations, the control system may be further configured to determine an assumed listening position and / or orientation of the reference spatial mode according to the reference spatial mode data received via the interface system. According to some such implementations, the reference spatial mode data may include microphone data from a microphone system and / or image data from a camera system. In some such examples, the reference spatial mode data may include microphone data corresponding to a voice command. Alternatively or additionally, the reference spatial mode data may include microphone data corresponding to a position of one or more vocalizations of a person in the listening environment. In some such examples, the reference spatial mode data may include image data indicative of a position and / or orientation of a person in the listening environment.

[0233] However, in some cases, the device or system may include a display device and a sensor system proximate to the display device. The control system may be configured to control the display device to present a graphical user interface. Receiving the reference spatial mode data may include receiving a sensor signal corresponding to a user input via the graphical user interface.

[0234] According to some implementations, the one or more elements of audio data each rendered in a more spatially distributed manner may correspond to front soundstage data, musical vocals, dialogue, bass, percussion, and / or other solo or lead instruments. In some cases, the front soundstage data may include left, right, or center signals of audio data received in or upmixed to a Dolby 5.1, Dolby 7.1, or Dolby 9.1 format. In some examples, the front soundstage data may include audio data received in a Dolby Atmos format and having spatial metadata indicating an (x,y) spatial location, where y<0.5.

[0235] In some cases, the audio data may include spatial distribution metadata indicating which elements of the audio data should be rendered in a more spatially distributed manner. In some such examples, the control system may be configured to identify the one or more elements of the audio data that should be rendered in a more spatially distributed manner in accordance with the spatial distribution metadata.

[0236] Alternatively or additionally, the control system may be configured to implement a content type classifier to identify the one or more elements of audio data that should be rendered in a more spatially distributed manner. In some examples, the content type classifier may refer to content type metadata (e.g., metadata indicating that the audio data is dialogue, vocals, percussion, bass, etc.) to determine whether the audio data should be rendered in a more spatially distributed manner. According to some such implementations, the content type metadata to be rendered in a more spatially distributed manner may be selectable by a user, for example, according to user input via a GUI displayed on a display device. Alternatively or additionally, the content type classifier may operate directly on the audio signals in combination with the rendering system. For example, the classifier may be implemented using a neural network trained on a variety of content types to analyze the audio signals and determine whether they belong to any content types (e.g., vocals, lead guitar, drums, etc.) that may be considered suitable for rendering in a more spatially distributed manner. Some such classifications may be performed in a continuous and dynamic manner, and the resulting classification results may adjust the set of signals to be rendered in a more spatially distributed manner in a continuous and dynamic manner. Some such implementations may include the use of techniques such as neural networks to implement such dynamic classification systems according to methods known in the art.

[0237] In some examples, at least one of the one or more distributed spatial modes may include applying a time-varying modification to the spatial location of at least one element. According to some such examples, the time-varying modification may be a periodic modification. For example, the periodic modification may include orbiting one or more rendering locations around a circumference of the listening environment. According to some such implementations, the periodic modification may include a tempo of music played in the environment, a beat of music played in the environment, or one or more other characteristics of audio data played in the environment. For example, some such periodic modifications may include alternating between two, three, four, or more rendering locations. The alternation may correspond to a beat of music played in the environment. In some implementations, the periodic modification may be selectable according to user input, e.g., according to one or more voice commands, according to user input received via a GUI, or the like, or the like.

[0238] FIG. 19 illustrates an example of a geometric relationship between three audio devices in an environment. In this example, the environment 1900 is a room that includes a television 1901, a sofa 1903, and five audio devices 1905. According to this example, the audio devices 1905 are located at positions 1-5 in the environment 1900. In this implementation, each audio device 1905 includes a microphone system 1920 having at least three microphones and a speaker system 1925 including at least one speaker. In some implementations, each microphone system 1920 includes an array of microphones. According to some implementations, each of the audio devices 1905 may include an antenna system including at least three antennas.

[0239] As with other examples disclosed herein, the types, numbers, and arrangements of elements shown in Figure 19 are merely examples. Other implementations may have different types, numbers and arrangements of elements, such as having more or fewer audio devices 1905, audio devices 1905 in different locations, etc.

[0240] In this example, triangle 1910a has vertices at locations 1, 2, and 3. Here, triangle 1910a has sides 12, 23a, and 13a. According to this example, the angle between sides 12 and 23 is θ2, the angle between sides 12 and 13a is θ1, and the angle between sides 23a and 13a is θ3. These angles may be determined according to DOA data, as described in more detail below.

[0241] In some implementations, only the relative lengths of the triangle's sides may be determined. In alternative implementations, the actual lengths of the triangle's sides may be estimated. According to some such implementations, the actual lengths of the triangle's sides may be estimated according to TOA data, for example according to the arrival time of a sound generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. Alternatively or additionally, the lengths of the triangle's sides may be estimated by electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. For example, the lengths of the triangle's sides may be estimated by the signal strength of electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. In some implementations, the lengths of the triangle's sides may be estimated according to a detected phase shift of the electromagnetic waves.

[0242] Figure 20 shows another example of the geometric relationships between three audio devices in the environment shown in Figure 19. In this example, triangle 1910b has vertices at locations 1, 3, and 4. Here, triangle 1910b has sides 13b, 14, and 34a. According to this example, the angle between sides 13b and 14 is θ4, the angle between sides 13b and 34a is θ5, and the angle between sides 34a and 14 is θ6.

[0243] 11 and 12, it can be observed that the length of side 13a of triangle 1910a should be equal to the length of side 13b of triangle 1910b. In some implementations, the length of a side of one triangle (e.g., triangle 1910a) may be assumed to be correct, and the length of the side shared by adjacent triangles is constrained to this length.

[0244] FIG. 21A shows both of the triangles shown in FIGS. 19 and 20, without the corresponding audio devices and other features of the environment. FIG. 21A shows estimates of the side lengths and angular orientations of triangles 1910a and 1910b. In the example shown in FIG. 21A, the length of side 13b of triangle 1910b is constrained to be the same length as side 13a of triangle 1910a. The lengths of the other sides of triangle 1910b are scaled in proportion to the resulting change in length of side 13b. The resulting triangle 1910b' is shown in FIG. 21A adjacent to triangle 1910a.

[0245] According to some implementations, the side lengths of other triangles adjacent to triangles 1910a and 1910b may all be determined in a similar manner until all audio device positions within environment 1900 have been determined.

[0246] Some examples of audio device positions may be as follows: Each audio device may report the DOA of all other audio devices in an environment (e.g., a room) based on the sounds produced by all other audio devices in the environment. The Cartesian coordinates of the ith audio device are

number

[0247] FIG. 21B shows an example of estimating the interior angles of a triangle formed by three audio devices. In this example, the audio devices are i, j, and k. The DOA of a sound source emanating from device j as observed from device i is θ ji The DOA of the sound source emanating from device k observed from device i is θ ki In the example shown in FIG. ji and θ ki is measured from axis 2105a, the orientation of which is arbitrary and may correspond, for example, to the orientation of audio device i. ki -θ ji It may be observed that the calculation of the interior angle a does not depend on the orientation of the axis 2105a.

[0248] In the example shown in FIG. ij and θ kj is measured from axis 2105b, which may have any orientation and may correspond to the orientation of audio device j. The interior angle b of triangle 2110 is given by b=θ ij -θ kj Similarly, in this example, θ jk and θ ik is measured from the axis 2105c. The interior angle c of the triangle 2110 is c=θ jk -θ ik It may be expressed as:

[0249] In the presence of measurement error, a+b+c ≠ 180°. Robustness can be improved by predicting each angle from the other two and averaging, for example:

number

[0250] In some implementations, the edge lengths (A,B,C) may be calculated (excluding scaling errors) by applying the sine law. In some examples, an edge length may be assigned an arbitrary value, such as 1. For example, if A=1 and vertex

number

number

[0251] According to some implementations, the triangle parameterization process may be repeated for every possible subset of the three audio devices in the environment.

number

[0252] FIG. 22 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. The blocks of method 2200, as well as other methods described herein, are not necessarily performed in the order shown. Moreover, such methods may include more or fewer blocks than shown and / or described. In this implementation, method 2200 includes estimating the position of a speaker in an environment. The blocks of method 2200 may be performed by one or more devices, which may be (or may include) device 100 shown in FIG. 1A.

[0253] In this example, block 2205 includes obtaining direction of arrival (DOA) data for each audio device of the plurality of audio devices. In some examples, the plurality of audio devices may include all of the audio devices in an environment, such as all of the audio devices 1905 shown in FIG.

[0254] However, in some cases, the plurality of audio devices may include only a subset of all audio devices in the environment, for example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0255] The DOA data may be obtained in various manners, depending on the particular implementation. In some instances, determining the DOA data may include determining DOA data for at least one audio device of the plurality of audio devices. For example, determining the DOA data may include receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining the DOA data may include receiving antenna data from one or more antennas corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0256] In some such examples, a single audio device may determine the DOA data itself. According to some such implementations, each audio device of the plurality of audio devices may determine its own DOA data. However, in other implementations, other devices, which may be local or remote devices, may determine the DOA data for one or more audio devices in the environment. According to some implementations, a server may determine the DOA data for one or more audio devices in the environment.

[0257] According to this example, block 2210 includes determining interior angles for each of a plurality of triangles based on the DOA data. In this example, each triangle of the plurality of triangles has vertices corresponding to audio device positions of three of the audio devices. Several such examples are described above.

[0258] 23 shows an example where each audio device in the environment is a vertex of multiple triangles, with each triangle side corresponding to the distance between two audio devices 1905.

[0259] In this implementation, block 2215 includes determining a side length for each side of each triangle. (The sides of a triangle are sometimes referred to herein as "edges.") According to this example, the side lengths are based, at least in part, on the interior angles. In some cases, the side lengths may be calculated by determining a first length of a first side of the triangle and determining lengths of a second and third side of the triangle based on the interior angles of the triangle. Several such examples are described above.

[0260] According to some such implementations, determining the first length may include setting the first length to a predetermined value. However, the determination of the first length may, in some examples, be based on time of arrival data and / or received signal strength data. The time of arrival data and / or received signal strength data may, in some implementations, correspond to sound waves from a first audio device in the environment that are detected by a second audio device in the environment. Alternatively or additionally, the time of arrival data and / or received signal strength data may correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) from a first audio device in the environment that are detected by a second audio device in the environment.

[0261] According to this example, block 2220 includes performing a forward alignment process to align each of the plurality of triangles in the first sequence. According to this example, the forward alignment process generates a forward alignment matrix.

[0262] According to some such examples, a triangle may be formed by dividing an edge (x i ,x j ) are expected to be aligned in such a way that they are equal to adjacent edges.

number

number

[0263] Figure 24 gives an example of a portion of the sorting process. The numbers 1 through 5 shown in bold in Figure 24 correspond to the audio device positions shown in Figures 1, 2 and 5. The sequence of the sorting process shown in Figure 24 and described herein is merely an example.

[0264] In this example, as in Figure 21A, the length of side 13b of triangle 1910b is forced to match the length of side 13a of triangle 1910a. The resulting triangle 1910b' is shown in Figure 24, with the same interior angles maintained. According to this example, the length of side 13c of triangle 1910c is also forced to match the length of side 13a of triangle 1910a. The resulting triangle 1910c' is shown in Figure 24, with the same interior angles maintained.

[0265] Next, in this example, the length of side 34b of triangle 1910d is forced to match the length of side 34a of triangle 1910b'. Additionally, in this example, the length of side 23b of triangle 1910d is forced to match the length of side 23a of triangle 1910a. The resulting triangle 1910d' is shown in Figure 24, with the same interior angles maintained. According to some such examples, the remaining triangles shown in Figure 5 may be treated in the same manner as triangles 1910b, 1910c, and 1910d.

[0266] The results of the sorting process may be stored in a data structure. According to some such examples, the results of the sorting process may be stored in a sorting matrix. For example, the results of the sorting process may be stored in a matrix

number

[0267] If the DOA data and / or the initial edge length determination contain errors, multiple estimates of the audio device position will result, and these errors will generally increase during the forward alignment process.

[0268] FIG. 25 shows an example of multiple estimates of audio device positions generated during the forward alignment process. In this example, the forward alignment process is based on a triangle with seven audio device positions as vertices, where the triangle does not align perfectly due to additive errors in the DOA estimates. The positions numbered 1-7 shown in FIG. 25 correspond to the estimated audio device positions generated by the forward alignment process. In this example, the audio device position estimates labeled "1" match, while the audio device position estimates for audio devices 6 and 7 show a larger difference. This is indicated by the relatively large area over which the numbers 6 and 7 are located.

[0269] Returning to FIG. 22, in this example, block 2225 includes an inverse sorting process that sorts each of the plurality of triangles in a second sequence that is the inverse of the first sequence. According to some implementations, the inverse sorting process may include traversing ε as before, but in reverse order. In alternative examples, the inverse sorting process may not be exactly the reverse of the sequence of operations of the forward sorting process. According to this example, the inverse sorting process generates an inverse sorting matrix, which is referred to herein as

number

[0270] Figure 26 shows an example of a portion of the reverse alignment process. Numbers 1-5 shown in bold in Figure 26 correspond to the audio device positions shown in Figures 19, 21, and 23. The sequence of the reverse alignment process shown in Figure 26 and described herein is merely an example.

[0271] In the example shown in FIG. 26, triangle 1910e is based on audio device positions 3, 4, and 5. In this implementation, the side lengths (or "edges") of triangle 1910e are assumed to be correct, and the side lengths of the adjacent triangles are forced to match them. According to this example, the length of side 45b of triangle 1910f is forced to match the length of side 45a of triangle 1910e. The resulting triangle 1910f', with its interior angles remaining the same, is shown in FIG. 26. In this example, the length of side 35b of triangle 1910c is forced to match the length of side 35a of triangle 1910e. The resulting triangle 1910c", with its interior angles remaining the same, is shown in FIG. 26. According to some such examples, the remaining triangles shown in FIG. 23 may be processed similarly to triangles 1910c and 1910f until the reverse alignment process has included all of the remaining triangles.

[0272] FIG. 27 shows an example of multiple estimates of audio device positions resulting from the inverse alignment process. In this example, the inverse alignment process is based on a triangle with the same seven audio device positions as vertices as described above with reference to FIG. 25. Positions numbered 1 through 7 shown in FIG. 27 correspond to estimated audio device positions generated by the inverse alignment process. Again, the triangles do not align perfectly due to additive errors in the DOA estimates. In this example, the audio device position estimates labeled 6 and 7 match, while the audio device position estimates for audio devices 1 and 2 show larger differences.

[0273] Returning to FIG. 22, block 2230 includes generating a final estimate of each audio device position based, at least in part, on values ​​of the forward alignment matrix and the inverse alignment matrix. In some examples, generating the final estimate of each audio device position may include translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling the inverse alignment matrix to generate a translated and scaled inverse alignment matrix.

[0274] For example, translation and scaling can be achieved by moving the center of mass to the origin and enforcing the unit Frobenius norm, e.g.

number

[0275] According to some such examples, generating a final estimate of each audio device position may include generating a rotation matrix based on a translated and scaled forward alignment matrix and a translated and scaled reverse alignment matrix. The rotation matrix may include a plurality of estimated audio device positions for each audio device. An optimal rotation between the forward alignment and the reverse alignment may be found, for example, by singular value decomposition. In some such examples, generating the rotation matrix may include performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, for example, as follows:

number

[0276] In the above formula, U represents the left singular vector and V is the matrix

number

number

[0277] According to some examples, the rotation matrix R = VU T After determining, the alignments may be averaged, for example as follows:

number

[0278] In some implementations, generating the final estimate of each audio device position also includes averaging the estimated audio device positions for each audio device to generate the final estimate of each audio device position. Various disclosed implementations have proven robust even when the DOA data and / or other calculations contain significant errors. For example,

number

number

[0279] FIG. 28 shows a comparison of the estimated audio device positions with the actual audio device positions. In the examples shown in FIG. 28, the audio device positions correspond to positions estimated during the forward and reverse alignment processes described above with reference to FIG. 17 and FIG. 19. In these examples, the error in the DOA estimation had a standard deviation of 15 degrees. Nevertheless, the final estimate of each audio device position (each represented by an "x" in FIG. 28) matches well with the actual audio device position (each represented by a circle in FIG. 28).

[0280] Much of the above discussion concerns auto-location of audio devices. The following discussion builds on some of the methods of determining the listener's position and listener's angular orientation briefly described above. In the above discussion, the term "rotation" is used essentially the same way that the term "orientation" is used in the following discussion. For example, the above "rotation" may refer to a global rotation of the final speaker geometry, and not a rotation of individual triangles during the process described above with reference to FIG. 14 et seq. This global rotation or orientation may be resolved with reference to the listener's angular orientation, e.g., by the direction the listener is looking, by the direction the listener's nose is pointing, etc.

[0281] Various satisfactory methods for estimating the listener position are described below. However, estimating the listener angular orientation can be difficult. Below, some related methods are detailed.

[0282] Determining the listener position and listener angular orientation can enable several desirable features, such as orienting a localized audio device relative to the listener. Knowing the listener's position and angular orientation allows for determining, for example, which speakers in the environment are in front of the listener, which are in the back, and which (if any) are near the center.

[0283] After correlating between the audio device positions and the listener positions and orientations, some implementations may involve providing the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data to an audio rendering system. Alternatively or additionally, some implementations may involve an audio data rendering process based at least in part on the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data.

[0284] FIG. 29 is a flow diagram outlining an example of a method that may be performed by an apparatus such as that shown in FIG. 1A. The blocks of method 2900, as well as other methods described herein, are not necessarily performed in the order shown. Furthermore, such methods may include more or fewer blocks than shown and / or described. In this example, the blocks of method 2900 are performed by a control system that may be (or may include) control system 110 shown in FIG. 1A. As noted above, in some implementations, control system 110 may reside in a single device, while in other implementations, control system 110 may reside in two or more devices.

[0285] In this example, block 2905 involves obtaining direction of arrival (DOA) data for each of a plurality of audio devices in an environment. In some examples, the plurality of audio devices may include all of the audio devices in the environment, such as all of the audio devices 1905 shown in FIG.

[0286] However, in some cases, the plurality of audio devices may include only a subset of all audio devices in the environment, for example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0287] The DOA data may be obtained in a variety of ways, depending on a particular implementation. In some instances, determining the DOA data may include determining DOA data for at least one audio device of the plurality of audio devices. In some examples, the DOA data may be obtained by controlling each loudspeaker of a plurality of loudspeakers in the environment to play a test signal. For example, determining the DOA data may involve receiving microphone data from each microphone of a plurality of audio device microphones corresponding to a single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining the DOA data may involve receiving antenna data from one or more antennas corresponding to the single audio device of the plurality of audio devices and determining the DOA data for the single audio device based at least in part on the antenna data.

[0288] In some such examples, a single audio device may determine the DOA data itself. According to some such implementations, each audio device of a plurality of audio devices may determine its own DOA data. However, in other implementations, other devices, which may be local or remote devices, may determine the DOA data for one or more audio devices in the environment. According to some implementations, a server may determine the DOA data for one or more audio devices in the environment.

[0289] 29, block 2910 includes generating, via the control system, audio device position data based at least in part on the DOA data. In this example, the audio device position data includes an estimate of an audio device position for each audio device referenced in block 2905.

[0290] The audio device position data may be (or may include) coordinates in a coordinate system, such as, for example, a Cartesian coordinate system, a spherical coordinate system, or a cylindrical coordinate system. This coordinate system may be referred to herein as the audio device coordinate system. In some such examples, the audio device coordinate system may be oriented with reference to one of the audio devices in the environment. In other examples, the audio device coordinate system may be oriented with reference to an axis defined by a line between two audio devices in the environment. However, in other examples, the audio device coordinate system may be oriented with reference to another part of the environment, such as a television, a wall of a room, etc.

[0291] In some examples, block 2910 may involve the process described above with reference to FIG. 22. According to some such examples, block 2910 may involve determining interior angles for each of a plurality of triangles based on the DOA data. In some instances, each triangle of the plurality of triangles may have vertices corresponding to audio device positions of three audio devices. Some such methods may involve determining side lengths for each side of each triangle based, at least in part, on the interior angles.

[0292] Some such methods may involve performing a forward alignment process to align each of the plurality of triangles in a first sequence to generate a forward alignment matrix. Some such methods may involve performing a reverse alignment process to align each of the plurality of triangles in a second sequence that is the reverse of the first sequence to generate a reverse alignment matrix. Some such methods may involve generating a final estimate of the position of each audio device based, at least in part, on values ​​of the forward alignment matrix and values ​​of the reverse alignment matrix. However, in some implementations of method 2900, block 2910 may involve applying methods other than those described above with reference to FIG. 22.

[0293] In this example, block 2915 involves determining, via the control system, listener position data indicative of the listener's position within the environment. The listener position data may be, for example, position data referenced to an audio device coordinate system. However, in other examples, the coordinate system may be oriented with reference to the listener or with reference to a part of the environment, such as a television, a wall of a room, etc.

[0294] In some examples, block 2915 may include prompting the listener (e.g., via audio prompts from one or more loudspeakers in the environment) to make one or more vocalizations and estimating the listener position according to the DOA data. The DOA data may correspond to microphone data acquired by a plurality of microphones in the environment. The microphone data may correspond to detection of the one or more vocalizations by the microphones. At least some of the microphones may be co-located with the loudspeakers. According to some examples, block 2915 may include a triangulation process. For example, block 2915 may include triangulating the voice of the user by finding an intersection between DOA vectors passing through the audio device, e.g., as described below with reference to FIG. 30A. According to some implementations, block 2915 (or another operation of method 2900) may include co-locating the origin of the audio device coordinate system and the listener coordinate system after the listener position is determined. Co-locating the origins of the audio device coordinate system and the listener coordinate system may include transforming the audio device positions from the audio device coordinate system to the listener coordinate system.

[0295] According to this implementation, block 2920 includes determining, via the control system, listener angular orientation data indicative of a listener angular orientation. The listener angular orientation data may be created with reference to a coordinate system used to represent the listener position data, such as, for example, an audio device coordinate system. In some such examples, the listener angular orientation data may be created with reference to an origin and / or axes of the audio device coordinate system.

[0296] However, in some implementations, the listener angular orientation data may be created with reference to axes defined by the listener position and another point in the environment, such as a television, audio equipment, a wall, etc. In some such implementations, the listener position may be used to define the origin of a listener coordinate system. In some such examples, the listener angular orientation data may be created with reference to axes of the listener coordinate system.

[0297] Various methods for implementing block 2920 are disclosed herein. According to some examples, the listener angular orientation may correspond to a listener viewing direction. In some such examples, the listener viewing direction may be estimated with reference to listener position data, for example by assuming that the listener is looking at a particular object such as a television. In some such implementations, the listener viewing direction may be determined according to the listener position and the television position. Alternatively or additionally, the listener viewing direction may be determined according to the listener position and the position of a sound bar of the television.

[0298] However, in some examples, the listener's viewing direction may be determined according to listener input. According to some such examples, the listener input may include inertial sensor data received from a device held by the listener. The listener may use the device to point to a location in the environment, e.g., a location that corresponds to a direction in which the listener is facing. For example, the listener may use the device to point to a loudspeaker (a speaker that is playing sound). Thus, in such examples, the inertial sensor data may include inertial sensor data that corresponds to a loudspeaker that is playing sound.

[0299] In some such instances, the listener input may include an indication of an audio device selected by the listener, which in some examples may include inertial sensor data corresponding to the selected audio device.

[0300] However, in other examples, the instruction of the audio device may be made in accordance with one or more listener utterances (e.g., "The TV is now in front of me," "Speaker 2 is now in front of me," etc.) Other examples of determining listener angular orientation data in response to one or more listener utterances are described below.

[0301] According to the example shown in FIG. 29, block 2925 includes determining, via the control system, audio device angular orientation data indicating an audio device angular orientation for each audio device relative to the listener position and listener angular orientation. According to some such examples, block 2925 may include rotating audio device coordinates about a point defined by the listener position. In some implementations, block 2925 may include transforming the audio device position data from the audio device coordinate system to the listener coordinate system. Some examples are described below.

[0302] Figure 30A shows examples of some of the blocks of Figure 29. According to some such examples, the audio device position data includes audio device position estimates for each of audio devices 1-5 with reference to an audio device coordinate system 3007. In this implementation, audio device coordinate system 3007 is a Cartesian coordinate system with its origin at the microphone position of audio device 2, where the x-axis of audio device coordinate system 3007 corresponds to line 3003 between the microphone position of audio device 2 and the microphone position of audio device 1.

[0303] In this example, the listener location is determined by prompting a listener 3005, shown seated on a couch 1903 (e.g., via audio prompts from one or more loudspeakers in the environment 3000a) to make one or more utterances 3027, and estimating the listener location according to time of arrival (TOA) data. The TOA data corresponds to microphone data acquired by a number of microphones in the environment. In this example, the microphone data corresponds to detection of said one or more utterances 3027 by at least some (e.g., three, four or all five) microphones of audio devices 1-5.

[0304] Alternatively or additionally, the listener position according to DOA data provided by at least some (e.g., two, three, four or all five) microphones of the audio devices 1-5. According to some such examples, the listener position may be determined according to the intersection of lines 3009a, 3009b, etc. corresponding to the DOA data.

[0305] According to this example, the listener position corresponds to the origin of the listener coordinate system 3020. In this example, the listener angular orientation data is represented by the y'-axis of the listener coordinate system 3020, which corresponds to a line 3013a between the listener's head 3010 (and / or the listener's nose 3025) and the soundbar 3030 of the television 101. In the example shown in FIG. 30A, the line 3013a is parallel to the y'-axis. Thus, the angle Θ represents the angle between the y-axis and the y'-axis. In this example, block 2925 of FIG. 29 may include a rotation of the audio device coordinates by the angle Θ about the origin of the listener coordinate system 3020. Thus, although the origin of audio device coordinate system 3007 is shown in FIG. 30A to correspond to audio device 2, some implementations include co-locating the origin of audio device coordinate system 3007 with the origin of listener coordinate system 3020 prior to the rotation of the audio device coordinates by angle Θ about the origin of listener coordinate system 3020. This co-location may be performed by a coordinate transformation from audio device coordinate system 3007 to listener coordinate system 3020.

[0306] The location of the soundbar 3030 and / or the television 1901 may be determined in some examples by having the soundbar emit a sound and estimating the location of the soundbar according to DOA and / or TOA data, which may correspond to detection of said sound by at least some (e.g., three, four or all five) microphones of the audio devices 1-5. Alternatively or additionally, the location of the soundbar 3030 and / or the television 1901 may be determined by prompting a user to walk up to the television and locating the user's speech by DOA and / or TOA data, which may correspond to detection of said sound by at least some (e.g., three, four or all five) microphones of the audio devices 1-5. Such methods may involve triangulation. Such examples may be useful in situations where the soundbar 3030 and / or the television 1901 do not have associated microphones.

[0307] In some other examples where the soundbar 3030 and / or television 1901 have associated microphones, the position of the soundbar 3030 and / or television 1901 can be determined according to a TOA or DOA method, such as the DOA methods disclosed herein. According to some such methods, the microphones may be co-located with the soundbar 3030.

[0308] According to some implementations, the soundbar 3030 and / or the television 1901 may have an associated camera 3011. The control system may be configured to capture an image of the listener's head 3010 (and / or the listener's nose 3025). In some such examples, the control system may be configured to determine a line 3013a between the listener's head 3010 (and / or the listener's nose 3025) and the camera 3011. The listener angular orientation data may correspond to the line 3013a. Alternatively or additionally, the control system may be configured to determine an angle Θ between the line 3013a and the y-axis of the audio device coordinate system.

[0309] FIG. 30B shows a further example of determining listener angular orientation data. According to this example, the listener position has already been determined in block 2915 of FIG. 29. Now, a control system is controlling the speakers of environment 3000b to render audio object 3035 at various positions within environment 3000b. In some such examples, the control system can cause the speakers to render audio object 3035 in such a way that it feels like it is rotating around listener 3005. Such rendering can be, for example, by rendering audio object 3035 such that it feels like it is rotating around the origin of listener coordinate system 3020. In this example, curved arrow 3040 shows a portion of the trajectory of audio object 3035 as it rotates around listener 3005.

[0310] According to some such examples, the listener 3005 may provide user input indicating when the audio object 3035 is in the direction the listener 3005 is facing (e.g., by saying "stop"). In some such examples, the control system may be configured to determine a line 3013b between the listener position and the position of the audio object 3035. In this example, the line 3013b corresponds to the y' axis of the listener coordinate system, indicating the direction the listener 3005 is facing. In alternative implementations, the listener 3005 may provide user input indicating when the audio object 3035 is in front of the environment, at a television location in the environment, at an audio device location, etc.

[0311] FIG. 30C shows a further example of determining listener angular orientation data. According to this example, the listener position has already been determined in block 2915 of FIG. 29. Here, listener 3005 is using handheld device 3045 to provide input regarding listener 3005's viewing direction by pointing handheld device 3045 towards television 1901 or soundbar 3030. The dashed outline of handheld device 3045 and listener's arm indicates that listener 3005 was pointing handheld device 3045, in this example towards audio device 2, at a time before listener 3005 was pointing handheld device 3045 towards television 1901 or soundbar 3030. In other examples, listener 3005 may be pointing handheld device 3045 towards another audio device, such as audio device 1. According to this example, the handheld device 3045 is configured to determine an angle α between the audio device 2 and the television 1901 or soundbar 3030. The angle α approximates the angle between the audio device 2 and the viewing direction of the listener 3005. The handheld device 3045 may, in some examples, be a cellular phone including an inertial sensor system and a wireless interface configured to communicate with a control system controlling the audio equipment of the environment 3000c. In some examples, the handheld device 3045 may be executing an application or "app" configured to control the handheld device 3045 to perform a desired function, for example, by providing a user prompt (e.g., via a graphical user interface), by receiving an input indicating that the handheld device 3045 is pointing in a desired direction, by storing corresponding inertial sensor data and / or by transmitting corresponding inertial sensor data to a control system controlling the audio equipment of the environment 3000c, etc.

[0312] According to this example, the control system (which may be a control system of the handheld device 3045 or a control system controlling the audio devices of the environment 3000c) is configured to determine the orientation of the lines 3013c and 3050 according to inertial sensor data, e.g., gyroscope data. In this example, the line 3013c is parallel to the axis y' and may be used to determine the listener angular orientation. According to some examples, the control system may determine an appropriate rotation for the audio device coordinates around the origin of the listener coordinate system 3020 depending on the angle α between the audio device 2 and the viewing direction of the listener 3005.

[0313] FIG. 30D illustrates an example of determining an appropriate rotation for audio device coordinates according to the method described with reference to FIG. 30C. In this example, the origin of the audio device coordinate system 3007 is co-located with the origin of the listener coordinate system 3020. Co-locating the origins of the audio device coordinate system 3007 and the listener coordinate system 3020 is possible after the process of determining the listener position 2915. Co-locating the origins of the audio device coordinate system 3007 and the listener coordinate system 3020 may involve transforming the audio device position from the audio device coordinate system 3007 to the listener coordinate system 3020. The angle α has been determined as described above with reference to FIG. 30C. Thus, the angle α corresponds to the desired orientation of audio device 2 in the listener coordinate system 3020. In this example, the angle β corresponds to the orientation of audio device 2 in the audio device coordinate system 3007. The angle θ, which in this example is β-α, indicates the rotation required to align the y-axis of the audio device coordinate system 3007 with the y′-axis of the listener coordinate system 3020.

[0314] In some implementations, the method of FIG. 29 may include controlling at least one of the audio devices in the environment based at least in part on the corresponding audio device positions, the corresponding audio device angular orientations, the listener position data, and the listener angular orientation data.

[0315] For example, some implementations may include providing the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data to an audio rendering system. In some examples, the audio rendering system may be implemented by a control system, such as the control system 110 of FIG. 1A. Some implementations may include controlling an audio data rendering process based at least in part on the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data. Some such implementations may include providing loudspeaker acoustic capability data to the rendering system. The loudspeaker acoustic capability data may correspond to one or more loudspeakers in the environment. The loudspeaker acoustic capability data may indicate an orientation of one or more drivers, a number of drivers, or a driver frequency response of one or more drivers. In some examples, the loudspeaker acoustic capability data may be retrieved from a memory and then provided to the rendering system.

[0316] One class of embodiments includes a method of rendering and / or playing audio for playback by at least one (e.g., all or a portion) of a plurality of coordinated (orchestrated) smart audio devices. For example, a collection of smart audio devices present in a user's home (in a system) may be orchestrated to handle a variety of simultaneous use cases, including flexible rendering of audio for playback by all or a portion of the smart audio devices (i.e., all or a portion of the speaker(s)). Many interactions with the system are possible that require dynamic modifications to the rendering and / or playback. Such modifications may be, but are not necessarily, focused on spatial fidelity.

[0317] In the context of rendering (or rendering and playing) a spatial audio mix (e.g., rendering a stream of audio or multiple streams of audio) for playback by a collection of smart audio devices (or by another set of speakers), the types of speakers (e.g., within or coupled to a smart audio device) may vary, and thus the corresponding acoustic capabilities of the speakers may vary greatly. In one example of an audio environment shown in FIG. 3A, loudspeakers 305d, 305f and 305h may be smart speakers with a single 0.6 inch speaker. In this example, loudspeakers 305b, 305c, 305e and 305f may be smart speakers with a 2.5 inch woofer and a 0.8 inch tweeter. According to this example, loudspeaker 305g may be a smart speaker with a 5.25 inch woofer, three 2 inch mid-range speakers, and a 1.0 inch tweeter. Here, loudspeaker 305a may be a soundbar with sixteen 1.1 inch beam drivers and two 4 inch woofers. Thus, the low frequency capabilities of smart speakers 305d and 305f will be significantly lower than other loudspeakers in environment 200, especially loudspeakers with 4 inch or 5.25 inch woofers.

[0318] Figure 31 is a block diagram illustrating examples of components of a system in which various aspects of the disclosure may be implemented. As with other figures provided herein, the types and number of elements shown in Figure 31 are provided merely as examples. Other implementations may include more elements, fewer elements, and / or different types and numbers of elements.

[0319] According to this example, the system 3100 includes a smart home hub 3105 and loudspeakers 3125a-3125m. In this example, the smart home hub 3105 includes an instance of the control system 110 shown in FIG. 1A and described above. According to this implementation, the control system 110 includes a listening environment dynamics processing configuration data module 3110, a listening environment dynamics processing module 3115, and a rendering module 3120. Some examples of the listening environment dynamics processing configuration data module 3110, the listening environment dynamics processing module 3115, and the rendering module 3120 are described below. In some examples, the rendering module 3120' may be configured for both rendering and listening environment dynamics processing.

[0320] As suggested by the arrows between the smart home hub 3105 and the loudspeakers 3125a-m, the smart home hub 3105 also includes an instance of the interface system 105 shown in FIG. 1A and described above. According to some examples, the smart home hub 3105 may be part of the environment 300 shown in FIG. 3A. In some cases, the smart home hub 3105 may be implemented by a smart speaker, a smart TV, a cellular phone, a laptop, etc. In some implementations, the smart home hub 3105 may be implemented by software, for example, via a downloadable software application or "app" software. In some cases, the smart home hub 3105 may be implemented in each of the loudspeakers 3125a-m, all operating in parallel to generate the same processed audio signal from the module 3120. According to some such examples, for each loudspeaker, the rendering module 3120 may then generate one or more speaker feeds associated with each loudspeaker or group of loudspeakers and provide these speaker feeds to a respective speaker dynamics processing module.

[0321] In some instances, loudspeakers 3125a-3125m may include loudspeakers 305a-305h of Figure 3A. In other examples, loudspeakers 3125a-3125m may be or include other loudspeakers. Thus, in this example, system 3100 includes M loudspeakers, where M is an integer greater than 2.

[0322] Smart speakers, like many other powered speakers, typically employ some type of internal dynamics processing to prevent the speaker from distorting. Such dynamics processing often involves a signal limiting threshold (e.g., a limiting threshold that is variable across frequency) below which the signal level is dynamically held. For example, Dolby's Audio Regulator, one of several algorithms in the Dolby Audio Processing (DAP) audio post-processing suite, provides such processing. In some cases, but not typically via a smart speaker's dynamics processing module, dynamics processing may also involve applying one or more compressors, gates, expanders, duckers, etc.

[0323] Thus, in this example, each of the loudspeakers 3125a-3125m includes a corresponding speaker dynamics processing (DP) module A-M. The speaker dynamics processing modules are configured to apply individual loudspeaker dynamics processing configuration data for each individual loudspeaker of the listening environment. Speaker DP module A is configured, for example, to apply individual loudspeaker dynamics processing configuration data appropriate for loudspeaker 3125a. In some examples, the individual loudspeaker dynamics processing configuration data may correspond to one or more capabilities of the individual loudspeaker, for example the ability of the loudspeaker to reproduce audio data within a particular frequency range at a particular level without perceptible distortion.

[0324] When spatial audio is rendered across a collection of heterogeneous speakers (e.g., the speakers of a smart audio device, or speakers coupled to a smart audio device), each with potentially different playback limitations, care must be taken when performing dynamics processing on the overall mix. A simple solution is to render the spatial mix to the speaker feeds of each participating speaker, and then allow the dynamics processing module associated with each speaker to act independently on its corresponding speaker feed according to the limitations of that speaker.

[0325] This approach avoids distorting each speaker, but may dynamically shift the spatial balance of the mix in a perceptually annoying way. For example, referring to FIG. 3A, assume that a television program is shown on the television 330 and the corresponding audio is being played by the loudspeakers of the environment 300. Assume that during the television program, audio associated with a stationary object (such as a heavy machinery unit in a factory) is intended to be rendered at a specific location in the environment 300. Furthermore, assume that the dynamics processing module associated with the loudspeaker 305d reduces the level of the bass range audio substantially more than the dynamics processing module associated with the loudspeaker 305b, since the loudspeaker 305b has a substantially greater ability to reproduce bass range sounds. If the volume of the signal associated with the stationary object fluctuates, when the volume increases, the dynamics processing module associated with the loudspeaker 305d reduces the level of the bass range audio substantially more than the level of the same audio is reduced by the dynamics processing module associated with the loudspeaker 305b. This level difference changes the apparent location of the stationary object. Thus, an improved solution is needed.

[0326] Some embodiments of the present disclosure are systems and methods for rendering (or rendering and playing) a spatial audio mix (e.g., rendering a stream of audio or multiple streams of audio) for playback by at least one (e.g., all or part) of a smart audio device of a collection of smart audio devices (e.g., a collection of coordinated smart audio devices) and / or at least one (e.g., all or part) of a speaker of another collection of speakers. Some embodiments are methods (or systems) for such rendering (e.g., including generating speaker feeds) and playing the rendered audio (e.g., playing the generated speaker feeds). Examples of such embodiments are as follows:

[0327] Systems and methods for audio processing may include rendering audio (e.g., rendering a spatial audio mix by rendering a stream of audio or multiple streams of audio) for playback by at least two speakers (e.g., all or a portion of the speakers of a collection of speakers), including by: (a) combining individual loudspeaker dynamics processing configuration data (e.g., limiting thresholds for individual loudspeakers) to thereby determine listening environment dynamics processing configuration data (e.g., combined thresholds) for the multiple loudspeakers; (b) performing dynamics processing on the audio (e.g., the stream of audio representing the spatial audio mix) using the listening environment dynamics processing configuration data (e.g., the combined thresholds) for the multiple loudspeakers to generate processed audio; (c) Rendering the processed audio to the speaker feeds.

[0328] According to some implementations, process (a) may be performed by a module such as the listening environment dynamics processing configuration data module 3110 shown in FIG. 31. The smart home hub 3105 may be configured to obtain, via the interface system, individual loudspeaker dynamics processing configuration data for each of the M loudspeakers. In this implementation, the individual loudspeaker dynamics processing configuration data includes an individual loudspeaker dynamics processing configuration data set for each loudspeaker of the plurality of loudspeakers. According to some examples, the individual loudspeaker dynamics processing configuration data for one or more loudspeakers may correspond to one or more capabilities of the one or more loudspeakers. In this example, each of the individual loudspeaker dynamics processing configuration data sets includes at least one type of dynamics processing configuration data. In some examples, the smart home hub 3105 may be configured to obtain the individual loudspeaker dynamics processing configuration data set by querying each loudspeaker 3125a-3125m. In other implementations, the smart home hub 3105 may be configured to obtain the individual loudspeaker dynamics processing configuration data sets by querying a data structure of previously obtained individual loudspeaker dynamics processing configuration data sets stored in memory.

[0329] In some examples, process (b) may be performed by a module such as the listening environment dynamics processing module 3115 of Figure 31. Some detailed examples of processes (a) and (b) are described below.

[0330] In some examples, the rendering of process (c) may be performed by a module such as the rendering module 3120 or the rendering module 3120' of FIG. 31. In some embodiments, the audio processing involves: (d) performing dynamics processing on the rendered audio signal in accordance with individual loudspeaker dynamics processing setting data for each loudspeaker (e.g., limiting the speaker feed in accordance with a playback limiting threshold associated with the corresponding speaker, thereby generating a limited speaker feed). Process (d) may be performed, for example, by dynamics processing modules A-M shown in FIG. 31.

[0331] The speaker may be a speaker of (or coupled to) at least one (e.g., all or a portion) of the smart audio devices of the collection of smart audio devices. In some implementations, to generate the limited speaker feeds in step (d), the speaker feeds generated in step (c) may be processed by a second stage of dynamics processing (e.g., by an associated dynamics processing system of each speaker) to, e.g., generate speaker feeds prior to final playback through the speakers. For example, the speaker feeds (or a subset or portion thereof) may be fed to a dynamics processing system of each different one of the speakers (e.g., a dynamics processing subsystem of a smart audio device, where the smart audio device includes or is coupled to an associated one of those speakers). The processed audio output from each said dynamics processing system may be used to generate a speaker feed for an associated one of the speakers. Following speaker-specific dynamics processing (i.e., dynamics processing performed independently for each speaker), the processed (e.g., dynamically limited) speaker feeds may be used to drive the speakers to cause playback of the audio.

[0332] The first stage of dynamics processing (step (b)) may be designed to reduce perceptually bothersome shifts in spatial balance that would occur if steps (a) and (b) were omitted and the dynamically processed (e.g., limited) speaker feeds resulting from step (d) were generated in response to the original audio (rather than in response to the processed audio generated in step (b)). This may prevent undesirable shifts in the spatial balance of the mix. The second stage of dynamics processing, operating on the rendered speaker feeds from step (c), may be designed to ensure that no speakers are distorted, since the dynamics processing of step (b) may not necessarily ensure that signal levels have fallen below threshold for all speakers. Combining the individual loudspeaker dynamics processing configuration data (e.g. combining the thresholds in the first stage (step (a))) may in some examples involve (e.g. include) averaging the individual loudspeaker dynamics processing configuration data (e.g. limiting thresholds) across speakers (e.g. across smart audio devices) or taking the minimum of the individual loudspeaker dynamics processing configuration data (e.g. limiting thresholds) across speakers (e.g. across smart audio devices).

[0333] In some implementations, if the first stage of dynamics processing (step (b)) operates on audio exhibiting a spatial mix (e.g., audio of an object-based audio program including at least one object channel and optionally also at least one speaker channel), this first stage may be implemented according to techniques for audio object processing through the use of spatial zones. In such a case, the combined individual loudspeaker dynamics processing configuration data (e.g., combined limiting thresholds) associated with each zone may be derived by (or as) a weighted average of the individual loudspeaker dynamics processing configuration data (e.g., individual speaker limiting thresholds), the weighting being given or determined, at least in part, by the spatial proximity of each speaker to and / or its position within the zone.

[0334] In one exemplary embodiment, assume a number M of speakers (M≧2), where each speaker is indexed by a variable i. Each speaker i is assigned a frequency-varying playback limit threshold T i [f], where the variable f represents an index into a finite set of frequencies for which thresholds are specified. (Note that if the set of frequencies has a size of 1, then the corresponding single threshold is considered broadband and applies across the entire frequency range.) These thresholds are utilized by each speaker in its own independent dynamics processing function to limit the audio signal below the threshold for a specific purpose, such as preventing the speaker from distorting, or from playing above some level that is deemed objectionable in its vicinity.

[0335] 32A, 32B, and 32C show examples of play limit thresholds and corresponding frequencies. The range of frequencies shown can span, for example, the range of frequencies audible to an average human (e.g., 20 Hz to 20 kHz). In these examples, the play limit threshold is shown by the vertical axis of graphs 3200a, 3200b, and 3200c, which is labeled "Level Threshold" in these examples. The play limit / level threshold increases in the direction of the arrows on the vertical axis. The play limit / level threshold can be expressed, for example, in decibels. In these examples, the horizontal axis of graphs 3200a, 3200b, and 3200c shows frequency, which increases in the direction of the arrows on the horizontal axis. The play limit thresholds shown by curves 3200a, 3200b, and 3200c can be implemented, for example, by dynamics processing modules of the individual loudspeakers.

[0336] Graph 3200a of FIG. 32A illustrates a first example of the playback limit threshold as a function of frequency. Curve 3205a illustrates the playback limit threshold for each corresponding frequency value. In this example, the base frequency f b At input level T i The input audio received at o The base frequency f b may be in the range of 60 to 250 Hz, for example. However, in this example, the treble frequency f t At input level T i The input audio received at the i and is output by the dynamics processing module. t may be, for example, in the range above 1280 Hz. Thus, in this example, curve 3205a corresponds to a dynamics processing module that applies a significantly lower threshold for the base frequencies than for the treble frequencies. Such a dynamics processing module may be suitable for loudspeakers without woofers (e.g., loudspeaker 305d of FIG. 3A).

[0337] Graph 3200b of Figure 32B shows a second example of the playback limit threshold as a function of frequency. Curve 3205b shows the same base frequency f b At input level T i The input audio received at the o 3A. Thus, in this example, curve 3205b corresponds to a dynamics processing module that does not apply a threshold for the base frequency as low as curve 3205a. Such a dynamics processing module is suitable for speakers with at least small woofers (e.g., speaker 305b of FIG. 3A).

[0338] Graph 3200c in Figure 32C shows a second example of the playback limit threshold as a function of frequency. Curve 3205c (which is a straight line in this example) shows the same base frequency f b At input level T i 3205c shows that input audio received at is output by the dynamics processing module at the same level. Thus, in this example, curve 3205c corresponds to a dynamics processing module that may be appropriate for a loudspeaker that can reproduce a wide range of frequencies, including the base frequency. For simplicity, it can be seen that the dynamics processing module could approximate curve 3205c by implementing curve 3205d, which applies the same threshold for all frequencies shown.

[0339] The spatial audio mix may be rendered for multiple speakers using known rendering systems such as Center of Mass Amplitude Panning (CMAP) or Flexible Virtualization (FV). From the components of the spatial audio mix, the rendering system generates one speaker feed for each of the multiple speakers. In some previous examples, the speaker feeds are then filtered by the associated dynamics processing function of each speaker, with a threshold Ti [f]. Without the benefit of this disclosure, this described rendering scenario may result in annoying shifts in the perceived spatial balance of the rendered spatial audio mix. For example, one of the M speakers, such as on the right side of the listening area, may be much less capable (e.g., in rendering bass-range audio) than the other speakers, and thus the threshold for that speaker may be significantly lower than the thresholds of the other speakers, at least in a certain frequency range. During playback, the dynamics processing module for this speaker will reduce the levels of the right-side spatial mix components more significantly than the left-side components. Listeners are very sensitive to such dynamic shifts between the left-right balance of the spatial mix and may find the result very annoying.

[0340] To address this issue, in some examples, individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) for individual speakers of a listening environment are combined to create listening environment dynamics processing configuration data for all loudspeakers in the listening environment. The listening environment dynamics processing configuration data may then be utilized to first perform dynamics processing in the context of the entire spatial audio mix before rendering to the speaker feeds. Because this first stage of dynamics processing has access to the entire spatial mix rather than just one individual speaker feed, processing may be performed in a manner that does not impart an undesirable shift to the perceived spatial balance of the mix. Individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) may be combined in a manner that eliminates or reduces the amount of dynamics processing performed by any of the individual speaker's independent dynamics processing functions.

[0341] In one example of determining listening environment dynamics processing configuration data, individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) for individual speakers are combined with listening environment dynamics processing configuration data (e.g., frequency-varying playback limiting thresholds) that are applied to all components of the spatial mix in a first stage of dynamics processing.

number

number

[0342] Such a combination essentially eliminates the action of individual dynamics processing for each speaker, since the spatial mix is ​​first limited at all frequencies below the threshold of the least capable speaker. However, such a strategy may be overly aggressive; many speakers will play at a lower level than they can accommodate, and the combined playback level of all speakers may be undesirably low. For example, if the threshold in the bass range shown in FIG. 32A were applied to the loudspeaker corresponding to the threshold for FIG. 32C, the playback level of the latter speaker would be unnecessarily low in the bass range. An alternative combination for determining listening environment dynamics processing configuration data is to average the individual loudspeaker dynamics processing configuration data across all speakers in the listening environment. For example, in the context of playback limiting thresholds, the average may be determined as follows:

number

[0343] This combination may increase the overall playback level compared to taking the minimum because the first stage of dynamics processing is limiting to a higher level, thereby allowing more capable speakers to play louder. For speakers whose individual limiting thresholds are below the average, their independent dynamics processing functions can limit the associated speaker feed if necessary. However, the first stage of dynamics processing may reduce this limiting requirement since some initial limiting has been performed on the spatial mix.

[0344] According to some examples of determining the listening environment dynamics processing configuration data, adjustable combinations can be generated that interpolate between the minimum and average of the individual loudspeaker dynamics processing configuration data through tuning parameters. For example, in the context of a playback limiting threshold, the interpolation can be determined as follows:

number

[0345] Other combinations of individual loudspeaker dynamics processing configuration data are possible, and this disclosure is intended to cover all such combinations.

[0346] 33A and 33B are graphs showing examples of dynamic range compression data. In graphs 3300a and 3300b, the input signal level in decibels is shown on the horizontal axis and the output signal level in decibels is shown on the vertical axis. As with other disclosed examples, the particular thresholds, ratios, and other values ​​are shown by way of example only and not by way of limitation.

[0347] In the example shown in Fig. 33A, the output signal level is equal to the input signal level below the threshold, which in this example is -10 dB. Other examples may involve different thresholds, for example -20 dB, -18 dB, -16 dB, -14 dB, -12 dB, -8 dB, -6 dB, -4 dB, -2 dB, 0 dB, 2 dB, 4 dB, 6 dB, etc. Above the threshold, various examples of compression ratios are shown. A ratio of N:1 means that above the threshold, the output signal level increases by 1 dB for every N dB increase in the input signal. For example, a compression ratio of 10:1 (line 3305e) means that above the threshold, the output signal level increases by 1 dB for every 10 dB increase in the input signal. A compression ratio of 1:1 (line 3305a) means that above the threshold, the output signal level is still the same as the input signal level. Lines 3305b, 3305c, and 3305d correspond to compression ratios of 3:2, 2:1, and 5:1. Other implementations may provide different compression ratios, such as 2.5:1, 3:1, 3.5:1, 4:3, 4:1, etc.

[0348] Figure 33B shows an example of a "knee," which controls how the compression ratio changes at or near a threshold, which in this example is 0 dB. According to this example, a compression curve with a "hard" knee consists of two straight line segments: a straight line segment 3310a up to the threshold and a straight line segment 3310b above the threshold. A hard knee is easier to implement, but can cause artifacts.

[0349] Also shown in FIG. 33B is an example of a "soft" knee. In this example, the soft knee spans 10 dB. According to this implementation, above and below the 10 dB span, the compression ratio of the compression curve with the soft knee is the same as the compression ratio of the compression curve with the hard knee. Other implementations may provide various other shapes of "soft" knees, which may span more or fewer decibels, may exhibit different compression ratios over the span, etc.

[0350] Other types of dynamic range compression data can include "attack" data and "release" data. Attack is the period during which the compressor decreases the gain, e.g., in response to an increased level at the input, to reach a gain determined by the compression ratio. Attack times for compressors typically range from 25 ms to 500 ms, although other attack times are practical. Release is the period during which the compressor increases the gain, e.g., in response to a reduced input level, to reach an output gain determined by the compression ratio (or the input level if the input level falls below a threshold). Release times may range, e.g., from 25 ms to 2 seconds.

[0351] Thus, in some examples, the individual loudspeaker dynamics processing configuration data may include a dynamic range compression data set for each loudspeaker of the plurality of loudspeakers. The dynamic range compression data set may include threshold data, input / output ratio data, attack data, release data, and / or knee data. One or more of these types of individual loudspeaker dynamics processing configuration data may be combined to determine the listening environment dynamics processing configuration data. As described above with respect to the combination of playback limit thresholds, in some examples, the dynamic range compression data may be averaged to determine the listening environment dynamics processing configuration data. In some instances, a minimum or maximum value of the dynamic range compression data may be used to determine the listening environment dynamics processing configuration data (e.g., maximum compression ratio). In other implementations, an adjustable combination that interpolates between a minimum and an average of the dynamic range compression data for the individual loudspeaker dynamics processing may be created, e.g., via a tuning parameter as described above with reference to Equation (32).

[0352] In some of the examples described above, a single set of listening environment dynamics processing configuration data (e.g., a combined threshold

number

number

number

[0353] To address such issues, some implementations allow independent or partially independent dynamics processing for different "spatial zones" of a spatial mix. A spatial zone may be thought of as a subset of the spatial region into which the entire spatial mix is ​​rendered. Much of the following discussion provides examples of dynamics processing based on playback limiting thresholds, but these concepts apply equally to other types of individual loudspeaker dynamics processing configuration data and listening environment dynamics processing configuration data.

[0354] An example of the spatial zones of a listening environment is shown in Figure 34. Figure 34 shows an example of an area of ​​a spatial mix (represented by a whole square), which is subdivided into three spatial zones: front, center, and surround.

[0355] Although the spatial zones in FIG. 34 are depicted with hard boundaries, in practice it is beneficial to treat the transition from one spatial zone to another as continuous. For example, a component of a spatial mix located in the center of the left edge of a square may have half its level assigned to the front zone and half assigned to the surround zone. The signal level from each component of the spatial mix may be assigned to each spatial zone and accumulated in this continuous manner. The dynamics processing function may then act on each spatial zone independently on the overall signal level assigned to it from the mix. For each component of the spatial mix, the results of the dynamics processing from each spatial zone (e.g., time-varying gain per frequency) may then be combined and applied to that component. In some instances, this combination of spatial zone results will be different for each component and is a function of the assignment of that particular component to each zone. The end result is that components of a spatial mix with similar spatial zone assignments will receive similar dynamics processing, but independence between spatial zones is allowed. The spatial zones may be advantageously selected to prevent undesirable spatial shifts such as left-right imbalance, while allowing spatially independent processing (e.g., to reduce other artifacts such as spatial ducking as discussed above).

[0356] Techniques for processing the spatial mix by spatial zone may be advantageously used in the first stage of dynamics processing of the present disclosure. For example, a different combination of individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) across speakers i may be calculated for each spatial zone. The set of combined zone thresholds is

number

number

[0357] K individual component signals x, each with an associated desired spatial location (possibly time-varying) k Consider that a spatial signal is to be rendered that consists of the sum of [t]. One specific way to implement zonal processing is to k A time-varying pan gain α that describes how much [t] contributes to zone j kj The objective of the present invention is to calculate the pan gains [t] as a function of the desired spatial position of the audio signal with respect to the position of the zone. These pan gains can be advantageously designed to follow the power-conserving pan law, which requires that the sum of the squared gains is equal to 1. From these pan gains, the zone signal s j [t] may be calculated as the sum of the component signals weighted by their pan gains for that zone:

number

number

number

number

number

[0358] The combination of the individual loudspeaker dynamics processing configuration data (e.g., speaker playback limiting thresholds) for each spatial zone may be implemented in a variety of ways.

number

number

[0359] Similar weighting functions may be applied to other types of individual loudspeaker dynamics processing configuration data. Advantageously, the combined individual loudspeaker dynamics processing configuration data (e.g. playback limiting thresholds) of a spatial zone may be biased towards the individual loudspeaker dynamics processing configuration data (e.g. playback limiting thresholds) of the speaker that contributes most to the playback component of the spatial mix associated with that spatial zone. This is done by weighting each speaker's contribution to rendering the component of the spatial mix associated with that zone for frequency f with a weight w ij This can be achieved by setting [f].

[0360] FIG. 35 shows an example of loudspeakers within the spatial zones of FIG. 34. FIG. 35 shows the same zones of FIG. 34, but overlaid with the positions of five exemplary loudspeakers (speakers 1, 2, 3, 4, 5) that contribute to rendering the spatial mix. In this example, loudspeakers 1, 2, 3, 4, 5 are represented as diamonds. In this particular example, speaker 1 is primarily responsible for rendering the center zone, speakers 2 and 5 the front zones, and speakers 3 and 4 the surround zones. Based on this conceptual one-to-one mapping of speakers to spatial zones, weights w ij [f] can be generated, but as with spatial zone-based processing of a spatial mix, a more continuous mapping may be preferable. For example, speaker 4 is very close to the front zone, and components of the audio mix located between speakers 4 and 5 will likely be reproduced primarily by speakers 4 and 5 combined (albeit in the conceptual front zone). It therefore makes sense for speaker 4's individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) to contribute to the combined individual loudspeaker dynamics processing configuration data (e.g., playback limiting thresholds) of the front zones as well as the surround zones.

[0361] One way to achieve this continuous mapping is to use weights w, which are equal to the speaker participation values ​​that describe the relative contribution of each speaker i in rendering the component associated with spatial zone j. ij The primary objective of the present invention is to set [f]. Such a value may be derived directly from a rendering system responsible for rendering to the loudspeakers (e.g., from step (c) above) and a set of one or more nominal spatial locations associated with each spatial zone. This set of nominal spatial locations may include a set of locations within each spatial zone.

[0362] Figure 36 shows an example of nominal spatial locations superimposed on the spatial zones and speakers of Figure 35. The nominal locations are shown as numbered circles: the front zone is associated with two locations located at the top corners of a square, the centre zone is associated with a single location in the top centre of a square, and the surround zone is associated with two locations located at the bottom corners of a square.

[0363] To calculate a speaker participation value for a spatial zone, each of the nominal positions associated with that zone may be rendered through a renderer to generate speaker activations associated with that position. These activations may, for example, be gains for each speaker in the case of CMAP, or complex values ​​at a given frequency for each speaker in the case of FV. Then, for each speaker and zone, these activations are accumulated over each nominal position associated with the spatial zone to produce a value g ij [f] may be generated, which represents the total activation of speaker i for rendering the entire set of nominal positions associated with spatial zone j. Finally, a speaker participation value in a spatial zone may be calculated as the cumulative activation normalized by the sum of all these accumulated activations across speakers. The weights may then be set to this speaker participation value:

number

[0364] According to some implementations, the above-described process for calculating speaker participation values ​​and combining thresholds as a function of these values ​​may be performed as a static process, where the resulting combined thresholds are calculated once during a setup procedure that determines the layout and capabilities of the speakers in the environment. In such a system, once set up, it may be assumed that both the dynamics processing configuration data of the individual loudspeakers and the manner in which the rendering algorithm activates the loudspeakers as a function of the desired audio signal position remain static. However, in certain systems, both of these aspects may change over time, for example in response to changing conditions in the playback environment, and thus it may be desirable to update the combined thresholds according to the above-described process, either continuously or in an event-triggered manner, to take such variations into account.

[0365] Both the CMAP and FV rendering algorithms may be extended to accommodate one or more dynamically configurable features in response to changes in the listening environment. For example, with reference to FIG. 35, a person located near Speaker 3 may issue a wake word for a smart assistant associated with the speaker, thereby placing the system in a state ready to listen for subsequent commands from the person. While the wake word is being issued, the system may determine the location of the person using a microphone associated with the loudspeaker. With this information, the system may then choose to divert the energy of the audio played from Speaker 3 to another speaker so that the microphone on Speaker 3 can better hear the person. In such a scenario, Speaker 2 in FIG. 35 may essentially "take over" the role of Speaker 3 for a period of time, resulting in a significant change in speaker participation values ​​for the surround zone, with Speaker 3's participation value decreasing and Speaker 2's participation value increasing. The zone thresholds may then be recalculated as they depend on the changed speaker participation values. Alternatively or in addition to these changes to the rendering algorithm, the limiting threshold for Speaker 3 may be lowered below a nominal value set to prevent the speaker from distorting. This can prevent the remaining audio being played from Speaker 3 from increasing beyond some threshold determined to cause interference to the human listening microphone. Zone thresholds may also be updated in this case as they are also a function of the individual speaker thresholds.

[0366] FIG. 37 is a flow diagram outlining an example of a method that may be implemented by a device or system such as those disclosed herein. The blocks of method 3700, as well as other methods described herein, are not necessarily performed in the order shown. In some implementations, one or more blocks of method 3700 may be performed simultaneously. Additionally, some implementations of method 3700 may include more or fewer blocks than shown and / or described. The blocks of method 3700 may be performed by one or more devices that may be (or may include) a control system, such as control system 110 shown in FIG. 1A and described above, or one of the other disclosed examples of control systems.

[0367] According to this example, block 3705 involves obtaining, by the control system, via the interface system, individual loudspeaker dynamics processing configuration data for each of a plurality of loudspeakers of the listening environment. In this implementation, the individual loudspeaker dynamics processing configuration data includes an individual loudspeaker dynamics processing configuration data set for each loudspeaker of the plurality of loudspeakers. According to some examples, the individual loudspeaker dynamics processing configuration data for one or more loudspeakers may correspond to one or more capabilities of the one or more loudspeakers. In this example, each data set of the individual loudspeaker dynamics processing configuration data sets includes at least one type of dynamics processing configuration data.

[0368] In some instances, block 3705 may involve obtaining an individual loudspeaker dynamics processing configuration data set from each of a plurality of loudspeakers in the listening environment. In other instances, block 3705 may involve obtaining the individual loudspeaker dynamics processing configuration data set from a data structure stored in memory. For example, the individual loudspeaker dynamics processing configuration data set may have been previously obtained and stored in the data structure, for example as part of a setup procedure for each loudspeaker.

[0369] According to some examples, the individual loudspeaker dynamics processing configuration data sets may be proprietary. In some such examples, the individual loudspeaker dynamics processing configuration data sets may have been previously estimated based on individual loudspeaker dynamics processing configuration data for speakers having similar characteristics. For example, block 3705 may involve a speaker matching process that determines the most similar speaker from a data structure representing a plurality of speakers and corresponding individual loudspeaker dynamics processing configuration data sets for each of the plurality of speakers. The speaker matching process may be based, for example, on a comparison of the sizes of one or more woofers, tweeters, and / or midrange speakers.

[0370] In this example, block 3710 involves determining, by the control system, listening environment dynamics processing configuration data for the plurality of loudspeakers. According to this implementation, determining the listening environment dynamics processing configuration data is based on an individual loudspeaker dynamics processing configuration data set for each loudspeaker of the plurality of loudspeakers. Determining the listening environment dynamics processing configuration data may involve combining the individual loudspeaker dynamics processing configuration data of the dynamics processing configuration data set, for example, by taking an average of the one or more types of individual loudspeaker dynamics processing configuration data. In some instances, determining the listening environment dynamics processing configuration data may involve determining a minimum or maximum value of the one or more types of individual loudspeaker dynamics processing configuration data. According to some such implementations, determining the listening environment dynamics processing configuration data may involve interpolating between a minimum or maximum value and an average value of the one or more types of individual loudspeaker dynamics processing configuration data.

[0371] In this implementation, block 3715 involves receiving, by the control system, via the interface system, audio data including one or more audio signals and associated spatial data. For example, the spatial data may indicate an intended perceived spatial location corresponding to the audio signals. In this example, the spatial data includes channel data and / or spatial metadata.

[0372] In this example, block 3720 involves performing, by the control system, dynamics processing on the audio data based on the listening environment dynamics processing configuration data to generate processed audio data. The dynamics processing of block 3720 may involve any of the dynamics processing methods of the present disclosure disclosed herein, including but not limited to applying one or more playout limiting thresholds, compression data, etc.

[0373] Here, block 3725 involves rendering, by the control system, the processed audio data to generate a rendered audio signal for playback via a collection of loudspeakers including at least a portion of the plurality of loudspeakers. In some examples, block 3725 may involve applying a CMAP rendering process, an FV rendering process, or a combination of both. In this example, block 3720 is performed before block 3725. However, as described above, block 3720 and / or block 3710 may be based, at least in part, on the rendering process of block 3725. Blocks 3720 and 3725 may involve performing processes such as those described above with reference to the listening environment dynamics processing module and rendering module 3120 of FIG. 31.

[0374] According to this example, block 3730 involves providing the rendered audio signals to a collection of loudspeakers via an interface system. In one example, block 3730 may involve providing the rendered audio signals to the loudspeakers 3125a-3125m by the smart home hub 3105 via its interface system.

[0375] In some examples, the method 3700 may involve performing dynamics processing on the rendered audio signal in accordance with individual loudspeaker dynamics processing configuration data for each loudspeaker of a collection of loudspeakers to which the rendered audio signal is provided. For example, referring again to Figure 31, dynamics processing modules A-M may perform dynamics processing on the rendered audio signal in accordance with individual loudspeaker dynamics processing configuration data for loudspeakers 3125a-3125m.

[0376] In some implementations, the individual loudspeaker dynamics processing configuration data may include a playback limit threshold data set for each loudspeaker of the plurality of loudspeakers. In some such examples, the playback limit threshold data set may include a playback limit threshold for each of a plurality of frequencies.

[0377] Determining the listening environment dynamics processing configuration data may in some instances involve determining a minimum play-back limiting threshold across the multiple loudspeakers. In some examples, determining the listening environment dynamics processing configuration data may involve averaging the play-back limiting thresholds to obtain an averaged play-back limiting threshold across the multiple loudspeakers. In some such examples, determining the listening environment dynamics processing configuration data may involve determining a minimum play-back limiting threshold across the multiple loudspeakers and interpolating between the minimum play-back limiting threshold and the averaged play-back limiting threshold.

[0378] According to some implementations, averaging the playback limit thresholds may involve determining a weighted average of the playback limit thresholds. In some such examples, the weighted average may be based at least in part on characteristics of a rendering process implemented by the control system, such as characteristics of the rendering process of block 3725.

[0379] In some implementations, performing dynamics processing on the audio data may be based on spatial zones, each of which may correspond to a subset of the listening environment.

[0380] According to some such implementations, dynamics processing may be performed separately for each of the spatial zones. For example, determining listening environment dynamics processing configuration data may be performed separately for each of the spatial zones. For example, combining the dynamics processing configuration data set across the multiple loudspeakers may be performed separately for each of the one or more spatial zones. In some examples, combining the dynamics processing configuration data set across the multiple loudspeakers separately for each of the one or more spatial zones may be based, at least in part, on loudspeaker activation by the rendering process as a function of desired audio signal positions across the one or more spatial zones.

[0381] In some examples, combining the dynamics processing configuration data sets across the multiple loudspeakers separately for each of the one or more spatial zones may be based, at least in part, on a loudspeaker participation value for each loudspeaker in each of the one or more spatial zones. Each loudspeaker participation value may be based, at least in part, on one or more nominal spatial positions within each of the one or more spatial zones. The nominal spatial positions may, in some examples, correspond to standard positions of channels within a Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround sound mix. In some such implementations, each loudspeaker participation value is based, at least in part, on activation of each loudspeaker corresponding to rendering of audio data at each of the one or more nominal spatial positions within each of the one or more spatial zones.

[0382] According to some such examples, the weighted average of the playback limiting thresholds may be based, at least in part, on activation of loudspeakers by the rendering process as a function of proximity of the audio signal to the spatial zone. In some instances, the weighted average may be based, at least in part, on loudspeaker participation values ​​for each loudspeaker in each spatial zone. In some such examples, each loudspeaker participation value may be based, at least in part, on one or more nominal spatial locations within each spatial zone. For example, the nominal spatial locations may correspond to standard locations of channels in a Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, or Dolby 9.1 surround sound mix. In some implementations, each loudspeaker participation value may be based, at least in part, on activation of each loudspeaker corresponding to the rendering of audio data at each of one or more nominal spatial locations within each spatial zone.

[0383] According to some implementations, rendering the processed audio data may involve determining relative activation of a collection of loudspeakers according to one or more dynamically configurable features. Some examples are described below with reference to FIG. 10 et seq. The one or more dynamically configurable features may be based on one or more attributes of the audio signal, one or more attributes of the collection of loudspeakers, or one or more external inputs. For example, the one or more dynamically configurable features may be based on the proximity of a loudspeaker to one or more listeners; the proximity of a loudspeaker to an attractive position (attractive force is a factor that favors higher loudspeaker activation relative to closer proximity to an attractive position); the proximity of a loudspeaker to a repulsive position (repulsive force is a factor that favors lower loudspeaker activation relative to closer proximity to a repulsive position); the capabilities of each loudspeaker relative to other loudspeakers in the environment; the synchronicity of the loudspeaker relative to other loudspeakers; wake word performance; or echo canceller performance.

[0384] The relative activation of the speakers may, in some examples, be based on a cost function of a model of the perceived spatial location of the audio signal when played through the speakers, a measure of the proximity of the intended perceived spatial location of the audio signal to the speaker location, and one or more dynamically configurable features.

[0385] In some examples, minimization of the cost function (including at least one dynamic speaker activation term) may lead to deactivation of at least one of the speakers (in the sense that each such speaker does not play the associated audio content) and activation of at least one of the speakers (in the sense that each such speaker plays at least a portion of the rendered audio content). The dynamic speaker activation term(s) may enable at least one of a variety of behaviors, including distorting the spatial presentation of audio away from a particular smart audio device, such that a microphone can better hear a speaker or a secondary audio stream can better be heard from a speaker(s) of the smart audio device.

[0386] According to some implementations, the individual loudspeaker dynamics processing configuration data may include, for each loudspeaker of the plurality of loudspeakers, a dynamic range compression data set. In some instances, the dynamic range compression data set may include one or more of threshold data, input / output ratio data, attack data, release data, or knee data.

[0387] As mentioned above, in some implementations, at least some of the blocks of method 3700 shown in FIG. 37 may be omitted. For example, in some implementations, blocks 3705 and 3710 are performed during a setup process. After the listening environment dynamics processing configuration data has been determined, in some implementations, steps 3705 and 3710 are not performed again during “runtime” operation unless the type and / or placement of speakers in the listening environment changes. For example, in some implementations, there may be an initial check to determine if any loudspeakers have been added or removed, any loudspeaker positions have changed, etc. If so, steps 3705 and 3710 may be performed. If not, steps 3705 and 3710 may not be performed again prior to “runtime” operations, which may involve blocks 3715-3730.

[0388] Figures 38A, 38B and 38C show examples of speaker participation values ​​corresponding to the examples of Figures 2C and 2D. In Figures 38A, 38B and 38C, angle -4.1 corresponds to speaker position 272 in Figure 2D, angle 4.1 corresponds to speaker position 274 in Figure 2D, angle -87 corresponds to speaker position 267 in Figure 2D, angle 63.6 corresponds to speaker position 275 in Figure 2D and angle 165.4 corresponds to speaker position 270 in Figure 2D. These speaker participation values ​​are examples of "weighting" for spatial zones as described with reference to Figures 34-37. According to these examples, the loudspeaker participation values ​​shown in Figures 38A, 38B and 38C correspond to the participation of each loudspeaker in each of the spatial zones shown in Figure 34: the loudspeaker participation values ​​shown in Figure 38A correspond to the participation of each loudspeaker in the center zone, the loudspeaker participation values ​​shown in Figure 38B correspond to the participation of each loudspeaker in the front left and right zones, and the loudspeaker participation values ​​shown in Figure 38C correspond to the participation of each loudspeaker in the rear zone.

[0389] Figures 39A, 39B and 39C show example loudspeaker participation values ​​corresponding to the examples of Figures 2F and 2G. In Figures 39A, 39B and 39C, angle -4.1 corresponds to speaker position 272 in Figure 2D, angle 4.1 corresponds to speaker position 274 in Figure 2D, angle -87 corresponds to speaker position 267 in Figure 2D, angle 63.6 corresponds to speaker position 275 in Figure 2D and angle 165.4 corresponds to speaker position 270 in Figure 2D. According to these examples, the loudspeaker participation values ​​shown in Figures 39A, 39B and 39C correspond to the participation of each loudspeaker in each spatial zone shown in Figure 34: the loudspeaker participation values ​​shown in Figure 39A correspond to the participation of each loudspeaker in the center zone, the loudspeaker participation values ​​shown in Figure 39B correspond to the participation of each loudspeaker in the front left and right zones, and the loudspeaker participation values ​​shown in Figure 39C correspond to the participation of each loudspeaker in the rear zone.

[0390] Figures 40A, 40B and 40C show examples of loudspeaker participation values ​​corresponding to the examples of Figures 2H and 2I. According to these examples, the loudspeaker participation values ​​shown in Figures 40A, 40B and 40C correspond to the participation of each loudspeaker in each spatial zone shown in Figure 34. The loudspeaker participation values ​​shown in Figure 40A correspond to the participation of each loudspeaker in the center zone, the loudspeaker participation values ​​shown in Figure 40B correspond to the participation of each loudspeaker in the front left and right zones and the loudspeaker participation values ​​shown in Figure 40C correspond to the participation of each loudspeaker in the rear zone.

[0391] Figures 41A, 41B and 41C show examples of loudspeaker participation values ​​corresponding to the examples of Figures 2J and 2K. According to these examples, the loudspeaker participation values ​​shown in Figures 41A, 41B and 41C correspond to the participation of each loudspeaker in each spatial zone shown in Figure 34. The loudspeaker participation values ​​shown in Figure 41A correspond to the participation of each loudspeaker in the center zone, the loudspeaker participation values ​​shown in Figure 41B correspond to the participation of each loudspeaker in the front left and right zones and the loudspeaker participation values ​​shown in Figure 41C correspond to the participation of each loudspeaker in the rear zone.

[0392] FIG. 42 is a diagram of an environment, in this example a living space. According to this example, the living space includes a set of smart audio devices (devices 1.1) for audio interaction, speakers (1.3) for audio output, and controllable lights (1.2). In one example, only device 1.1 contains a microphone and therefore knows where the user (1.4) is located who issues the wake word command. Using various methods, information can be obtained from these devices collectively to provide a location estimate (e.g., a fine-grained location estimate) of the user who issues (e.g., speaks) the wake word.

[0393] In such a living space there is a collection of natural action zones where people perform or cross tasks and activities. These action areas (zones) are places where there may be efforts to estimate the user's position (e.g., determining an uncertain position) or the user's context to assist other aspects of the interface.

[0394] In the example in Figure 1, the key action areas are: 1. Kitchen sink and cooking area (upper left area of ​​living space); 2. Refrigerator door (to the right of the sink and cooking area); 3. Dining area (lower left area of ​​living space); 4. Open area of ​​living space (to the right of the sink and cooking and dining areas); 5. TV couch (right of open area); 6. The TV itself; 7. Table; 8. Door area or entrance (top right area of ​​living space).

[0395] It will be clear that there will often be a similar number of lights in similar positions to fit the action area. Some or all of the lights may be individually controllable networked agents. According to some embodiments, the audio is rendered (e.g., by one of devices 1.1 or other devices in the system of FIG. 42) for playback (according to any embodiment of the method of the present invention) by one or more of the speakers (and / or one or more of the speakers of device (1.1)).

[0396] One class of embodiments is a method of rendering audio for playback and / or playing audio by at least one (e.g., all or some) of a plurality of coordinated (orchestrated) smart audio devices. For example, a collection of smart audio devices present in a user's home (in a system) may be orchestrated to handle a variety of simultaneous use cases, including flexible rendering of audio for playback by all or some of the smart audio devices (i.e., by speakers of all or some of the smart audio devices). Many interactions with the system are possible that...

Claims

1. 13. A method of audio processing comprising: receiving, by a control system, first audio data corresponding to a first audio program stream, the first audio data including one or more first audio signals and first spatial data indicative of an associated desired perceived spatial location for each of the one or more first audio signals; rendering, by the control system, the first audio data to generate first rendered audio data for at least two loudspeakers of a collection of loudspeakers in an environment; the rendering includes determining a relative activation of loudspeakers of the collection of loudspeakers based on a perceived spatial location of the first audio signal reproduced through the loudspeakers, a proximity of the desired perceived spatial location of the first audio signal to a location of the loudspeakers, and one or more additional dynamically configurable features that depend on at least one or more attributes of the first audio signal, one or more attributes of the collection of loudspeakers, or one or more external inputs; the additional dynamically configurable features include at least the proximity of one or more loudspeakers to an attractive force location or the proximity of one or more loudspeakers to a repulsive force location, where attractive force is a factor that favors relatively higher loudspeaker activation closer to the attractive force location and repulsive force is a factor that favors relatively lower loudspeaker activation closer to the repulsive force location; method.

2. The method of claim 1 , wherein the additional dynamically configurable features include proximity of one or more loudspeakers to one or more listeners.

3. The method of claim 1 , wherein the additional dynamically configurable features include audibility of one or more loudspeakers at a position in the environment.

4. 2. The method of claim 1 , wherein the additional dynamically configurable features include at least one of the capabilities of one or more loudspeakers or synchronization of the one or more loudspeakers with one or more other loudspeakers in the environment.

5. The method of claim 1 , wherein the additional dynamically configurable functionality includes at least one of a wake word capability or an echo canceller capability.

6. The method of claim 1 , wherein the rendering comprises minimizing a cost function, the cost function including at least one dynamic speaker activation term.

7. 7. The method of claim 6, wherein the cost function is based at least in part on a sum of a term corresponding to spatial audio and a term corresponding to a proximity of a loudspeaker to a desired perceived spatial location for one of the one or more first audio signals.

8. The method of claim 6 , wherein the cost function is based at least in part on Center of Mass Amplitude Panning (CMAP), Flexible Virtualization (FV), or a combination of CMAP and FV.

9. receiving, by a control system, second audio data corresponding to a second audio program stream, the second audio data including one or more second audio signals and second spatial data indicative of an associated desired perceived spatial location for each of the one or more second audio signals; rendering, by the control system, the second audio data to generate second rendered audio data; mixing the first rendered audio signal and the second rendered audio signal to generate a mixed audio signal; providing the mixed audio signal to at least some loudspeakers in the environment. The method of claim 1.

10. 10. The method of claim 9, further comprising modifying a rendering process for the first audio data based at least in part on at least one of the second audio signal, the second rendered audio data, characteristics of the second audio data, or characteristics of the second rendered audio data to generate modified first rendered audio data.

11. 11. The method of claim 10, wherein modifying a rendering process for the first audio signal comprises modifying a rendering of the first audio signal such that a spatial presentation of the first audio signal is warped away from or towards a rendering location of the second rendered audio data.

12. an interface system; Control system and 1. An audio processing system comprising: The control system has a first rendering module, the first rendering module being configured to: receiving, via the interface system, first audio data corresponding to a first audio program stream, the first audio data including one or more first audio signals and first spatial data indicative of an associated desired perceived spatial location for each of the one or more first audio signals; rendering, by the control system, the first audio data to generate first rendered audio data for at least two loudspeakers of a collection of loudspeakers in an environment; the rendering includes determining a relative activation of loudspeakers of the collection of loudspeakers based on a perceived spatial location of the first audio signal reproduced through the loudspeakers, a proximity of the desired perceived spatial location of the first audio signal to a location of the loudspeakers, and one or more additional dynamically configurable features that depend on at least one or more attributes of the first audio signal, one or more attributes of the collection of loudspeakers, or one or more external inputs; the additional dynamically configurable features include at least the proximity of one or more loudspeakers to an attractive force location or the proximity of one or more loudspeakers to a repulsive force location, where attractive force is a factor that favors relatively higher loudspeaker activation closer to the attractive force location and repulsive force is a factor that favors relatively lower loudspeaker activation closer to the repulsive force location; Audio processing system.

13. 13. The audio processing system of claim 12, wherein the additional dynamically configurable features include proximity of one or more loudspeakers to one or more listeners.

14. The audio processing system of claim 12 , wherein the additional dynamically configurable features include audibility of one or more loudspeakers at a position in the environment.

15. 13. The audio processing system of claim 12, wherein the additional dynamically configurable features include at least one of the capabilities of one or more loudspeakers or synchronization of the one or more loudspeakers with one or more other loudspeakers in the environment.

16. 13. The audio processing system of claim 12, wherein the additional dynamically configurable functionality includes at least one of a wake word capability or an echo canceller capability.

17. 13. The audio processing system of claim 12, wherein the rendering comprises minimizing a cost function, the cost function including at least one dynamic speaker activation term.

18. 20. The audio processing system of claim 17, wherein the cost function is based at least in part on a sum of a term corresponding to spatial audio and a term corresponding to a proximity of a loudspeaker to a desired perceived spatial location for one of the one or more first audio signals.

19. 20. The audio processing system of claim 17, wherein the cost function is based at least in part on Center of Mass Amplitude Panning (CMAP), Flexible Virtualization (FV), or a combination of CMAP and FV.

20. receiving, via the interface system, second audio data corresponding to a second audio program stream, the second audio data including one or more second audio signals and second spatial data indicative of an associated desired perceived spatial position for each of the one or more second audio signals; rendering the second audio data to generate second rendered audio data; a second rendering module configured to execute: a mixing module configured to mix the first rendered audio signal and the second rendered audio signal to generate a mixed audio signal. and the control system is further configured to provide the mixed audio signal to at least some loudspeakers in the environment.

13. An audio processing system according to claim 12.

21. A non-transitory computer readable storage medium storing computer executable instructions for executing the method of any one of claims 1 to 11 on one or more processors.

22. A computer program product comprising instructions which, when executed by a processor, cause the processor to perform a method according to any one of claims 1 to 11.

Images