Audio equipment coordination

Abstract

A method for audio session management for an audio environment with multiple audio devices may involve receiving a first route initiation request for initiating a first route for a first audio session from a first device implementing a first application and by a device implementing an audio session manager. The first route initiation request may indicate a first audio source and a first audio environment destination. The first audio environment destination may correspond to at least a first person in the audio environment, but in some instances, the first audio environment destination will not indicate an audio device. The method may involve establishing a first route corresponding to the first route initiation request. Establishing the first route may involve determining a first location of at least a first person in the audio environment, determining at least one audio device for a first stage of the first audio session, and initiating or scheduling the first audio session.

Description

[0001] Inventors: G. N. Dickins, R. J. Cartwright, D. Gunawan, C. G. Hines, M. R. P. Thomas, A. J. Seyfed, J. B. Landau, C. E. M. Dionisio, D. Arteaga

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS

[0003] This application claims the benefit of U.S. Provisional Patent Application No. 62 / 992,068 filed on March 19, 2020, U.S. Provisional Patent Application No. 62 / 949,998 filed on December 18, 2019, European Patent Application No. 19217580.0 filed on December 18, 2019, Spanish Patent Application No. P201930702 filed on July 30, 2019, U.S. Provisional Patent Application No. 62 / 971,421 filed on February 7, 2020, U.S. Provisional Patent Application No. 62 / 705,410 filed on June 25, 2020, U.S. Provisional Patent Application No. 62 / 880,114 filed on July 30, 2019, U.S. Provisional Patent Application No. 62 / 705,314 filed on June 23, 2020 51. U.S. Provisional Patent Application No. 62 / 880,115 filed on July 30, 2019, U.S. Provisional Patent Application No. 62 / 705,143 filed on June 12, 2020, U.S. Provisional Patent Application No. 62 / 880,118 filed on July 30, 2019, U.S. Provisional Patent Application No. 16 / 929,215 filed on July 15, 2020, U.S. Provisional Patent Application No. 62 / 705,883 filed on July 20, 2020, U.S. Provisional Patent Application No. 62 / 880,121 filed on July 30, 2019, and U.S. Provisional Patent Application No. 62 / 705,884 filed on July 20, 2020, each of which is incorporated herein by reference in its entirety. Technical Field

[0004] The present disclosure relates to systems and methods for coordinating (orchestrating) and implementing audio devices, which may include smart audio devices. Background Art

[0005] Audio devices, including but not limited to smart audio devices, have been widely deployed and are becoming a common feature of many homes. Although existing systems and methods for controlling audio devices provide benefits, improved systems and methods would still be desirable.

[0006] Symbols and terminology

[0007] Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to refer to any sound-emitting transducer (or group of transducers) driven by a single speaker feed. A typical headphone includes two speakers. A speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, one or more speaker feeds can undergo different processing in different circuit branches coupled to different transducers.

[0008] Throughout this disclosure, including in the claims, expressions referring to “performing an operation on” a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data or performing the operation on a processed version of the signal or data (e.g., a version of the signal that has been initially filtered or preprocessed before the operation is performed on it).

[0009] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system that includes such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M of the inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0010] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio, video, or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chipsets.

[0011] Throughout this disclosure, including in the claims, the terms "couple" or "couples" are used to refer to either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections.

[0012] As used herein, a "smart device" is an electronic device that can operate interactively and / or autonomously to some extent, and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near-field communication, Wi-Fi, Light Fidelity (Li-Fi), 3G, 4G, 5G, etc. Various notable types of smart devices are smart phones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablets, smart watches, smart bracelets, smart key chains, and smart audio devices. The term "smart device" may also refer to a device that exhibits certain properties of ubiquitous computing, such as artificial intelligence.

[0013] The expression "smart audio device" is used herein to refer to a smart device, which is a single-purpose audio device or a multi-purpose audio device (for example, an audio device that implements at least some aspects of the functionality of a virtual assistant). A single-purpose audio device is a device (for example, a television (TV) or a mobile phone) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to implement a single purpose. For example, while a TV can typically play (and is considered capable of playing) audio from program material, in most cases, modern TVs run some kind of operating system on which applications (including applications for watching TV) run locally. Similarly, the audio input and output in a mobile phone can do a lot of things, but these are all served by applications running on the phone. In this sense, a single-purpose audio device with one or more speakers and one or more microphones is typically configured to run local applications and / or services to directly use the one or more speakers and one or more microphones. Some single-purpose audio devices can be configured to be combined to enable audio to be played in a certain area or user-configured area.

[0014] A common type of multi-purpose audio device is an audio device that implements at least some aspects of a virtual assistant's functionality, although other aspects of the virtual assistant's functionality may be implemented by one or more other devices (e.g., one or more servers with which the multi-purpose audio device is configured to communicate). Such a multi-purpose audio device may be referred to herein as a "virtual assistant." A virtual assistant is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (other than the virtual assistant) for applications that are cloud-enabled in some sense or that are otherwise not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant's functionality (e.g., speech recognition functionality) may be implemented (at least in part) by one or more servers or other devices with which the virtual assistant can communicate over a network (e.g., the Internet). Virtual assistants may sometimes work together, for example, in a discrete and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them (e.g., the virtual assistant that is most confident of having heard the wake word) responds to the wake word. In some implementations, connected virtual assistants may form a constellation, which may be managed by a master application that may be (or implement) a virtual assistant.

[0015] As used herein, a "wake-up word" is used broadly to refer to any sound (e.g., a human-spoken word or other sound) that a smart audio device is configured to wake up in response to detecting ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some instances, what may be referred to herein as a "wake-up word" may include more than one word, for example, a phrase.

[0016] As used herein, the expression "wake-up word detector" refers to a device (or software comprising instructions for configuring the device to continuously search for alignments between real-time sound (e.g., speech) features and a trained model). Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability that a wake-up word has been detected exceeds a predefined threshold. For example, the threshold may be a predetermined threshold that is adjusted to give a reasonable compromise between a false acceptance rate and a false rejection rate. Following a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or an "attention" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer. Summary of the Invention

[0017] In one class of embodiments, a continuous layered audio session manager (CHASM) is used to coordinate audio devices (which may include smart audio devices). In some disclosed embodiments, at least some aspects of CHASM can be implemented by so-called "smart home hubs (hub)" herein. According to some examples, CHASM can be implemented by the specific equipment of the audio environment. In some instances, CHASM can be implemented at least in part via software that can be executed by one or more devices of the audio environment. In some embodiments, a device (e.g., a smart audio device) is included in a network-connectable element or subsystem (e.g., a media engine and device property descriptor) sometimes referred to as a discoverable opportunistic orchestration distributed audio subsystem (DOODAD) in this article, and multiple (e.g., a large number of) devices (e.g., smart audio devices or other devices including DOODADS) are collectively managed by CHASM, or managed in another way to implement the orchestration function (e.g., replacing the function known or intended for the device when first purchased). Development architecture and a control language for the audio function of an audio system that supports CHASM are described herein. This document also describes the orchestration language and explains the fundamentals and differences in working with collective audio systems without direct reference to the devices (or routes) of the audio. It also describes persistent sessions, destinations, priorities, and audio routing, as well as requesting confirmations, which are specific to the idea of ​​orchestration and routing audio between people and places.

[0018] Aspects of the present disclosure include a system configured (e.g., programmed) to perform any embodiment of the disclosed method or its steps, and a tangible, non-transitory computer-readable medium (e.g., a disc or other tangible storage medium) that implements non-transitory storage of data, the tangible, non-transitory computer-readable medium storing code for performing any embodiment of the disclosed method or its steps (e.g., code executable to perform any embodiment of the disclosed method or its steps). For example, some embodiments may be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor that is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data according to one or more of the disclosed methods or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform multiple of the disclosed methods (or its steps) in response to data asserted thereto.

[0019] At least some aspects of the present disclosure may be implemented via methods. In some instances, the methods may be implemented at least in part by a control system such as those disclosed herein. Some such methods may relate to audio session management for an audio system of an audio environment.

[0020] Some such methods involve establishing a first smart audio device communication link between an audio session manager and at least a first smart audio device of an audio system. In some examples, the first smart audio device is or includes a single-purpose audio device or a multi-purpose audio device. In some such examples, the first smart audio device includes one or more loudspeakers. Some such methods involve establishing a first application communication link between the audio session manager and a first application device executing a first application.

[0021] Some such methods involve determining, by an audio session manager, one or more first media engine capabilities of a first media engine of a first smart audio device. In some examples, the first media engine is configured to manage one or more audio media streams received by the first smart audio device and to perform first smart audio device signal processing on the one or more audio media streams according to a first media engine sampling clock.

[0022] In some such examples, a method involves receiving, by an audio session manager, a first application control signal from a first application via a first application communication link. Some such methods involve controlling a first smart audio device based on first media engine capabilities. According to some implementations, controlling is accomplished by the audio session manager via a first audio session management control signal transmitted to the first smart audio device via the first smart audio device communication link. In some such examples, the audio session manager transmits the first audio session management control signal to the first smart audio device without reference to a first media engine sampling clock.

[0023] In some implementations, the first application communication link can be established in response to a first routing initiation request from the first application device. According to some examples, the first application control signal can be transmitted from the first application without reference to the first media engine sampling clock. In some examples, the first audio session management control signal can cause the first smart audio device to delegate control of the first media engine to the audio session manager.

[0024] According to some examples, a device other than the audio session manager or the first smart audio device can be configured to execute the first application. However, in some instances, the first smart audio device can be configured to execute the first application.

[0025] In some examples, the first smart audio device can include a specific-purpose audio session manager. According to some such examples, the audio session manager can communicate with the specific-purpose audio session manager via the first smart audio device communication link. According to some such examples, the audio session manager can obtain one or more first media engine capabilities from the specific-purpose audio session manager.

[0026] According to some embodiments, the audio session manager can act as a gateway for all applications that control the first media engine, whether the application is running on the first smart audio device or on another device.

[0027] Some such methods may also involve establishing at least a first audio stream corresponding to a first audio source. The first audio stream may include a first audio signal. In some such examples, establishing at least the first audio stream may involve causing the first smart audio device to establish the at least the first audio stream by transmitting a first audio session management control signal to the first smart audio device via a first smart audio device communication link.

[0028] In some examples, such a method may also involve a rendering process that causes the first audio signal to be rendered as a first rendered audio signal.In some examples, the rendering process may be performed by the first smart audio device in response to the first audio session management control signal.

[0029] Some such methods may also involve causing the first smart audio device to establish an inter-smart audio device communication link between the first smart audio device and each of one or more other smart audio devices in the audio environment via the first audio session management control signal. Some such methods may also involve causing the first smart audio device to transmit the original microphone signal, the processed microphone signal, the rendered audio signal, and / or the unrendered audio signal to the one or more other smart audio devices via the inter-smart audio device communication link or multiple inter-smart audio device communication links.

[0030] In some examples, such methods may also involve establishing a second smart audio device communication link between the audio session manager and at least a second smart audio device of the home audio system. The second smart audio device may be or may include a single-purpose audio device or a multi-purpose audio device. The second smart audio device may include one or more microphones. Some such methods may also involve determining, by the audio session manager, one or more second media engine capabilities of a second media engine of the second smart audio device. The second media engine may be configured to receive microphone data from the one or more microphones and to perform second smart audio device signal processing on the microphone data. Some such methods may also involve controlling, by the audio session manager, the second smart audio device based on the second media engine capabilities by transmitting a second audio session manager control signal to the second smart audio device via the second smart audio device communication link.

[0031] According to some such examples, controlling the second smart audio device may further involve causing the second smart audio device to establish an inter-smart audio device communication link between the second smart audio device and the first smart audio device. In some examples, controlling the second smart audio device may involve causing the second smart audio device to transmit processed and / or unprocessed microphone data from the second media engine to the first media engine via the inter-smart audio device communication link.

[0032] In some examples, controlling the second smart audio device can involve receiving, by the audio session manager and via the first application communication link, a first application control signal from the first application, and determining a second audio session manager control signal based on the first application control signal.

[0033] Alternatively or additionally, some audio session management methods involve receiving, from a first device implementing a first application and by a device implementing an audio session manager, a first route initiation request for initiating a first route for a first audio session. In some examples, the first route initiation request indicates a first audio source and a first audio environment destination, and the first audio environment destination corresponds to at least a first person in the audio environment, but the first audio environment destination does not indicate an audio device.

[0034] Some such methods involve establishing, by a device implementing an audio session manager, a first route corresponding to a first route initiation request. According to some examples, establishing the first route involves determining a first location of at least a first person in an audio environment, determining at least one audio device for a first phase of a first audio session, and initiating or scheduling the first audio session.

[0035] According to some examples, the first routing initiation request may include a first audio session priority. In some instances, the first routing initiation request may include a first connection mode. For example, the first connection mode may be a synchronous connection mode, a transactional connection mode, or a scheduled connection mode.

[0036] In some embodiments, the first routing initiation request may include an indication of whether confirmation from at least the first person is required. In some instances, the first routing initiation request may include a first audio session objective. For example, the first audio session objective may include intelligibility, audio quality, spatial fidelity, audibility, inaudibility, and / or privacy.

[0037] Some such methods may involve determining a first persistently unique audio session identifier for the first route.Such methods may involve transmitting the first persistently unique audio session identifier to the first device.

[0038] According to some examples, establishing the first route may involve causing at least one device in the environment to establish at least a first media stream corresponding to the first route, the first media stream including a first audio signal. Some such methods may involve causing the first audio signal to be rendered as a first rendered audio signal.

[0039] Some such methods may involve determining a first orientation of a first person for a first phase of an audio conversation. According to some such examples, causing a first audio signal to be rendered as a first rendered audio signal may involve determining a first reference spatial pattern corresponding to a first position and a first orientation of the first person, and determining a first relative activation of loudspeakers in an audio environment corresponding to the first reference spatial pattern.

[0040] Some such methods may involve determining a second position and / or a second orientation of the first person for a second phase of the first audio conversation. Some such methods may involve determining a second reference spatial pattern corresponding to the second position and / or second orientation, and determining a second relative activation of loudspeakers in the audio environment corresponding to the second reference spatial pattern.

[0041] According to some examples, a method may involve receiving, from a second device implementing a second application and a device implementing an audio session manager, a second route initiation request for initiating a second route for a second audio session. The second route initiation request may indicate a second audio source and a second audio environment destination. The second audio environment destination may correspond to at least a second person in the audio environment. In some examples, the second audio environment destination does not indicate an audio device.

[0042] Some such methods may involve, by a device implementing an audio session manager, establishing a second route corresponding to a second route initiation request. In some implementations, establishing the second route may involve determining a first location of at least a second person in an audio environment, determining at least one audio device for a first phase of a second audio session, and initiating the second audio session. In some examples, establishing the second route may involve establishing at least a second media stream corresponding to the second route, the second media stream including a second audio signal. Some such methods may involve causing the second audio signal to be rendered as a second rendered audio signal.

[0043] Some such methods may involve modifying a rendering process for a first audio signal based at least in part on at least one of a second audio signal, a second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. According to some examples, modifying the rendering process for the first audio signal may involve warping the rendering of the first audio signal away from a rendering position of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may involve modifying the loudness of one or more of the first rendered audio signal in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

[0044] In some examples, the first routing initiation request may indicate at least a first area of ​​the audio environment as the first routing source or the first routing destination. In some implementations, the first routing initiation request may indicate at least a first service (e.g., an online content providing service, such as a service providing music or a service providing podcasts) as the first audio source.

[0045] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as the memory devices described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon.

[0046] For example, the software may include instructions for controlling one or more devices to perform one or more methods related to audio session management for an audio system of an audio environment. Some such methods involve establishing a first smart audio device communication link between an audio session manager and at least a first smart audio device of the audio system. In some examples, the first smart audio device is or includes a single-purpose audio device or a multi-purpose audio device. In some such examples, the first smart audio device includes one or more loudspeakers. Some such methods involve establishing a first application communication link between the audio session manager and a first application device executing a first application.

[0047] Some such methods involve determining, by an audio session manager, one or more first media engine capabilities of a first media engine of a first smart audio device. In some examples, the first media engine is configured to manage one or more audio media streams received by the first smart audio device and to perform first smart audio device signal processing on the one or more audio media streams according to a first media engine sampling clock.

[0048] In some such examples, a method involves receiving, by an audio session manager, a first application control signal from a first application via a first application communication link. Some such methods involve controlling a first smart audio device based on first media engine capabilities. According to some implementations, controlling is accomplished by the audio session manager via a first audio session management control signal transmitted to the first smart audio device via the first smart audio device communication link. In some such examples, the audio session manager transmits the first audio session management control signal to the first smart audio device without reference to a first media engine sampling clock.

[0049] In some implementations, the first application communication link can be established in response to a first routing initiation request from the first application device. According to some examples, the first application control signal can be transmitted from the first application without reference to the first media engine sampling clock. In some examples, the first audio session management control signal can cause the first smart audio device to delegate control of the first media engine to the audio session manager.

[0050] According to some examples, a device other than the audio session manager or the first smart audio device can be configured to execute the first application. However, in some instances, the first smart audio device can be configured to execute the first application.

[0051] In some examples, the first smart audio device can include a specific-purpose audio session manager. According to some such examples, the audio session manager can communicate with the specific-purpose audio session manager via the first smart audio device communication link. According to some such examples, the audio session manager can obtain one or more first media engine capabilities from the specific-purpose audio session manager.

[0052] According to some embodiments, the audio session manager can act as a gateway for all applications that control the first media engine, whether the application is running on the first smart audio device or on another device.

[0053] Some such methods may also involve establishing at least a first audio stream corresponding to a first audio source. The first audio stream may include a first audio signal. In some such examples, establishing at least the first audio stream may involve causing the first smart audio device to establish the at least the first audio stream by transmitting a first audio session management control signal to the first smart audio device via a first smart audio device communication link.

[0054] In some examples, such a method may also involve a rendering process that causes the first audio signal to be rendered as a first rendered audio signal.In some examples, the rendering process may be performed by the first smart audio device in response to the first audio session management control signal.

[0055] Some such methods may also involve causing the first smart audio device to establish an inter-smart audio device communication link between the first smart audio device and each of one or more other smart audio devices in the audio environment via the first audio session management control signal. Some such methods may also involve causing the first smart audio device to transmit the original microphone signal, the processed microphone signal, the rendered audio signal, and / or the unrendered audio signal to the one or more other smart audio devices via the inter-smart audio device communication link or multiple inter-smart audio device communication links.

[0056] In some examples, such methods may also involve establishing a second smart audio device communication link between the audio session manager and at least a second smart audio device of the home audio system. The second smart audio device may be or may include a single-purpose audio device or a multi-purpose audio device. The second smart audio device may include one or more microphones. Some such methods may also involve determining, by the audio session manager, one or more second media engine capabilities of a second media engine of the second smart audio device. The second media engine may be configured to receive microphone data from the one or more microphones and to perform second smart audio device signal processing on the microphone data. Some such methods may also involve controlling, by the audio session manager, the second smart audio device based on the second media engine capabilities by transmitting a second audio session manager control signal to the second smart audio device via the second smart audio device communication link.

[0057] According to some such examples, controlling the second smart audio device may further involve causing the second smart audio device to establish an inter-smart audio device communication link between the second smart audio device and the first smart audio device. In some examples, controlling the second smart audio device may involve causing the second smart audio device to transmit processed and / or unprocessed microphone data from the second media engine to the first media engine via the inter-smart audio device communication link.

[0058] In some examples, controlling the second smart audio device can involve receiving, by the audio session manager and via the first application communication link, a first application control signal from the first application, and determining a second audio session manager control signal based on the first application control signal.

[0059] Alternatively or additionally, the software may include instructions for controlling one or more devices to perform one or more other methods involving audio session management for an audio system of an audio environment. Some such audio session management methods involve receiving, from a first device implementing a first application and a device implementing an audio session manager, a first route initiation request to initiate a first route for a first audio session. In some examples, the first route initiation request indicates a first audio source and a first audio environment destination, and the first audio environment destination corresponds to at least a first person in the audio environment, but the first audio environment destination does not indicate an audio device.

[0060] Some such methods involve establishing, by a device implementing an audio session manager, a first route corresponding to a first route initiation request. According to some examples, establishing the first route involves determining a first location of at least a first person in an audio environment, determining at least one audio device for a first phase of a first audio session, and initiating or scheduling the first audio session.

[0061] According to some examples, the first routing initiation request may include a first audio session priority. In some instances, the first routing initiation request may include a first connection mode. For example, the first connection mode may be a synchronous connection mode, a transactional connection mode, or a scheduled connection mode.

[0062] In some embodiments, the first routing initiation request may include an indication of whether confirmation from at least the first person is required. In some instances, the first routing initiation request may include a first audio session objective. For example, the first audio session objective may include intelligibility, audio quality, spatial fidelity, audibility, inaudibility, and / or privacy.

[0063] Some such methods may involve determining a first persistently unique audio session identifier for the first route.Such methods may involve transmitting the first persistently unique audio session identifier to the first device.

[0064] According to some examples, establishing the first route may involve causing at least one device in the environment to establish at least a first media stream corresponding to the first route, the first media stream including a first audio signal. Some such methods may involve causing the first audio signal to be rendered as a first rendered audio signal.

[0065] Some such methods may involve determining a first orientation of a first person for a first phase of an audio conversation. According to some such examples, causing a first audio signal to be rendered as a first rendered audio signal may involve determining a first reference spatial pattern corresponding to a first position and a first orientation of the first person, and determining a first relative activation of loudspeakers in an audio environment corresponding to the first reference spatial pattern.

[0066] Some such methods may involve determining a second position and / or a second orientation of the first person for a second phase of the first audio conversation. Some such methods may involve determining a second reference spatial pattern corresponding to the second position and / or second orientation, and determining a second relative activation of loudspeakers in the audio environment corresponding to the second reference spatial pattern.

[0067] According to some examples, a method may involve receiving, from a second device implementing a second application and a device implementing an audio session manager, a second route initiation request for initiating a second route for a second audio session. The second route initiation request may indicate a second audio source and a second audio environment destination. The second audio environment destination may correspond to at least a second person in the audio environment. In some examples, the second audio environment destination does not indicate an audio device.

[0068] Some such methods may involve, by a device implementing an audio session manager, establishing a second route corresponding to a second route initiation request. In some implementations, establishing the second route may involve determining a first location of at least a second person in an audio environment, determining at least one audio device for a first phase of a second audio session, and initiating the second audio session. In some examples, establishing the second route may involve establishing at least a second media stream corresponding to the second route, the second media stream including a second audio signal. Some such methods may involve causing the second audio signal to be rendered as a second rendered audio signal.

[0069] Some such methods may involve modifying a rendering process for a first audio signal based at least in part on at least one of a second audio signal, a second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. According to some examples, modifying the rendering process for the first audio signal may involve shifting the rendering of the first audio signal away from a rendering position of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may involve modifying the loudness of one or more of the first rendered audio signal in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

[0070] In some examples, the first routing initiation request may indicate at least a first area of ​​the audio environment as the first routing source or the first routing destination. In some implementations, the first routing initiation request may indicate at least a first service (e.g., an online content providing service, such as a service providing music or a service providing podcasts) as the first audio source.

[0071] In some embodiments, the device (or system) may include an interface system and a control system. The control system may include one or more general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or a combination thereof.

[0072] In some embodiments, a control system can be configured to implement one or more of the methods disclosed herein. Some such methods can involve audio session management for an audio system of an audio environment. According to some such examples, a control system can be configured to implement what may be referred to herein as an audio session manager.

[0073] Some such methods involve establishing a first smart audio device communication link between an audio session manager (e.g., a device implementing the audio session manager) and at least a first smart audio device of an audio system. In some examples, the first smart audio device is or includes a single-purpose audio device or a multi-purpose audio device. In some such examples, the first smart audio device includes one or more loudspeakers. Some such methods involve establishing a first application communication link between the audio session manager and a first application device executing a first application.

[0074] Some such methods involve determining, by an audio session manager, one or more first media engine capabilities of a first media engine of a first smart audio device. In some examples, the first media engine is configured to manage one or more audio media streams received by the first smart audio device and to perform first smart audio device signal processing on the one or more audio media streams according to a first media engine sampling clock.

[0075] In some such examples, a method involves receiving, by an audio session manager, a first application control signal from a first application via a first application communication link. Some such methods involve controlling a first smart audio device based on first media engine capabilities. According to some implementations, controlling is accomplished by the audio session manager via a first audio session management control signal transmitted to the first smart audio device via the first smart audio device communication link. In some such examples, the audio session manager transmits the first audio session management control signal to the first smart audio device without reference to a first media engine sampling clock.

[0076] In some implementations, the first application communication link can be established in response to a first routing initiation request from the first application device. According to some examples, the first application control signal can be transmitted from the first application without reference to the first media engine sampling clock. In some examples, the first audio session management control signal can cause the first smart audio device to delegate control of the first media engine to the audio session manager.

[0077] According to some examples, a device other than the audio session manager or the first smart audio device can be configured to execute the first application. However, in some instances, the first smart audio device can be configured to execute the first application.

[0078] In some examples, the first smart audio device can include a specific-purpose audio session manager. According to some such examples, the audio session manager can communicate with the specific-purpose audio session manager via the first smart audio device communication link. According to some such examples, the audio session manager can obtain one or more first media engine capabilities from the specific-purpose audio session manager.

[0079] According to some embodiments, the audio session manager can act as a gateway for all applications that control the first media engine, whether the application is running on the first smart audio device or on another device.

[0080] Some such methods may also involve establishing at least a first audio stream corresponding to a first audio source. The first audio stream may include a first audio signal. In some such examples, establishing at least the first audio stream may involve causing the first smart audio device to establish the at least the first audio stream by transmitting a first audio session management control signal to the first smart audio device via a first smart audio device communication link.

[0081] In some examples, such a method may also involve a rendering process that causes the first audio signal to be rendered as a first rendered audio signal.In some examples, the rendering process may be performed by the first smart audio device in response to the first audio session management control signal.

[0082] Some such methods may also involve causing the first smart audio device to establish an inter-smart audio device communication link between the first smart audio device and each of one or more other smart audio devices in the audio environment via the first audio session management control signal. Some such methods may also involve causing the first smart audio device to transmit the original microphone signal, the processed microphone signal, the rendered audio signal, and / or the unrendered audio signal to the one or more other smart audio devices via the inter-smart audio device communication link or multiple inter-smart audio device communication links.

[0083] In some examples, such methods may also involve establishing a second smart audio device communication link between the audio session manager and at least a second smart audio device of the home audio system. The second smart audio device may be or may include a single-purpose audio device or a multi-purpose audio device. The second smart audio device may include one or more microphones. Some such methods may also involve determining, by the audio session manager, one or more second media engine capabilities of a second media engine of the second smart audio device. The second media engine may be configured to receive microphone data from the one or more microphones and to perform second smart audio device signal processing on the microphone data. Some such methods may also involve controlling, by the audio session manager, the second smart audio device based on the second media engine capabilities by transmitting a second audio session manager control signal to the second smart audio device via the second smart audio device communication link.

[0084] According to some such examples, controlling the second smart audio device may further involve causing the second smart audio device to establish an inter-smart audio device communication link between the second smart audio device and the first smart audio device. In some examples, controlling the second smart audio device may involve causing the second smart audio device to transmit processed and / or unprocessed microphone data from the second media engine to the first media engine via the inter-smart audio device communication link.

[0085] In some examples, controlling the second smart audio device can involve receiving, by the audio session manager and via the first application communication link, a first application control signal from the first application, and determining a second audio session manager control signal based on the first application control signal.

[0086] Alternatively or additionally, the control system can be configured to implement one or more other audio session management methods. Some such audio session management methods involve receiving, from a first device implementing a first application and by a device implementing an audio session manager, a first route initiation request to initiate a first route for a first audio session. In some examples, the first route initiation request indicates a first audio source and a first audio environment destination, and the first audio environment destination corresponds to at least a first person in the audio environment, but the first audio environment destination does not indicate an audio device.

[0087] Some such methods involve establishing, by a device implementing an audio session manager, a first route corresponding to a first route initiation request. According to some examples, establishing the first route involves determining a first location of at least a first person in an audio environment, determining at least one audio device for a first phase of a first audio session, and initiating or scheduling the first audio session.

[0088] According to some examples, the first routing initiation request may include a first audio session priority. In some instances, the first routing initiation request may include a first connection mode. For example, the first connection mode may be a synchronous connection mode, a transactional connection mode, or a scheduled connection mode.

[0089] In some embodiments, the first routing initiation request may include an indication of whether confirmation from at least the first person is required. In some instances, the first routing initiation request may include a first audio session objective. For example, the first audio session objective may include intelligibility, audio quality, spatial fidelity, audibility, inaudibility, and / or privacy.

[0090] Some such methods may involve determining a first persistently unique audio session identifier for the first route.Such methods may involve transmitting the first persistently unique audio session identifier to the first device.

[0091] According to some examples, establishing the first route may involve causing at least one device in the environment to establish at least a first media stream corresponding to the first route, the first media stream including a first audio signal. Some such methods may involve causing the first audio signal to be rendered as a first rendered audio signal.

[0092] Some such methods may involve determining a first orientation of a first person for a first phase of an audio conversation. According to some such examples, causing a first audio signal to be rendered as a first rendered audio signal may involve determining a first reference spatial pattern corresponding to a first position and a first orientation of the first person, and determining a first relative activation of loudspeakers in an audio environment corresponding to the first reference spatial pattern.

[0093] Some such methods may involve determining a second position and / or a second orientation of the first person for a second phase of the first audio conversation. Some such methods may involve determining a second reference spatial pattern corresponding to the second position and / or second orientation, and determining a second relative activation of loudspeakers in the audio environment corresponding to the second reference spatial pattern.

[0094] According to some examples, a method may involve receiving, from a second device implementing a second application and a device implementing an audio session manager, a second route initiation request for initiating a second route for a second audio session. The second route initiation request may indicate a second audio source and a second audio environment destination. The second audio environment destination may correspond to at least a second person in the audio environment. In some examples, the second audio environment destination does not indicate an audio device.

[0095] Some such methods may involve, by a device implementing an audio session manager, establishing a second route corresponding to a second route initiation request. In some implementations, establishing the second route may involve determining a first location of at least a second person in an audio environment, determining at least one audio device for a first phase of a second audio session, and initiating the second audio session. In some examples, establishing the second route may involve establishing at least a second media stream corresponding to the second route, the second media stream including a second audio signal. Some such methods may involve causing the second audio signal to be rendered as a second rendered audio signal.

[0096] Some such methods may involve modifying a rendering process for a first audio signal based at least in part on at least one of a second audio signal, a second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. According to some examples, modifying the rendering process for the first audio signal may involve shifting the rendering of the first audio signal away from a rendering position of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may involve modifying the loudness of one or more of the first rendered audio signal in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

[0097] In some examples, the first routing initiation request may indicate at least a first area of ​​the audio environment as the first routing source or the first routing destination. In some implementations, the first routing initiation request may indicate at least a first service (e.g., an online content providing service, such as a service providing music or a service providing podcasts) as the first audio source.

[0098] The details of one or more embodiments of the subject matter described in this specification are set forth in the following drawings and description. Other features, aspects, and advantages will become apparent from the description, drawings, and claims. Note that the relative dimensions of the following figures may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] Figure 1A An example of a person and a smart audio device in an audio environment is shown.

[0100] Figure 1B yes Figure 1A A modified version of the scene in FIG.

[0101] Figure 1C According to an embodiment of the present disclosure.

[0102] Figure 2A It is a block diagram of a traditional system.

[0103] Figure 2B Shown Figure 2AAn example of a modified version of the device shown in .

[0104] Figure 2C is a block diagram of an example of a disclosed embodiment.

[0105] Figure 2D An example of multiple applications interacting with a Continuous Hierarchical Audio Session Manager (CHASM) is shown.

[0106] Figure 3A is shown according to an example Figure 1A A block diagram of the details of device 101.

[0107] Figure 3B Shown according to an example Figure 1B Details of the implementation method.

[0108] Figure 3C is a block diagram illustrating an example of CHASM orchestrating two audio devices of an audio environment.

[0109] Figure 4 is a block diagram illustrating another disclosed embodiment.

[0110] Figure 5 is a flow chart including blocks of an audio session management method according to some implementations.

[0111] Figure 6 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0112] Figure 7 is a block diagram depicting blocks of a CHASM according to one example.

[0113] Figure 8 Shown according to an example Figure 7 The details of the routing table are shown in .

[0114] Figure 9A An example of a context-free grammar representing a route-initiated request in an orchestration language.

[0115] Figure 9B Provides an example of an audio session target.

[0116] Figure 10 The flow of a request to modify a route according to an example is shown.

[0117] Figure 11A and Figure 11B Additional examples of the flow of requests for modifying routes are shown.

[0118] Figure 11C An example of a process for deleting a route is shown.

[0119] Figure 12 is a flow chart including blocks of an audio session management method according to some implementations.

[0120] Figure 13 is a flow chart including blocks of an audio session management method according to some implementations.

[0121] Figure 14 is a flow chart including blocks of an audio session management method according to some implementations.

[0122] Figure 15 is a flow diagram including blocks of an automatic setup process for one or more audio devices newly introduced to an audio environment in accordance with some implementations.

[0123] Figure 16 is a flow diagram including blocks of a process for installing a virtual assistant application according to some implementations.

[0124] Figure 17 is a flow chart including blocks of an audio session management method according to some implementations.

[0125] Figure 18A is a block diagram of a minimal version of an embodiment.

[0126] Figure 18B Another (more capable) embodiment is depicted having additional features.

[0127] Figure 19 is an overview of the Figure 6 、 Figure 18A or Figure 18B Flowchart of one example of a method performed by those devices or systems shown in FIG.

[0128] Figure 20 and Figure 21 An example of a floor plan showing connected living spaces.

[0129] Figure 22 and Figure 23 An example of a multi-stream renderer providing simultaneous playback of a spatial music mix and voice assistant responses is shown.

[0130] Figure 24 Depicts an example starting point where all speakers across the living room and kitchen are playing a spatial music mix optimally for the many people at a party.

[0131] Figure 25 An example of a baby trying to sleep in a bedroom is shown.

[0132] Figure 26 An example of reproducing an additional audio stream is described.

[0133] Figure 27 Shown Figure 18A Frequency domain / transform domain example of a multi-stream renderer shown in .

[0134] Figure 28 Shown Figure 18B Frequency domain / transform domain example of a multi-stream renderer shown in .

[0135] Figure 29 Depicted is a floor plan of a listening environment, which in this example is a living space.

[0136] Figure 30 、 Figure 31 、 Figure 32 and Figure 33 Shown for Figure 29 An example of flexible rendering of spatial audio in reference space mode for multiple different listening positions and orientations in a living space is shown in FIG.

[0137] Figure 34 An example of reference spatial mode rendering is shown when two listeners are at different locations in the listening environment.

[0138] Figure 35 An example of a GUI for receiving user input regarding the position and orientation of a listener is shown.

[0139] Figure 36 Shown is an example of the geometric relationship between three audio devices in an environment.

[0140] Figure 37 Shown Figure 36 Another example of the geometric relationship between three audio devices in the environment is shown in .

[0141] Figure 38 Shown without corresponding audio equipment and other characteristics of the environment Figure 36 and Figure 37 The two triangles depicted in .

[0142] Figure 39 An example of estimating the interior angle of a triangle formed by three audio devices is shown.

[0143] Figure 40 is an overview of the Figure 6 Flowchart of an example of a method performed by the apparatus shown in FIG.

[0144] Figure 41 An example is shown where each audio device in the environment is a vertex of multiple triangles.

[0145] Figure 42 An example of a portion of the forward alignment process is provided.

[0146] Figure 43 An example of multiple audio device position estimates that have occurred during the forward alignment process is shown.

[0147] Figure 44 An example of a portion of the reverse alignment process is provided.

[0148] Figure 45 An example of multiple audio device position estimates that have occurred during the reverse alignment process is shown.

[0149] Figure 46 A comparison of the estimated audio device position and the actual audio device position is shown.

[0150] Figure 47 is an overview of the Figure 6 Flowchart of another example of a method performed by the apparatus shown in .

[0151] Figure 48A Shown Figure 47 Examples of some boxes.

[0152] Figure 48B Additional examples of determining listener angular orientation data are shown.

[0153] Figure 48C Additional examples of determining listener angular orientation data are shown.

[0154] Figure 48D Shown according to reference Figure 48C An example of a method is described to determine the appropriate rotation of audio device coordinates.

[0155] Figure 49 is a block diagram illustrating an example of components of a system capable of implementing various aspects of the present disclosure.

[0156] Figure 50A 、 Figure 50B and Figure 50C Examples of playback limit thresholds and corresponding frequencies are shown.

[0157] Figure 51A and Figure 51B is a diagram showing an example of dynamic range compression data.

[0158] Figure 52 Examples of spatial regions of a listening environment are shown.

[0159] Figure 53 Shown Figure 52 Example of a loudspeaker within a region of space.

[0160] Figure 54 Shown covered in Figure 53 Examples of spatial regions and nominal spatial positions on the loudspeakers.

[0161] Figure 55 is a flow chart outlining one example of a method that may be performed by an apparatus or system such as those disclosed herein.

[0162] Figure 56A and Figure 56B An example of a system that may be implemented according to some embodiments is shown.

[0163] Figure 57 is a block diagram of a system implemented in an environment (eg, a home) according to an embodiment.

[0164] Figure 58 yes Figure 57 A block diagram of elements of an example embodiment of module 5701.

[0165] Figure 59 yes Figure 57 Another example embodiment of module 5701 (in Figure 59 A block diagram of the device and its operation is shown in FIG. 5 .

[0166] Figure 60 is a block diagram of another example embodiment. DETAILED DESCRIPTION

[0167] Many embodiments are disclosed. It will be apparent to those skilled in the art from this disclosure how to implement the embodiments.

[0168] Currently, designers view audio devices as a single interface point for audio, which can be a mix of entertainment, communication, and information services. Using audio for notifications and voice control offers the advantage of avoiding visual or physical distractions. The expanding device landscape is fragmented, with more systems competing for our ears. While wearable augmented audio is becoming available, progress toward the ideal ubiquitous audio personal assistant seems off, and seamless capture, connection, and communication with the myriad of devices around us is no longer possible.

[0169] It would be useful to develop services that bridge devices and better manage location, context, content, timing, and user preferences. A set of standards, infrastructure, and APIs together could enable better access to a unified audio space around the user. Imagine an operating system for audio devices that manages basic audio input and output and allows our audio devices to connect to specific applications. This kind of thinking and design creates a scaffold for interactive audio transmission, for example to provide services that allow rapid and organic development of improvements and provide device-independent audio connections for others.

[0170] The scope of audio interaction includes real-time communication, asynchronous chat, alerts, transcription, history, archiving, music, recommendations, reminders, promotions and context-aware help. A platform that promotes a unified approach and can implement smart media formats is disclosed herein. The platform can include or implement ubiquitous wearable audio and / or can implement user positioning, selection of single or multiple (e.g., collective) audio devices for optimal use, management of identity, privacy, temporality, geolocation and / or infrastructure for transmission, storage, retrieval and algorithm execution. Some aspects of the present disclosure can include identity, priority (rank) and respect for user preferences, for example, managing the expectation of hearing and the value of being heard. The cost of unwanted audio is high. It is envisioned that an 'audio internet' can provide or implement the complete elements of security and trust.

[0171] While the categories of single-purpose audio devices and multi-purpose audio devices are not strictly orthogonal, one or more speakers and one or more microphones of an audio device (e.g., a smart audio device) may be assigned functionality enabled by or attached to (or implemented by) the smart audio device. However, one or more speakers and / or one or more microphones of an audio device considered separately (as distinct from an audio device) are not typically added to the collective.

[0172] This document describes a class of audio device connectivity in which local audio devices (each of which may include a speaker and a microphone) are advertised and made available to a collective audio platform that exists in an abstract sense independent of any of the local audio devices. Also described are embodiments including at least one discoverable opportunistically orchestrated distributed audio device subsystem (DOODAD), which implements a design methodology and set of steps aimed at enabling this concept of collective audio device orchestration and utilization.

[0173] Will refer to Figure 1A Simple examples are described to demonstrate the application and results of some embodiments of the present disclosure.

[0174] Figure 1A An example of people and smart audio devices in an audio environment is shown. In this example, the audio environment 100 is a home audio environment.

[0175] exist Figure 1A In the scenario, a person (102) has a smart audio device (101) that can capture the user's voice with its microphone (103) and can play it back through its speakers (104). The user can speak at a considerable distance from the device 101, which limits the duplex capability of the device 101.

[0176] exist Figure 1A The marked elements are:

[0177] 100. An audio environment depicting an example scenario using a smart audio device. A person 102 sitting in a chair is interacting with the device 101.

[0178] 101. A smart audio device capable of playing back audio through one or more speakers and capturing audio from one or more microphones.

[0179] 102. A person participating in an audio experience using device 101 (also referred to as a user or listener).

[0180] 103. The sound produced when person 102 speaks to device 101.

[0181] 104. Audio playback from one or more speakers of device 101.

[0182] Therefore, in this example, Figure 1A 1 is a diagram of a person (102) at home and an intelligent audio device (101), which in this case is a telephone device used for communication. Device 101 is capable of outputting audio that person 102 hears, and is also capable of capturing sound (103) from person 102. However, device 101 is some distance away from person 102, so the microphone on device 101 faces the challenge of hearing person 102 through the audio being output by device 101—a challenge known as the echo problem. Established techniques for addressing this problem typically employ echo cancellation, a form of processing that is severely limited by full-duplex activity (two-way speech or audio in two directions simultaneously).

[0183] Figure 1B yes Figure 1A FIG2 is a modified version of a scenario in which a second smart audio device (105) is also present in the audio environment 100. The smart audio device 105 is located near the person 102 but is designed to output audio. In some examples, the smart audio device 105 may at least partially implement a virtual assistant.

[0184] ( Figure 1A The person 102 has now acquired a second smart audio device 105, but the device 105 ( Figure 1B ) can only perform a specific purpose (106) that is completely separate from the purpose of the first device (101). In this example, the two smart audio devices (101 and 105) cannot share information and choreographed experiences despite being in the same acoustic space as the user.

[0185] exist Figure 1B The marked elements are:

[0186] 100-104. See Figure 1A .

[0187] 105. Additional intelligent audio devices capable of audio playback through one or more speakers and audio capture from one or more microphones.

[0188] 106. Audio playback through one or more speakers of device 105.

[0189] While it is possible to pair or transfer an audio call from a phone (101) to the smart audio device (105), this has not previously been possible without user intervention and detailed configuration. Figure 1B The scenario depicted in

[10] has two independent audio devices, each performing a very specific application. In this example, the smart audio device 105 was purchased later than the device 100. The smart audio device (105) was purchased for its specific use(s) and is usable out of the box only for its specific use(s), and does not immediately add value to the device (101) that is already present in the audio environment 100 and is used as a communication device.

[0190] Figure 1C In this example, the smart audio device 105 was purchased later than the smart audio device 100 .

[0191] exist Figure 1C In an embodiment of the present invention, smart audio devices 101 and 105 are capable of being arranged. In this arrangement example, when smart audio device 101 plays sound 104 from one or more of its speakers, smart audio device (105) is better positioned to pick up the voice (103B) of person 102 for a phone call involving smart audio device (101).

[0192] exist Figure 1C The marked elements are:

[0193] 100-104. See Figure 1A .

[0194] 105. See Figure 1B .

[0195] 103B. The sound produced by the person 102 is better captured by the smart audio device (105) because the microphone or microphones of the smart audio device (105) are closer to the user.

[0196] exist Figure 1CIn some manner, a new smart audio device 105 (an example of which is described herein) is detected so that the microphone in the device 105 can be used to support applications running on the smart audio device (101). Figure 1C The new smart audio device 105 is in some sense similar to Figure 1C The device 101 coordinates or orchestrates (according to some aspects of the present disclosure) such that the proximity of the smart audio device 105 to the person 102 is detected and opportunistically understood as directed to Figure 1C The superior microphone for the situation described. Figure 1C In the example, audio 104 originates from a relatively distant speakerphone 101; however, audio 103B to be transmitted to phone 101 is captured by local smart audio device 105. In some embodiments, opportunistic use of components of different smart audio devices is possible without phone 101 and / or the application used to provide the phone call being aware of the routing complexities and capabilities of smart audio device 105. Instead, in some such examples, a hierarchical system can be implemented for discovering, routing, and utilizing such capabilities.

[0197] Later, the concept of an abstracted Continuous Hierarchical Audio Session Manager (CHASM) is described in more detail. Some implementations of CHASM can provide audio capabilities to applications without requiring the application to understand all the details of managing devices, device connections, simultaneous device usage, and / or device leveling and tuning. In a sense, this approach assumes that the device that typically runs the application (and has at least one speaker and at least one microphone) is relinquishing control of the audio experience. However, in situations where the number of speakers, and importantly, microphones, in a room far exceeds the number of people, it is seen that the solution to many audio problems may include the step of locating the device closest to the person of interest—which may not be the device typically used for such an application.

[0198] One way to think about audio transducers (speakers and microphones) is that they can implement a step in the path of audio from a person's mouth to an application, and a return step in the path from the application to the person's ear. In this sense, it can be seen that any application that needs to transmit or capture audio from a user can be improved (or at least not made worse) by taking advantage of the device's opportunistic advantages and interfacing with the audio subsystem on any device to output or obtain input audio. In some examples, such decisions and routing can be made in a continuous manner as devices and users move or become available or are removed from the system. In this regard, the continuous hierarchical audio session manager (CHASM) disclosed herein is useful. In some embodiments, a discoverable opportunistic orchestration distributed audio device subsystem (DOODAD) can be included in CHASM as a collective, or DOODAD can be used together with CHASM as a collective.

[0199] Some disclosed embodiments implement the concept of a collective audio system designed to route signals between people and places. This departs from traditional "device-centric" designs that typically involve inputting and outputting audio from devices and then managing the devices collectively.

[0200] refer to Figures 2A to 2D Next, some example embodiments of the present disclosure are described. Starting with a device that has communication capabilities. In this case, for example, consider a doorbell audio intercom device that, when activated, starts a local application that creates a full-duplex audio link from the local device to a remote user. In this mode, the device's basic functionality is to manage a speaker and microphone and relay the speaker and microphone signals via a bidirectional network stream (which may be referred to herein as a "link").

[0201] Figure 2A is a block diagram of a conventional system. In this example, the operation of the smart audio device 200A involves the application (205A) sending media (212) and control information (213) to and from the media engine (201). Both the media engine 201 and the application 205A can be implemented by the control system of the smart audio device 200A. According to this example, the media engine 201 is responsible for managing audio input (203) and output (204) and can be configured for signal processing and other real-time audio tasks. The application 205A can also have other network input and output connections (210).

[0202] exist Figure 2A The marked elements are:

[0203] ·200A. Intelligent audio equipment for specific purposes.

[0204] 201. A media engine responsible for managing the real-time audio media stream from the application 205A and signal processing for one or more microphone inputs and one or more speaker outputs. Examples of signal processing may include acoustic echo cancellation, automatic gain control, wake-up word detection, limiters, noise suppression, dynamic beamforming, speech recognition, encoding / decoding to lossy formats, voice activity detection and other classifiers, etc. In this context, the phrase "real-time" may, for example, refer to the requirement that the processing of an audio block be completed within the time it takes for an analog-to-digital converter (ADC) implemented by the device 200A to sample the audio block. For example, in a particular embodiment, an audio block may be 10ms-20ms in length and may contain 480-960 consecutive samples sampled at 48,000 samples per second.

[0205] 203. One or more microphone inputs. Inputs from one or more microphones capable of sensing acoustic information connected to the media engine (201) via multiple ADC interfaces.

[0206] 204. One or more speaker outputs. Output from one or more speakers capable of reproducing acoustic energy connected to the media engine (201) through a plurality of digital-to-analog converters (DACs) and / or amplifier interfaces.

[0207] 205A. An application ("app") running on device 200A that handles media to and from the network and is responsible for sending and receiving media streams to and from media engine 201. In this example, app 205A also manages control information sent and received by the media engine. Examples of app 205A include:

[0208] oControl logic in the webcam that connects to the internet and streams data packets from the microphone to a web service;

[0209] o Conference calls that interface with a user interface via a touch screen, enabling users to dial phone numbers, browse contact lists, change volume, and initiate and terminate calls; and

[0210] o Voice-driven application in smart speakers that enables playback of music from a song library.

[0211] 210. Optional network connection that connects device 200A to a network (e.g., to the Internet via Wi-Fi or Ethernet or 4G / 5G cellular radio). The network can carry streaming media traffic.

[0212] 212. Media streamed to and from the media engine (201). For example, an app 205A in a special-purpose teleconferencing device may receive Real-time Transport Protocol (RTP) packets from the network (210), remove headers, and forward the G.711 payload to the media engine 201 for processing and playback. In some such examples, the app 205A may be responsible for receiving the G.711 stream from the media engine 201 and packaging the RTP packets for upstream transmission over the network (210).

[0213] 213. Control signals sent to or from app 205A for controlling the media engine (201). For example, when the user presses the volume up button on the user interface, app 205A sends control information to the media engine 201 to amplify the playback signal (204). In the specific-purpose device 200A, control of the media engine (201) comes only from the local app (205A), which does not have the ability to control the media engine externally.

[0214] Figure 2B Shown Figure 2A The audio device 200B is capable of executing a second application. For example, the second application may be an application that continuously streams audio from a doorbell device for use in a security system. In this case, the same audio subsystem (see Figure 2A Description) and a second application that controls network flow in only one direction.

[0215] In this example, Figure 2B 1 is a block diagram of a specific-purpose smart audio device (200B) capable of hosting two applications or "apps" (app 205B and app 206) that send information to a media engine 202 (of device 200B) via a specific-purpose audio session manager (SPASM). Using the SPASM as an interface for apps 205B and 206, network media can now flow directly to media engine 202, where media 210 is media to and from the first app (206) and media 211 is media for the second app (205B).

[0216] The term SPASM (or Specific Application Audio Session Manager) is used herein to refer to an element or subsystem (of a device) that is configured to implement an audio chain for a single type of functionality that the device is manufactured to provide. A SPASM may need to be reconfigured (e.g., including disassembling the entire audio system) to implement changes in the device's operating mode. For example, the audio in most laptop computers is implemented as or using a SPASM, where the SPASM is configured (and reconfigurable) to implement any desired single-purpose audio chain for a specific functionality.

[0217] exist Figure 2B The marked elements are:

[0218] 200B. A smart audio device with a Specific Application Audio Session Manager (SPASM) 207B hosting two apps (app 205B and app 206).

[0219] 202-204. See Figure 2A .

[0220] 205B, 206. Apps running on the local device 200B.

[0221] 207B. A special-purpose audio session manager responsible for managing audio processing and exposing the capabilities of the media engine (202) of device 200B. The description between each app (206 or 205B) and SPASM 207B indicates how different apps may want to use different audio functions (for example, apps may require different sampling rates or different numbers of inputs and outputs), and SPASM exposes and manages all audio functions for different apps. The limitation of SPASM is that it is designed for a specific purpose and can only perform operations that it understands.

[0222] • 210. Media information streamed to or from the network for the first app (205B). The SPASM (207B) allows the media stream to be streamed directly to the media engine (202).

[0223] 211. Media information streamed to the network for the second app (206). In this example, app 206 does not have any media streams to receive.

[0224] 214. Control information for managing the functions of the media engine is sent between the SPASM (207B) and the media engine (202).

[0225] 215, 216. Control information sent between the app (205B, 206) and the SPASM (207B).

[0226] Use SPASM 207B as Figure 2B The inclusion of a separate subsystem for device 200B might seem to implement an artificial step in the design. In fact, from a single-purpose audio device design perspective, it does involve unnecessary work. Much of the value of a CHASM (described below) is enabled as a network effect that (in some sense) scales with the square of the number of nodes in the network. However, including a SPASM in a smart audio device (e.g., Figure 2B SPASM 207B) does offer advantages and value, including the following:

[0227] - The abstraction of control performed by the SPASM 207B more easily allows multiple applications to run on the same device;

[0228] The SPASM 207B is tightly coupled to the audio device and reduces latency between audio data on the network and physical input and output sound by bringing the network streaming connection directly to the SPASM 207B. For example, the SPASM 207B may reside at a lower layer (e.g., a lower OSI or TCP / IP layer) within the smart audio device 200B, closer to the device driver / data link layer or below the physical hardware layer. If the SPASM 207B were implemented at a higher layer, for example, as an application running within the device operating system, such an implementation would likely result in a latency penalty because the audio data would need to be copied back from the lower layers to the application layer via the operating system. A potentially more detrimental feature of such an implementation is that latency may be variable or unpredictable.

[0229] -The design is now ready for greater interconnectivity at the lower audio level, before the application level.

[0230] In some examples, in a smart audio device whose operating system runs a SPASM, many apps can gain shared access to one or more speakers and one or more microphones of the smart audio device. According to some examples, by introducing a SPASM that does not need to send or receive one or more audio streams, the media engine can be optimized for very low latency because the media engine is separated from the control logic. Devices with a SPASM can allow applications to establish additional media streams (e.g., Figure 2B The benefit comes from separating the media engine functionality from the control logic of the SPASM. Figure 1A and Figure 2A In contrast to the case where the media engine shown in FIG is application specific and independent, in which the device is not designed to have access to / from attached devices by including SPASM (e.g., Figure 2BIn some such examples, if, for example, Figure 1A and Figure 2A If the devices shown in FIG are designed to be standalone, then simply updating application 205A, for example, to provide orchestration may not be feasible. However, in some examples, device 200B is designed to be orchestration-ready.

[0231] refer to Figure 2C Next, an aspect of some disclosed embodiments of advertising and allowing control of the SPASM itself (of a smart audio device) is described. This can allow other devices and / or systems in the network to utilize the protocol to better understand the audio capabilities of the device, and from a security and usability perspective, where applicable, can allow audio streams to be connected directly to the device to be played out through one or more speakers or sourced from one or more microphones. In this case, it is seen that a second application for setting up the ability to continuously stream audio from the device does not require a locally running application to control the media engine (e.g., 202) to stream out, for example, the monitoring stream (e.g., 211) mentioned above.

[0232] Figure 2C is a block diagram of an example of a disclosed embodiment. In this example, at least one of two or more apps (e.g., Figure 2C The app 205B) is implemented by a device other than the smart audio device 200C, for example, by one or more servers implementing a cloud-based service, another device in the audio environment where the smart audio device 200C resides, etc. Therefore, another controller (in this example, Figure 2C In this embodiment, the CHASM 208C is a controller that bridges the gap between one or more remote apps and the smart audio device (200C) with audio capabilities. In various embodiments, the CHASM (e.g., Figure 2C The CHASM 208C) can be implemented as a device or as a subsystem of a device (e.g., implemented in software of the device), wherein the device that is or includes the CHASM is different from one or more (e.g., many) smart audio devices. However, in some embodiments, the CHASM can be implemented via software that can potentially be executed by one or more devices of the audio environment. In some such embodiments, the CHASM can be implemented via software that can potentially be executed by one or more smart audio devices of the audio environment. Figure 2C , CHASM 208C and SPASM of device 200C (ie, Figure 2CThe SPASM 207C) coordinates to gain access to the media engine (202) that controls the audio input (203) and output (204) of the device 200C.

[0233] The term "CHASM" is used herein to refer to a manager (e.g., a device that is or implements an audio session manager (e.g., an ongoing audio session manager)) to which multiple devices (e.g., a collection of devices) (which may include, but are not limited to, smart audio devices) can make themselves available. In some embodiments, CHASM can continuously (at least during the time it implements what is referred to herein as "routing") adjust routing and signal processing for at least one software application. Depending on the specific implementation, the application may or may not be implemented on any of the devices in the audio environment. In other words, CHASM can be implemented or configured as an audio session manager (also referred to herein as a "session manager") for one or more software applications executed by one or more devices within the audio environment and / or one or more software applications executed by one or more devices outside the audio environment. Software applications may sometimes be referred to herein as "apps."

[0234] In some examples, as a result of using CHASM, an audio device can ultimately be used for purposes that were not envisioned by the creator and / or manufacturer of the audio device. For example, a smart audio device (including at least one speaker and microphone) can enter a mode in which the smart audio device provides a speaker feed signal and / or microphone signal to one or more other audio devices in an audio environment because the app (e.g., implemented on another device different from the smart audio device) requires CHASM (coupled to the smart audio device) to find and use all available speakers and / or microphones (or a group of available speakers and / or microphones selected by CHASM) that can include speakers and / or microphones of more than one audio device from the audio environment. In many such embodiments, the application does not need to select a device, speaker and / or microphone because CHASM will provide this functionality. In some examples, the application may not know (e.g., CHASM may not indicate to the application) which specific audio devices are involved in implementing the commands provided to CHASM by the application.

[0235] exist Figure 2C The marked elements are:

[0236] • 200C. A special-purpose smart audio device that is a session manager running local apps (206) and remote apps (205B) via CHASM (208C).

[0237] 202-204. See Figure 2A .

[0238] 205B. An app running remotely from device 200C, e.g., on a server with which CHASM 208C is configured to communicate via the Internet or on another device in the audio environment (a device different from device 200C, e.g., another smart audio device such as a cell phone). In some examples, app 205B can be implemented on a first device and CHASM 208C can be implemented on a second device, where both the first device and the second device are different from device 200C.

[0239] 206. App running locally on device 200C.

[0240] 207C. SPASM, which in addition to interfacing with the Media Engine (202) is also able to manage control inputs from the CHASM 208C.

[0241] 208C. A Continuous Layered Audio Session Manager (CHASM) that enables the app (205B) to utilize the audio capabilities, inputs (203), and outputs (204) of the media engine (202) of the device 200C. In this example, the CHASM 208C is configured to do so by obtaining at least partial control of the media engine (202) from the SPASM 207C via the SPASM (207C).

[0242] 210-211. See Figure 2C .

[0243] 217. Control information sent between the SPASM (207B) and the media engine (202) for managing the functions of the media engine.

[0244] 218. Control information between the native app 206 and the SPASM (207C) for implementing the native app 206. In some embodiments, such control information may be in accordance with an orchestration language, such as the orchestration languages ​​disclosed herein.

[0245] 219. Control information sent between the CHASM (208C) and the SPASM (207C) for controlling the functionality of the media engine 202. In some instances, such control information can be the same as or similar to the control information 217. However, in some implementations, the control information 219 can have a lower level of detail because, in some examples, device-specific details can be delegated to the SPASM 207C.

[0246] 220. Control information between app (205B) and CHASM (208C). In some examples, this control information may be in what is referred to herein as an "orchestration language."

[0247] For example, the control information 217 may include control signals from the SPASM 207C to the media engine 202 that have the effect of adjusting the output level of one or more output loudspeaker feeds (e.g., a gain adjustment specified in decibels or a linear scalar value), changing an equalization curve applied to one or more output loudspeaker feeds, etc. In some examples, the control information 217 from the SPASM 207C to the media engine 202 may include control signals that have the effect of changing one or more equalization curves applied to one or more output loudspeaker feeds, for example, by providing a new equalization curve that is described parametrically (as a series of combinations of basic filter stages) or a new equalization curve that is a list of gain values ​​at specific frequencies. In some examples, the control information 217 from the SPASM 207C to the media engine 202 may include control signals that have the effect of changing the upmixing or downmixing process of rendering multiple audio source feeds into one or more output loudspeaker feeds, for example, by providing a mixing matrix for combining the source feeds into the loudspeaker feeds. In some examples, control information 217 from SPASM 207C to media engine 202 may include control signals that have the effect of changing the dynamics processing applied to one or more output loudspeaker feeds (eg, changing the dynamic range of the audio content).

[0248] In some examples, the control information 217 from the SPASM 207C to the media engine 202 may indicate a change to the set of media streams provided to the media engine. In some examples, the control information 217 from the SPASM 207C to the media engine 202 may indicate a need to establish or terminate media streams with other media engines or other media content sources (e.g., a cloud-based streaming service).

[0249] In some instances, control information 217 may include control signals from media engine 202 to SPASM 207C, such as wake-up word detection information. In some instances, such wake-up word detection information may include a wake-up word confidence value or a message indicating that a possible wake-up word has been detected. In some examples, the wake-up word confidence value may be transmitted once per time interval (e.g., once every 100 ms, once every 150 ms, once every 200 ms, etc.).

[0250] In some instances, the control information 217 from the media engine 202 may include a speech recognition phone probability that allows the SPASM, CHASM, or another device (e.g., a device of a cloud-based service) to perform decoding (e.g., Viterbi decoding) to determine what command is being issued. In some instances, the control information 217 from the media engine 202 may include sound pressure level (SPL) information from a SPL meter. According to some such examples, the SPL information may be sent at time intervals, e.g., once per second, once per half a second, once every N seconds or N milliseconds, etc. In some such examples, the CHASM may be configured to determine whether there is a correlation between the SPL meter readings across multiple devices, e.g., to determine whether the devices are in the same room and / or are detecting the same sound.

[0251] According to some examples, the control information 217 from the media engine 202 may include information derived from a microphone feed presented as a media stream available to the media engine, such as an estimate of background noise, an estimate of direction of arrival (DOA) information, an indication of the presence of speech through voice activity detection, current echo cancellation performance, and the like. In some such examples, the DOA information may be provided to an upstream CHASM (or another device) that is configured to perform acoustic mapping of audio devices in an audio environment and, in some such examples, is configured to create an acoustic map of the audio environment. In some such examples, the DOA information may be associated with a wake-up word detection event. In some such implementations, the DOA information may be provided to an upstream CHASM (or another device) that is configured to perform acoustic mapping to locate the user who spoke the wake-up word.

[0252] In some examples, control information 217 from the media engine 202 may include state information, such as information about which active media streams are available, time positions within a linear time media stream (e.g., a television program, a movie, streaming video), information associated with current network performance (such as latency associated with active media streams), reliability information (e.g., packet loss statistics), etc.

[0253] Figure 2C The design of the device 200C can be extended in various ways according to various aspects of the present disclosure. Figure 2CThe function of SPASM 207C is to implement a set of hooks or functions to control local media engine 202. Therefore, device 200C can be considered closer (e.g., closer than the functionality of device 200A or device 200B) to a media engine that can connect to audio devices, network streams, perform signal processing, and respond to configuration commands from both one or more local applications of device 200C and also from the audio session manager. In this case, it is important that the device has information about itself (e.g., stored in a memory device and available to the audio session manager) to assist the audio session manager. Simple examples of this information include the number of speakers, the capabilities of one or more speakers, dynamic processing information, the number of microphones, information about the microphone arrangement and sensitivity, etc.

[0254] Figure 2D Shows examples of multiple applications interacting with CHASM. Figure 2D In the example shown in FIG, all apps, including apps local to the smart audio device (e.g., app 206 stored in the memory of the smart audio device 200D), need to interface with the CHASM 208D in order to provide functionality related to the smart audio device 200D. In this example, since the CHASM 208D has taken over the interfacing, the smart audio device 200D only needs to advertise its properties 207D, or make the properties available to the CHASM 208D, and the SPASM is no longer needed. This leaves the CHASM 208D as the primary controller for orchestrating experiences such as audio sessions for apps.

[0255] exist Figure 2D The marked elements are:

[0256] • 200D. Smart audio device implementing local app 206. CHASM 208B runs local app (206) and remote app (205B).

[0257] 202-204. See Figure 2A .

[0258] 205B. An application running remotely from the smart audio device 200D (in other words, running on a device other than the smart audio device 200D) (and in this example, also remotely from the CHASM 208D). In some examples, the application 205B can be executed by a device with which the CHASM 208D is configured to communicate, for example, via the Internet or via a local network. In some examples, the application 205B can be stored on another device in the audio environment (e.g., another smart audio device). According to some embodiments, the application 205B can be stored on a mobile device (e.g., a cell phone) that can be transported into or out of the audio environment.

[0259] 206. An app running locally on the smart audio device 200D, but control information (223) for the app is sent to and / or from the CHASM 208D.

[0260] 207D. Property Descriptors. With CHASM 208D assuming responsibility for managing media engine 202, smart audio device 200D can replace simple property descriptors with SPASMs. In this example, descriptor 207D indicates to CHASM the capabilities of media engine 202, such as the number of inputs and outputs, possible sampling rates, and signal processing components. In some examples, descriptor 207D can indicate data corresponding to one or more loudspeakers of smart audio device 200D, such as data indicating the type, size, and number of the one or more loudspeakers, data corresponding to the capabilities of the one or more loudspeakers, data related to dynamic processing that media engine 202 will apply to audio data before the audio data is reproduced by the one or more loudspeakers, and the like. In some examples, the descriptor 207D can indicate whether the media engine 202 (or more generally, the control system of the smart audio device 200D) is configured to provide functionality related to coordination of audio devices in the environment (such as rendering audio data to be reproduced by the smart audio device 200D and / or other audio devices of the audio environment), whether the control system of the smart audio device 200D is capable of implementing the functionality of the CHASM 208D if a device currently providing the functionality of the CHASM 208D (e.g., according to CHASM software stored in the device's memory) is turned off or otherwise ceases to operate, and the like.

[0261] 208D. In this example, CHASM 208D acts as a gateway for all apps (whether local or remote) to interact with media engine 202. Even local apps (e.g., app 206) gain access to local media engine 202 via CHASM 208D. In some instances, CHASM 208D may be implemented in only a single device of the audio environment, for example, via CHASM software stored on a wireless router, smart speaker, etc. However, in some implementations, more than one device of the audio environment may be configured to implement at least some aspects of CHASM functionality. In some examples, if the device currently providing the functionality of CHASM 208D is turned off or otherwise ceases to function, the control systems of one or more other devices in the audio environment (e.g., one or more smart audio devices of the audio environment) may be able to implement the functionality of CHASM 208D.

[0262] 210-211. See Figure 2C .

[0263] • 221. Control information sent between (eg, to and from) the CHASM 208D and the media engine 202.

[0264] 222. Data from property descriptor 207D sent to CHASM to indicate the capabilities of media engine 202.

[0265] 223. Control information sent between the local app 206 and the CHASM 208D.

[0266] 224. Control information sent between remote app 205B and CHASM 208D.

[0267] Additional embodiments are described below. To implement some of these embodiments, a single device (such as a communication device) is initially designed and coded for a specific purpose. An example of such a device is Figure 1C The smart audio device 101 can be Figure 3C As background, the Figure 1A and Figure 1B Implementation of device 101.

[0268] Figure 3A is shown according to an example Figure 1A When the user's voice 103 is captured by the microphone 303, the local app 308A is responsible for managing the network stream 317 received via the network interface of the device 101, managing the media stream 341 and providing control signal transmission 340 to the media engine 301A.

[0269] exist Figure 3A The marked elements are:

[0270] 101, 103-104. See Figure 1A .

[0271] 301A. Media engine responsible for managing real-time audio media streams from app 308A.

[0272] 303. Microphone.

[0273] 304. Loudspeaker.

[0274] 308A. Local app.

[0275] 317. Media streaming to and from the network.

[0276] 340. Control information sent between app 308A and media engine 301A.

[0277] 341. Media streams sent to and from app 308A.

[0278] Figure 3B Shown according to an example Figure 1B In this example, the implementation details of Figure 1B Device 105 and Figure 1B In this case, both devices are designed to be 'orchestration ready' in the sense that a generic or flexible media engine is controlled through the abstraction of SPASM.

[0279] exist Figure 3B , the output 106 of the second device 105 is independent of the output 104 of the first device 101, and the input 103 to the microphone 303 of the device 101 can potentially capture the output 106 of the device 105. In this example, the devices 105 and 101 cannot work in an orchestrated manner.

[0280] exist Figure 3B The marked elements are:

[0281] 101, 103-106. See Figure 1B .

[0282] 301, 303-304. See Figure 3A .

[0283] 302. Media engine of device 105.

[0284] 305. Microphone of device 105.

[0285] 306. Loudspeaker of device 105.

[0286] 308B. Local app of device 101.

[0287] 312B. SPASM of device 101.

[0288] 314B. SPASM of device 105.

[0289] 317. Media streaming to and from the network.

[0290] 320. Local app of device 105.

[0291] 321. Control information sent between app 308B and SPASM 312B.

[0292] 322. Control information sent between SPASM 312B and media engine 301.

[0293] 323. Control information sent between (to and from) app 320 and SPASM 314B.

[0294] 324. Control information sent between (to and from) the SPASM 314B and the media engine 302.

[0295] 325. Media stream entering the media engine 302 from the network.

[0296] Figure 3C is a block diagram showing an example of CHASM orchestrating two audio devices in an audio environment. Based on the foregoing discussion, it should be understood that using Figure 3B The situation of the system will be determined by (for example, Figure 3CIn some embodiments, CHASM 307 may be used to better manage audio communication between phone 101 and device 105, where CHASM runs on modified versions of device 101 and device 105 or on another device. In some examples, by using CHASM 307, application 308 on phone 101 relinquishes direct control of its audio device 101 and defers to CHASM 307 for all audio control. According to some such examples, the signal from microphone 305 may include less echo than the signal from microphone 303. In some examples, CHASM 307 may infer that the signal from microphone 305 includes less echo than the signal from microphone 303 based on an estimation that microphone 305 is closer to person 102. In some such instances, CHASM may be utilized by routing raw or processed microphone signals from microphone 305 on device 105 as network streams to phone device 101, and these microphone signals may be used in preference to signals from local microphone 303 to achieve a better voice communication experience for person 102.

[0297] According to some embodiments, CHASM 307 can ensure that this remains the optimal configuration, for example, by monitoring the location of person 102, by monitoring the location of device 101 and / or device 105, etc. In some examples, CHASM 307 can ensure that this remains the optimal configuration. According to some such examples, CHASM 307 can ensure that this remains the optimal configuration by exchanging low-rate (e.g., low bit rate) data and / or metadata. For example, the location of person 102 can be tracked when only a small amount of information is shared between devices. If information is exchanged at a low bit rate between devices, considerations of limited bandwidth may not be a significant issue. Examples of low-bit rate information that can be exchanged between devices include, but are not limited to, information derived from microphone signals, for example, as described below with reference to the "follow me" embodiment. An example of low-bit rate information that can help determine which device's microphone has a higher speech echo ratio is an estimate of the SPL caused by the sound emitted by the local loudspeaker on each of the multiple audio devices in the audio environment during a time interval (e.g., during the last second). An audio device that emits more energy from one or more of its loudspeakers may capture fewer of the other sounds in the audio environment above the echo caused by the one or more loudspeakers. Another example of low-bitrate information that can be helpful in determining which device's microphone has a higher speech-to-echo ratio is the amount of energy in the echo prediction of the acoustic echo canceller of each device. A large amount of predicted echo energy indicates that one or more microphones of the audio device may be overwhelmed by the echo. In some such instances, there may be some echoes that the acoustic echo canceller will not be able to eliminate (assuming that the acoustic echo canceller has converged at this time). In some examples, if information about a problem or non-existence of microphone 305 is to be provided, CHASM 307 can be continuously ready for controlling device 101 to resume using the microphone signal from local microphone 303.

[0298] Figure 3C An example of an underlying system for coordinating an app 308 running on device 101 with device 105 is depicted. In this example, CHASM 307 causes the user's voice 103B to be captured into microphone 305 of device 105 and causes the captured audio 316 to be used in media engine 301 of device 101, while speakerphone output 104 comes from first device 101—thereby orchestrating an experience for user 102 across devices 101 and 105.

[0299] exist Figure 3C The marked elements are:

[0300] 101, 103B, 104, 105. See Figure 1C .

[0301] 301-306. See Figure 3B .

[0302] 307.CHASM.

[0303] 309. Control information between CHASM 307 and media engine 302.

[0304] 310. Control information between CHASM 307 and media engine 301.

[0305] 311. Control information between CHASM 307 and app 308 (to or from).

[0306] 312C. Device property descriptor of device 101.

[0307] 313. Control information and / or data from the device property descriptor 312C to the CHASM 307.

[0308] 314C. Device property descriptor of device 105.

[0309] 315. Control information and / or data from the device property descriptor 314C to the CHASM 307.

[0310] 316. Media flow from device media engine 302 to device media engine 301.

[0311] 317. Media stream to or from the network to the media engine 301.

[0312] In some embodiments, if DOODAD is included in a CHASM (e.g., CHASM 307) to interact with smart audio devices (e.g., to send and receive control information to and from each of the smart audio devices), and / or if DOODAD is provided (e.g., as a subsystem of a device such as devices 101 and 105 that is separate from a device implementing CHASM (e.g., a device implementing CHASM 307)) to interact with a CHASM (e.g., discussed below), Figure 4 If the smart audio device operates together with a CHASM 401, or CHASM 307), the need for a SPASM (in each of the one or more smart audio devices) is replaced by the smart audio devices advertising their audio capabilities (to the CHASM) and the applications following the operation of a single abstract control point (e.g., CHASM) for audio functions.

[0313] Figure 4 is a block diagram illustrating another disclosed embodiment. Figure 4The design introduces important abstractions, where in some embodiments, the application does not need direct selection or control, and in some instances, information may not be provided to the application about which specific audio devices are involved in performing functions related to the application, the specific functions of those audio devices, etc.

[0314] Figure 4 4 is a block diagram of a system including three separate physical audio devices 420, 421, and 422. In this example, each of the devices implements a discoverable opportunistically orchestrated distributed audio subsystem (DOODAD) and is controlled by a CHASM 401 running applications (410-412). According to this example, CHASM 401 is configured to manage media demands for each of the applications 410-412.

[0315] exist Figure 4 The marked elements are:

[0316] 400. Example of an orchestrated audio system across three different devices. Devices 420, 421, and 422 each implement DOODAD (DUDAD). Each of devices 420, 421, and 422 that implement DOODAD is sometimes referred to as a DOODAD itself. In this example, each DOODAD is (or is implemented by) an intelligent audio device that communicates with Figure 3C The smart audio device 101 is different in that it is or implements Figure 4 The DOODAD device itself does not implement the relevant application, but the device 101 implements the application 308;

[0317] 401.CHASM;

[0318] 410-412. apps, which in this example have different audio requirements. In some examples, each of apps 410-412 can be stored on and executable by a device in the audio environment or a mobile device sometimes located in the audio environment;

[0319] 420-422. Smart audio devices, each of which implements a discoverable opportunistically orchestrated distributed audio subsystem (DOODAD), such that each DOODAD operates within a separate physical smart audio device;

[0320] 430, 431 and 432. Control information sent between (to and from) apps 410, 411 and 412 and CHASM 401;

[0321] 433, 434, and 435. Control information sent between DOODADS 420-422 and CHASM 401;

[0322] 440, 441, and 442. Media Engine;

[0323] 450, 451 and 452. Device property descriptors. In this example, DOODADS 420-422 are configured to provide device property descriptors to CHASM 401;

[0324] 460-462. Loudspeaker;

[0325] 463-465. Microphone;

[0326] 470. Media streams flowing out from the media engine 442 to the network;

[0327] 471. Media flow from media engine 441 to media engine 442;

[0328] 472. Media flow from media engine 441 to media engine (440);

[0329] 473. A media stream from a cloud-based service for providing media to the media engine 441 via a network (e.g., a cloud-based service provided by one or more servers in a data center that provides media to the media engine 441 via the Internet);

[0330] 474. Media streaming from media engine 441 to cloud-based services.

[0331] 477. One or more cloud-based services, which may include one or more music streaming services, movie streaming services, TV show streaming services, podcast providers, etc.

[0332] Figure 4 4 is a block diagram of a system in which multiple audio devices have the potential to be routed to create an audio experience. According to this example, audio data corresponding to media stream 473 is received by media engine 441 of smart audio device 421, which provides media stream 471 to media engine 442, and media stream 472 is provided to media engine 440. According to some such embodiments, media engine 441 can process media stream 473 according to a sampling clock of media engine 441 (an example of a so-called "media engine sampling clock" herein).

[0333] In some such examples, CHASM 401 may have provided instructions and information regarding procedures related to obtaining and processing media stream 473 to media engine 441 via control information 434. Such instructions and information are examples of what are referred to herein as "audio session management control signals."

[0334] However, in some embodiments, CHASM 401 can transmit audio session management control signals without reference to the media engine sampling clock of media engine 441. Such an example has potential advantages because CHASM 401 is not required to synchronize media transmissions to, for example, the audio devices of the audio environment. Instead, in some embodiments, any such synchronization can be delegated to another device, such as smart audio device 421 in the aforementioned example.

[0335] According to some such embodiments, CHASM 401 may have provided audio session management control signals related to obtaining and processing media stream 473 to media engine 441 in response to control information 430, 431, or 432 from application 410, application 411, or application 412. Such control information is an example of what is referred to herein as an "application control signal." According to some embodiments, application control signals may be transmitted from the application to CHASM 401 without reference to the media engine sampling clock of media engine 441.

[0336] In some examples, CHASM 401 can provide audio processing information (including, but not limited to, rendering information) to media engine 441, as well as instructions for processing audio corresponding to media stream 473 accordingly. However, in some embodiments, a device implementing CHASM 401 (or a device implementing similar functionality (e.g., the functionality of a smart home hub as described elsewhere herein)) can be configured to provide at least some audio processing functionality. Some examples are provided below. In some such embodiments, CHASM 401 can be configured to receive and process audio data and provide the processed (e.g., rendered) audio data to an audio device of an audio environment.

[0337] Figure 5 is a flow diagram including blocks of an audio session management method according to some embodiments. As with other methods described herein, the blocks of method 500 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 500 can be performed simultaneously. For example, in some instances, blocks 505 and 510 can be performed simultaneously. Furthermore, some embodiments of method 500 may include more or fewer blocks than shown and / or described. The blocks of method 500 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6 The control system 610 shown in and described below, or one of the other disclosed control system examples.

[0338] According to some embodiments, the blocks of method 500 may be performed at least in part by a device implementing what is referred to herein as an audio session manager (e.g., CHASM). In some such examples, the blocks of method 500 may be performed at least in part by the device implementing what is referred to herein as an audio session manager (e.g., CHASM). Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 The described CHASM 208C, CHASM 208D, CHASM 307, and / or CHASM 401 may be performed. More specifically, in some embodiments, the functionality of the "audio session manager" referred to in the blocks of method 500 may be at least partially performed by CHASM 208C, CHASM 208D, CHASM 307, and / or CHASM 401.

[0339] According to this example, block 505 involves establishing a first application communication link between a first application device executing a first application and an audio session manager of an audio environment. In some examples, the first application communication link can be established via any suitable wireless communication protocol suitable for use within the audio environment, such as Zigbee, Apple's Bonjour (Rendezvous), Wi-Fi, Bluetooth, Bluetooth Low Energy (Bluetooth LE), 5G, 4G, 3G, General Packet Radio Service (GPRS), Amazon Sidewalk, or a custom protocol in Nordic's RF24L01 chip. In some examples, the first application communication link can be established in response to a "handshake" process, which in some examples can be initiated via a "handshake initiate" transmitted by the first application device to a device implementing the audio session manager. In some examples, the first application communication link can be established in response to what is referred to herein as a "routing initiation request" from the first application device. For convenience, the routing initiation request from the first application device may be referred to herein as a "first routing initiation request" to indicate that the routing initiation request corresponds to the "first application device." In other words, depending on the particular implementation, the term "first" may or may not have a temporal meaning in this context.

[0340] In one such example, the first application communication link may be executing on Figure 4A first application communication link may be established between a device on which application 410 is executing and CHASM 401. In some such examples, the first application communication link may be established in response to CHASM 401 receiving a first routing initiation request from the device on which application 410 is executing. For example, the device on which application 410 is executing may be a smart audio device of an audio environment. In some instances, the device on which application 410 is executing may be a mobile phone. Application 410 may be used to access media, such as music, television programs, movies, etc., via CHASM 401. In some instances, the media may be available for streaming via a cloud-based service.

[0341] Various examples of what is meant by "routing" are described in detail below. Generally, a route indicates the parameters of an audio session to be managed by the audio session manager. For example, a route initiation request may indicate an audio source and an audio environment destination. In some instances, the audio environment destination may correspond to at least one person in the audio environment. In some instances, the audio environment destination may correspond to a region or zone of the audio environment.

[0342] However, in most instances, the audio environment destination will not indicate any specific audio device that will be involved in reproducing the media in the audio environment. Instead, an application (such as application 410) can provide, for example, a route request for a specific type of media that should be available to a specific person in the audio environment. In various disclosed embodiments, the audio session manager is responsible for determining which audio devices will be involved in the routing, e.g., determining which audio devices will be involved in obtaining, rendering, and reproducing the audio data associated with the media. In some embodiments, the audio session manager is responsible for determining whether the audio devices involved in the routing have changed (e.g., in response to determining that the person who is the intended recipient of the media has changed location), updating corresponding data structures, etc. Detailed examples are provided below.

[0343] In this example, block 510 involves receiving, by the audio session manager, a first application control signal from a first application via a first application communication link. Figure 4 In some examples, the application control signal can correspond to control information 430 sent between app 410 and CHASM 401 (to and from). In some examples, the first application control signal can be sent after the audio session manager (e.g., CHASM 401) has initiated routing. However, in some instances, the first application control signal can correspond to a first routing initiation request. In some such examples, blocks 505 and 510 can be performed at least partially concurrently.

[0344] According to this example, block 515 involves establishing a first smart audio device communication link between the audio session manager of the audio environment and at least a first smart audio device. In this example, the first smart audio device is or includes a single-purpose audio device or a multi-purpose audio device. According to this embodiment, the first smart audio device includes one or more loudspeakers.

[0345] In some examples, as described above, the first application control signal and / or the first routing initiation request do not indicate any particular audio device to be involved in routing. According to some such examples, method 500 can involve, prior to block 515, determining (e.g., by the audio session manager) which audio devices of the audio environment will be at least initially involved in routing.

[0346] For example, Figure 4 The CHASM 401 may determine that audio devices 420, 421, and 422 will be involved in routing at least initially. Figure 4 In the example shown in FIG, the first smart audio device communication link of block 515 can be established between the audio session manager (CHASM 401 in this example) and the smart audio device 421. The first smart audio device communication link can be established with Figure 4 4 corresponds to the dashed line shown between the CHASM 401 and the smart audio device 421 in FIG, through which the control information 434 is transmitted. In some such examples, the first smart audio device communication link can be established via any suitable wireless communication protocol suitable for use within the audio environment, such as Apple Airplay, Miracast, Blackfire, Bluetooth 5, Real-time Transport Protocol (RTP), or the like.

[0347] exist Figure 5 In the example shown in FIG, block 520 involves determining, by the audio session manager, one or more first media engine capabilities of a first media engine of a first smart audio device. According to this example, the first media engine is configured to manage one or more audio media streams received by the first smart audio device and to perform first smart audio device signal processing on the one or more audio media streams according to a first media engine sampling clock. In the example described above, block 520 may involve the CHASM 401 receiving information about one or more capabilities of the media engine 441 from the smart audio device 421, for example, by providing the device property descriptor 451 to the CHASM 401. According to some embodiments, block 520 may involve the CHASM 401 receiving information about the capabilities of one or more loudspeakers of the smart audio device 421. In some examples, the CHASM 401 may have predetermined some or all of this information, for example, prior to blocks 505, 510, and / or 515 of method 500.

[0348] According to this example, block 525 involves the audio session manager controlling the first smart audio device based on the first media engine capabilities and by transmitting a first audio session management control signal to the first smart audio device via the first smart audio device communication link. According to some examples, the first audio session management control signal may cause the first smart audio device to delegate control of the first media engine to the audio session manager. In this example, the audio session manager transmits the first audio session management control signal to the first smart audio device without reference to the first media engine sampling clock. In some such examples, the first application control signal may be transmitted from the first application to the audio session manager without reference to the first media engine sampling clock.

[0349] In one example of block 525, CHASM 401 can control media engine 441 to receive media stream 473. In some such examples, CHASM 401 can provide, via the first audio session management control signal, to media engine 441 a uniform resource locator (URL) corresponding to a website from which media stream 473 can be received, and instructions for initiating media stream 473. According to some such examples, CHASM 401 can also have provided, via the first audio session management control signal, to media engine 441 instructions for providing media stream 471 to media engine 442 and for providing media stream 472 to media engine 440.

[0350] In some such examples, CHASM 401 may have provided audio processing information (including, but not limited to, rendering information) to media engine 441 via the first audio session management control signal, as well as instructions for accordingly processing the audio corresponding to media stream 473. For example, CHASM 401 may have provided the following instructions to media engine 441: for example, smart audio device 420 is to receive a speaker feed corresponding to the left channel, smart audio device 421 is to reproduce a speaker feed corresponding to the center channel, and smart audio device 422 is to receive a speaker feed corresponding to the right channel.

[0351] Various other examples of rendering are disclosed herein, some of which may involve CHASM 401 or another audio session manager communicating different types of audio processing information to smart audio devices. For example, in some embodiments, one or more devices of an audio environment may be configured to implement flexible rendering, such as centroid amplitude translation (CMAP) and / or flexible virtualization (FV). In some such embodiments, a set of audio device positions, an estimated current listener position, and an estimated current listener orientation may be provided to a device configured to implement flexible rendering. A device configured to implement flexible rendering may be configured to render audio for a group of audio devices in the environment based on the set of audio device positions, the estimated current listener position, and the estimated current listener orientation. Some detailed examples are described below.

[0352] In reference Figure 4 In the aforementioned examples of the method 500 described above, a device other than the audio session manager or the first smart audio device is configured to execute the first application. However, for example, as described above with reference to Figure 2C and Figure 2D As described, in some examples, the first smart audio device can be configured to execute a first application.

[0353] According to some such examples, for example, as mentioned above Figure 2C As described, the first smart audio device may include a specific-purpose audio session manager. In some such embodiments, the audio session manager may communicate with the specific-purpose audio session manager via a first smart audio device communication link. In some examples, the audio session manager may obtain one or more first media engine capabilities from the specific-purpose audio session manager. According to some such examples, the audio session manager may act as a gateway for all applications that control the first media engine, regardless of whether the application is running on the first smart audio device or on another device.

[0354] As described above, in some examples, method 500 may involve establishing at least a first audio stream corresponding to a first audio source (e.g., Figure 4 In some embodiments, establishing at least the first audio stream may involve causing the first smart audio device to establish at least the first audio stream by transmitting a first audio session management control signal to the first smart audio device via the first smart audio device communication link.

[0355] In some examples, method 500 can involve a rendering process that causes a first audio signal to be rendered as a first rendered audio signal. In some such implementations, the rendering process can be performed by a first smart audio device in response to a first audio session management control signal. In the examples described above, media engine 441 can render an audio signal corresponding to media stream 473 as a speaker feed signal in response to the first audio session management control signal.

[0356] According to some examples, method 500 may involve causing a first smart audio device to establish an inter-smart audio device communication link between the first smart audio device and each of one or more other smart audio devices in the audio environment via a first audio session management control signal. Figure 4 In the example described, media engine 441 can establish a wired or wireless communication link between smart audio devices with media engines 440 and 442. Figure 3C In the depicted example, the media engine 302 can establish a wired or wireless inter-smart audio device communication link to provide the media stream 316 to the media engine 301 .

[0357] In some examples, method 500 may involve causing a first smart audio device to transmit one or more of an original microphone signal, a processed microphone signal, a rendered audio signal, or an unrendered audio signal to one or more other smart audio devices via an inter-smart audio device communication link or multiple inter-smart audio device communication links. Figure 4 In the described examples, the intelligent inter-audio device communication link can be used to provide rendered audio signals or unrendered audio signals via media stream 471 and media stream 472. In some such examples, media stream 471 can include speaker feed signals for media engine 442, and media stream 472 can include speaker feed signals for media engine 440. Figure 3C In the depicted example, media engine 302 may provide raw microphone signals or processed microphone signals to media engine 301 via media stream 316 .

[0358] According to some examples, method 500 may involve establishing a second smart audio device communication link between an audio session manager and at least a second smart audio device of an audio environment. In some such examples, the second smart audio device may be a single-purpose audio device or a multi-purpose audio device. In some instances, the second smart audio device may include one or more microphones. Some such methods may involve determining, by the audio session manager, one or more second media engine capabilities of a second media engine of the second smart audio device. For example, the second media engine may be configured to receive microphone data from the one or more microphones and to perform second smart audio device signal processing on the microphone data.

[0359] For example, reference Figure 3C , the “first smart audio device” may be the smart audio device 101. According to some such examples, the “second smart audio device” may be the smart audio device 105. The “first smart audio device communication link” may be used to provide the control signal 310, and the “second smart audio device communication link” may be used to provide the control signal 309. The CHASM 307 may determine one or more media engine capabilities of the media engine 302 based at least in part on the device property descriptor 314c.

[0360] Some such methods may involve the audio session manager controlling the second smart audio device based on the second media engine capabilities by transmitting a second audio session manager control signal to the second smart audio device via a second smart audio device communication link. In some instances, controlling the second smart audio device may involve causing the second smart audio device to establish an inter-smart audio device communication link (e.g., an inter-smart audio device communication link for providing media stream 316) between the second smart audio device and the first smart audio device. Some such examples may involve causing the second smart audio device to transmit at least one of processed or unprocessed microphone data (e.g., processed or unprocessed microphone data from microphone 305) from the second media engine to the first media engine via the inter-smart audio device communication link.

[0361] In some examples, controlling the second smart audio device can involve receiving, by the audio session manager and via the first application communication link, a first application control signal from the first application. Figure 3C In the example of , CHASM 307 receives a control signal 311 from an application 308, which is a phone application in this example. Some such examples may involve determining a second audio session manager control signal based on the first application control signal. For example, referring again to Figure 3C, CHASM 307 may be configured to optimize a speech echo ratio (SER) for the conference call being provided based on a control signal 311 from application 308. CHASM 307 may determine that the SER of the conference call may be improved by capturing the speech of person 102 using microphone 305 instead of microphone 303 (see Figure 1C ). In some examples, this determination can be based on an estimate of the position of person 102. Some detailed examples of estimating the position and / or orientation of a person in an audio environment are disclosed herein.

[0362] Figure 6 is a block diagram illustrating an example of components of an apparatus capable of implementing various aspects of the present disclosure. As with the other figures provided herein, Figure 6 The types and quantities of elements shown in are provided as examples only. Other embodiments may include more, fewer and / or different types and quantities of elements. According to some examples, the device 600 may be or may include an intelligent audio device configured to perform at least some of the methods disclosed herein. In other embodiments, the device 600 may be or may include another device configured to perform at least some of the methods disclosed herein, such as a laptop computer, a cellular phone, a tablet device, a smart home hub, etc. In some such embodiments, the device 600 may be or may include a server. In some such embodiments, the device 600 may be configured to implement what is referred to herein as CHASM.

[0363] In this example, the device 600 includes an interface system 605 and a control system 610. In some embodiments, the interface system 605 can be configured to communicate with one or more devices that are executing or configured to execute a software application. Such software applications may sometimes be referred to as "applications" or simply "apps" in this document. In some embodiments, the interface system 605 can be configured to exchange control information and associated data related to the application. In some embodiments, the interface system 605 can be configured to communicate with one or more other devices of an audio environment. In some examples, the audio environment can be a home audio environment. In some embodiments, the interface system 605 can be configured to exchange control information and data associated with the audio devices of the audio environment. In some examples, the control information and associated data may be related to one or more applications with which the device 600 is configured to communicate.

[0364] In some embodiments, the interface system 605 can be configured to receive audio data. The audio data can include audio signals that are arranged to be reproduced by at least some speakers of the audio environment. The audio data can include one or more audio signals and associated spatial data. For example, the spatial data can include channel data and / or spatial metadata. The interface system 605 can be configured to provide rendered audio signals to at least some of a set of speakers in the environment. In some embodiments, the interface system 605 can be configured to receive input from one or more microphones in the environment.

[0365] The interface system 605 may include one or more network interfaces and / or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). In some embodiments, the interface system 605 may include one or more wireless interfaces. The interface system 605 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. In some examples, the interface system 605 may include a control system 610 and a memory system (e.g., Figure 6 However, in some examples, control system 610 may include a memory system.

[0366] The control system 610 may include, for example, a general purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0367] In some embodiments, the control system 610 can reside in more than one device. For example, a portion of the control system 610 can reside in a device within one of the environments depicted herein, and another portion of the control system 610 can reside in a device outside the environment (such as a server, a mobile device (e.g., a smart phone or tablet), etc.). In other examples, a portion of the control system 610 can reside in a device within one of the environments depicted herein, and another portion of the control system 610 can reside in one or more other devices of the environment. For example, control system functionality can be distributed across multiple smart audio devices of an environment, or can be shared by an orchestration device (such as a device that may be referred to as a smart home hub herein) and one or more other devices of the environment. In some such examples, the interface system 605 can also reside in more than one device.

[0368] In some implementations, the control system 610 can be configured to at least partially perform the methods disclosed herein.According to some examples, the control system 610 can be configured to implement an audio session management method.

[0369] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices such as those described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, etc. The one or more non-transitory media may reside, for example, on a computer. Figure 6 6 and / or reside in the control system 610. Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media having software stored thereon. For example, the software may include instructions for controlling at least one device to implement the audio session management method. In some examples, the software may include instructions for controlling one or more audio devices of the audio environment to obtain, process, and / or provide audio data. For example, the software may be provided by a computer such as a computer. Figure 6 The control system 610 is executed by one or more components of the control system.

[0370] In some examples, apparatus 600 may include Figure 6 Optional microphone system 620 is shown in FIG. Optional microphone system 620 may include one or more microphones. In some embodiments, one or more of the microphones may be part of or associated with another device (e.g., a speaker of a speaker system, a smart audio device, etc.). In some examples, device 600 may not include microphone system 620. However, in some such embodiments, device 600 may still be configured to receive microphone data from one or more microphones in the audio environment via interface system 610.

[0371] According to some embodiments, the apparatus 600 may include Figure 6Optional loudspeaker system 625 is shown in FIG. Optional loudspeaker system 625 may include one or more loudspeakers. Loudspeakers may sometimes be referred to herein as "speakers." In some examples, at least some of the loudspeakers of optional loudspeaker system 625 may be positioned arbitrarily. For example, at least some of the loudspeakers of optional loudspeaker system 625 may be placed in positions that do not correspond to any standard-mandated loudspeaker layout (e.g., Dolby 5.1, Dolby 5.1.2, Dolby 7.1, Dolby 7.1.4, Dolby 9.1, Hamasaki 22.2, etc.). In some such examples, at least some of the loudspeakers of optional loudspeaker system 625 may be placed in positions that are convenient for the space (e.g., in positions where there is room to accommodate the loudspeakers), but not in any standard-mandated loudspeaker layout.

[0372] In some embodiments, the apparatus 600 may include Figure 6 Optional sensor system 630 is shown in . Optional sensor system 630 may include one or more cameras, touch sensors, gesture sensors, motion detectors, and the like. According to some embodiments, optional sensor system 630 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of optional sensor system 630 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of optional sensor system 630 may reside in a television, a mobile phone, or a smart speaker. In some examples, device 600 may not include sensor system 630. However, in some such embodiments, device 600 may still be configured to receive sensor data from one or more sensors in the audio environment via interface system 610.

[0373] In some embodiments, the apparatus 600 may include Figure 6 Optional display system 635 is shown in FIG. Optional display system 635 may include one or more displays, such as one or more light emitting diode (LED) displays. In some instances, optional display system 635 may include one or more organic light emitting diode (OLED) displays. In some examples where device 600 includes display system 635, sensor system 630 may include a touch sensor system and / or a gesture sensor system proximate to one or more displays of display system 635. According to some such embodiments, control system 610 may be configured to control display system 635 to present one or more graphical user interfaces (GUIs).

[0374] According to some examples, the apparatus 600 may be or may include a smart audio device. In some such embodiments, the apparatus 600 may be or may (at least in part) implement a wake-up word detector. For example, the apparatus 600 may be or may (at least in part) implement a virtual assistant.

[0375] Return now Figure 4 , according to some examples, it is possible to implement Figure 4 The system enables CHASM 401 to provide an abstraction to applications 410, 411, and 412, so that applications 410, 411, and 412 can implement presentations (e.g., audio sessions) by publishing routes in an orchestration language. Using this language, applications 410, 411, and 412 can command CHASM 401 through control links 430, 431, and 432.

[0376] Refer to the above description Figure 1A 、 Figure 1B and Figure 1C , considers the application of the orchestration language and the description of the resulting situation in Figure 1A The situation and Figure 1C The situations are the same.

[0377] To describe the syntax examples and the choreography language examples, we first consider Figure 1A and Figure 1C Some examples of situations.

[0378] In some examples, what is referred to herein as a "route" in the orchestration language may include an indication of a media source (including but not limited to an audio source) and a media destination. For example, the media source and media destination may be specified in a route initiation request sent by an application to CHASM. According to some embodiments, the media destination may be or may include an audio environment destination. In some instances, the audio environment destination may correspond to at least one person who is present in the audio environment at least some of the time. In some instances, the audio environment destination may correspond to one or more areas or zones of the audio environment. Some examples of audio environment zones are disclosed herein. However, the audio environment destination will typically not include any specific audio device in the audio environment to which the route will relate. By making the orchestration language (including but not limited to the details required by the application to establish a route) more general, the details of how the route is implemented can be determined by CHASM and updated as needed.

[0379] In some examples, the route may include other information such as audio session priority, connection mode, one or more audio session targets or criteria, etc. In some implementations, the route will have a corresponding code or identifier, which may be referred to herein as an audio session identifier. In some instances, the audio session identifier may be a persistently unique audio session identifier.

[0380] In the initial example described above, the corresponding "routes" may include: Figure 1C The terminology is similar to the natural language statement that one might want to "call person X." However, the terminology is completely different from the statement (reference includes the device 101). Figure 1A (Situation):

[0381] Connecting the device microphone (i.e., the microphone of device 101) to processing (noise / echo removal);

[0382] Connect the processed output (noise / echo) to the network;

[0383] Connect the network to process input (dynamically);

[0384] Connect the processed output (dynamics) to the device speakers;

[0385] Connect the processing output (dynamic) to the processing input (reference).

[0386] At this point, it can be observed that executing a telephony application according to such a command list would involve details of the device and the processing required (echo and noise removal), and would not be possible if the device 105 were to be introduced (in other words, if the Figure 1C The context of the device 105 and the device 101 executing the phone application) needs to be completely changed, for example, as follows:

[0387] connecting the device microphone (i.e., the microphone of device 105) to the network;

[0388] Connect the network to input processing (noise / echo removal);

[0389] Connect the processed output (noise / echo) to the network;

[0390] Connect the network to process input (dynamically);

[0391] Connecting the processed output (dynamically) to the device speaker (i.e., the speaker of device 101);

[0392] Connect the processing output (dynamic) to the processing input (reference).

[0393] In some examples, the details of where best to do signal processing, how to connect the signals, and generally what the best results are for a user (who may be in a known or unknown location) may involve optimizations that can be pre-computed for a limited number of use cases, but may become unmanageable for a large number of devices and / or a large number of simultaneous audio sessions. It has been recognized that it would be desirable to provide an underlying framework that can allow for better connectivity, capabilities, knowledge, and control of smart audio devices (including by orchestrating or coordinating the devices), and then create a portable and efficient syntax for controlling the devices.

[0394] Some disclosed embodiments employ a method and language that is efficient in design and also very general. Certain aspects of the language are easiest to understand when one considers audio devices as part of a route (e.g., in embodiments where the system includes CHASM as described herein rather than SPASM as described herein) and not as specific endpoint audio devices. Aspects of some embodiments include one or more of: a route specification syntax; a persistent unique session identifier; and a continuous notion of delivery, acknowledgment, and / or quality.

[0395] Route specification syntax (addressing the requirement that each published route have either explicit or implicit elements) can include:

[0396] o Source (human / device / automated decision) and therefore implicit authorization;

[0397] o The priority of the desired audio routing in terms of importance relative to other audio that may already be in progress or occur later;

[0398] o Destination (ideally a person or group of people, and possibly generalizable to a place);

[0399] oConnection mode, including synchronous, transactional, or scheduled;

[0400] o the degree to which a message must be confirmed or the certainty required in terms of confidence in its delivery; and / or

[0401] o Understand what are the most important aspects of what is heard (intelligibility, quality, spatial fidelity, consistency, or perhaps inaudibility). This last point can include the concept of negative routing, where one is not only interested in hearing and being heard, but also in controlling what is not heard and / or cannot be heard. Some such examples involve ensuring that one or more areas of the audio environment remain relatively quiet, such as the "don't wake the baby" implementation described in detail below. Other such examples can involve preventing others in the audio environment from overhearing confidential conversations, for example, by reproducing "white noise" in one or more nearby loudspeakers, by increasing the playback level of one or more other audio content in one or more nearby loudspeakers, etc.

[0402] Aspects of the persistent unique session identifier may include the following. In some examples, a key aspect of some embodiments is that the audio session corresponding to a route is persistent until completed or otherwise closed. For example, this may allow the system (e.g., via CHASM) to track ongoing audio sessions and end or remove audio sessions to change routes, rather than requiring the application to determine which individual sets of connections must be changed. Once created, the persistent unique session identifier may relate to aspects of control and state that may allow the system to implement message-driven or poll-driven management. For example, controls for an audio session or route may be or include:

[0403] -Finish;

[0404] - the destination of movement; and

[0405] - Increase or decrease priority.

[0406] The content that can be queried about an audio session or route can be or include:

[0407] - Is it in the right place?

[0408] - How well the stated objectives are being achieved amidst competing priorities;

[0409] - How the user feels or feels confident hearing / confirming the audio;

[0410] - What is the quality (e.g., different targets for fidelity, spatiality, intelligibility, information, attention, consistency, or inaudibility); and

[0411] - If desired, query down to the actual routing layer as to which audio devices are being used.

[0412] A continuous concept of delivery, confirmation, and quality can include the following. While there may be some sense of a networking socket approach (and session layer), audio routing can be very different, especially considering the amount of audio activity that can be routed or queued simultaneously, etc. Similarly, since the destination can be at least one person and since in some instances there may be uncertainty about the location of that person relative to the audio devices that may be routed through, it can be useful to have a fairly continuous sense of certainty. Networking can include or be associated with links that are datagrams that may or may not arrive and streams that are guaranteed to arrive. In the case of audio, there can be a sense of things being heard or not being heard and / or a sense of someone that we think we can hear or not hear.

[0413] These items are introduced in some embodiments of the orchestration language, which may have some aspects of simple networking. Above this (in some embodiments) is a presentation layer and an application layer (e.g., a sample application for implementing a "phone call").

[0414] Embodiments of the orchestration language may have aspects related to the Session Initiation Protocol (SIP) and / or Media Server Markup Language (MSML) (e.g., device-centric, continuously and autonomously adapting routing based on the current set of audio sessions). SIP is a signaling protocol used to initiate, maintain, and end sessions that may include voice, video, and / or messaging applications. In some cases, SIP can be used to signal and control communication sessions for Internet telephony, such as for voice calls, video calls, dedicated IP phone systems, instant messaging over Internet Protocol (IP) networks, mobile phone calls, and the like. SIP is a text-based protocol that defines the format of messages and the communication sequence of participants. SIP includes elements of the Hypertext Transfer Protocol (HTTP) and the Simple Mail Transfer Protocol (SMTP). In some cases, a call established using SIP can include multiple media streams, but applications that exchange data as the payload of SIP messages (e.g., text messaging applications) do not require separate streams.

[0415] MSML is described in Request for Comments (RFC) 5707. MSML is used to control various types of services on IP media servers. According to MSML, a media server is a device specifically used to control and / or manipulate media streams, such as Real-time Transport Protocol media streams. According to MSML, an application server is separate from a media server and is configured to establish and terminate call connections. According to MSML, the application server is configured to establish a control "tunnel" via SIP or IP, which the application server uses to exchange requests and responses encoded in MSML with the media server.

[0416] MSML can be used to define how multimedia sessions interact on a media server and to apply services to individual users or groups of users. MSML can be used to control media server conference features (such as video layout and audio mixing), create sidebar conferences or personal mixes, and set the properties of media streams.

[0417] Some embodiments need not allow a user to control a cluster of audio devices by issuing specific commands. However, it is contemplated that some embodiments may effectively implement all desired presentations at the application layer without reference to the devices themselves.

[0418] Figure 7 is a block diagram depicting blocks of a CHASM according to one example. Figure 7 Depicts Figure 4An example of CHASM 401 is shown in FIG. Figure 7 It is shown that CHASM 401 receives routes from multiple apps using an orchestration language and stores information about the routes in routing table 701 . Figure 7 The elements include:

[0419] 401: CHASM;

[0420] 430: From the first application using the orchestration language ( Figure 4 Application 410) and the response from CHASM 401;

[0421] 431: From a second application using an orchestration language ( Figure 4 Application 411) commands and responses from CHASM 401;

[0422] 432: From a third application using an orchestration language ( Figure 4 Application 412) commands and responses from CHASM 401;

[0423] 703: From Figure 4 Commands to additional applications using an orchestration language not shown in FIG and responses from CHASM 401;

[0424] 701: routing table maintained by CHASM 401;

[0425] 702: An optimizer that continuously controls multiple audio devices based on current routing information, also referred to herein as an audio session manager;

[0426] 435: From CHASM 401 to the first audio device ( Figure 4 a command from the audio device 420) and a response from the first audio device;

[0427] 434: From CHASM 401 to the second audio device ( Figure 4 a command from the audio device 421) and a response from the second audio device;

[0428] 435: From CHASM 401 to the third audio device ( Figure 4 audio device 422) and a response from a third audio device.

[0429] Figure 8 Shown according to an example Figure 7 The details of the routing table are shown in . Figure 8 The elements include:

[0430] 701: Routing table maintained by CHASM. According to this example, each route has the following fields:

[0431] ID or “persistent unique session identifier”;

[0432] A record of which application has requested the route;

[0433] ·source;

[0434] A destination, which in this example may include one or more people or locations, but not an audio device;

[0435] Priority

[0436] a connection mode, which in this example is selected from a list of modes including synchronous mode, scheduled mode, and transactional mode;

[0437] Indication of whether confirmation is required;

[0438] Prioritize which audio quality aspect(s), which are also referred to herein as one or more audio session targets. In some examples, the audio session manager or prioritizer 702 optimizes the audio session according to the one or more audio session targets;

[0439] 801: A route in routing table 701 that has been requested by app 410 and assigned ID 50. This route specifies that Alex (the destination) wants to listen to Spotify with a priority of 4. In this example, priorities are integer values, with 1 being the highest priority. The connection mode is synchronous, meaning ongoing in this example. In this instance, Alex is not required to verify or confirm whether the corresponding music was provided to her. In this example, the only specified audio session goal is music quality;

[0440] 802: Route 51 has been requested by app 811 in routing table 701. Angus will hear a timer alarm with a priority of 4. This audio session, stored by CHASM 401 but not shown in routing table 701, is scheduled for a future time. In this example, Angus is asked to confirm that he has heard the alarm. In this example, the only specified audio session goal is audibility, to increase the likelihood that Angus will hear the alarm.

[0441] 803: The route in routing table 701 that has been requested by app 410 and assigned ID 52. Although the destination is "baby," the underlying audio session goal is inaudibility of the baby in the audio environment. Thus, this is an example of a "Don't wake the baby!" implementation, a detailed example of which is described below. The priority of this audio session is 2 (more important than almost anything else). The connection mode is synchronous (ongoing). No confirmation is required from the baby who has not yet been woken. In this example, the only specified audio session goal is inaudibility of sounds at the baby's location.

[0442] 804: A route in routing table 701 that has been requested by app 411 and assigned ID 53. In this example, app 411 is a phone app. In this instance, George is on the phone. Here, the audio session has a priority of 3. The connection mode is synchronous (ongoing). There is no need to confirm that George is still on the call. For example, when George is ready to end the call, he may intend to ask the virtual assistant to end the call. In this example, the only specified audio session goal is speech intelligibility.

[0443] 805: Route in routing table 701 that has been requested by app 412 and assigned ID 54. In this example, the underlying purpose of the audio session is to notify Richard that the plumber is at the door and needs to speak with Richard. The connection mode is transactional: the message is played to Richard as quickly as possible, given the priority of other audio sessions. In this example, Richard has just put the baby to bed and is still in the baby's room. Given that route 803 has a higher priority, CHASM's audio session manager will wait until Richard leaves the baby's room and until the message corresponding to route 805 is delivered. In this example, confirmation is required: in this instance, Richard is required to verbally confirm that he has heard the message and is on his way to see the plumber. According to some examples, if Richard does not confirm within a specified amount of time, CHASM's audio session manager may cause all audio devices in the audio environment (in some examples, except any audio devices in the baby's room) to provide the message until Richard responds. In this example, the only specified audio session goal is speech intelligibility so that Richard hears and understands the message.

[0444] 806: A route in routing table 701 that has been requested by the fire alarm system app 810 and assigned ID 55. The underlying purpose of this route is to sound a fire alarm in certain circumstances (e.g., based on a response from a smoke detection sensor) to evacuate the house. This route has the highest possible priority: even waking up a baby is acceptable. The connection mode is synchronous. No confirmation is required. In this example, the only specified audio session target is audibility. According to this example, CHASM will control all audio devices in the audio environment to play the alarm loudly to ensure that everyone in the audio environment hears the alarm and evacuates.

[0445] In some embodiments, the audio session manager (e.g., CHASM) maintains information corresponding to each route in one or more memory structures. According to some such embodiments, the audio session manager can be configured to update the information corresponding to each route based on changing conditions in the audio environment (e.g., a person changing positions in the audio environment) and / or based on control signals from the audio session manager 702. For example, with reference to route 801, the audio session manager can store and update a memory structure that includes or corresponds to the following information:

[0446]

[0447] Table 1

[0448] For the purpose of providing an example, the information shown in Table 1 is in a human-readable format. Depending on the particular implementation, the actual format used by the audio session manager to store such information (eg, destination location and destination orientation) may or may not be understandable to humans.

[0449] In this example, the audio session manager is configured to monitor Alex's location and orientation, the destination of route 801, and determine which audio devices will be involved in providing audio content for route 801. According to some such examples, the audio session manager can be configured to determine the audio device location, the person's location, and the person's orientation according to the methods described in detail below. In some embodiments, if the information in Table 1 changes, the audio session manager will send corresponding command / control signals to the device that is rendering audio from the media stream for route 801 and will update the memory structure as depicted via Table 1.

[0450] Figure 9A Examples of context-free grammars for routing requests in an orchestration language. In some examples, Figure 9AThe syntax of a request for routing from an application to a CHASM may be represented. For example, a routing request may be triggered based on a user having selected an application and interacting with the application, such as by selecting an icon corresponding to the application on a cell phone, via a voice command, etc.

[0451] In this example, the combination of element 901 and elements 902A, 902B, 902C, and 902D allows for definition of routing sources. In this example, as depicted by elements 902A, 902B, 902C, and 902D, routing sources can be or may include one or more people, services, and audio environment locations. For example, a service can be a cloud-based media streaming service, a home service that provides an audio feed from an external doorbell or from an audio device associated with the doorbell, and the like. In some embodiments, a service can be specified based on a URL (e.g., Spotify's URL), a service name, the IP address of my house doorbell, and the like. In some embodiments, an audio environment location can correspond to an audio environment zone described below. In some examples, an audio environment location source can correspond to one or more microphones in a zone. The comma in element 902D indicates that more than one source can be specified. For example, a routing request might indicate "routes from Roger, Michael," "routes from Spotify," "routes from the kitchen," "routes from Roger and the kitchen," and the like.

[0452] In this example, the combination of element 903 and elements 904A, 904B, 904C, and 904D allows for definition of a routing destination. In this embodiment, a routing destination may be or may include one or more people, services, and audio environment locations. For example, a routing request may indicate "route to David," "route to the kitchen," "route to the back deck," "route to Roger and the kitchen," and so on.

[0453] In this example, each route may select only one connection mode. Depending on the embodiment, the connection mode options are synchronous, scheduled, or transactional. However, in some embodiments, each route may select more than one connection mode. For example, in some such embodiments, a route initiation request may indicate that a route may be both scheduled and transactional. For example, a route initiation request may indicate that a message should be delivered to David at a predetermined time and that David should reply to the message. Although Figure 9A Not shown, but in some embodiments, a specific message may be included in the routing initiation request, such as a pre-recorded message.

[0454] In this example, the audio session target is referred to as a "trait." According to this example, one or more audio session targets may be indicated in the route initiation request via a combination of quality 907 and one or more traits 908A. Comma 908B indicates that, according to this example, one or more traits may be specified. However, in alternative embodiments, only one audio session target may be indicated in the route initiation request.

[0455] Figure 9B An example of an audio session goal is provided. According to this example, the "Qualities" list 908A allows for the specification of one or more important qualities. In some embodiments, a routing initiation request may specify more than one quality, for example, in descending order. For example, a routing initiation request may specify (Quality=Intelligibility, Spatial Fidelity), which means that intelligibility is the most important quality, followed by spatial fidelity. A routing initiation request may specify (Quality=Audience), which means that the only audio session goal is that people can hear, such as an alarm.

[0456] The route initiation request may specify (quality=inaudible), which means that the only audio session goal is to make the audio reproduced in the audio environment inaudible to the person designated as the routing destination (e.g., a baby). This is an example of a route initiation request for a "don't wake the baby" implementation.

[0457] In another example, a route initiation request may specify (quality = audibility, privacy). For example, this may mean that the primary audio session goal is to allow the person designated as the route destination to hear the delivered audio, but the secondary audio session goal is to limit the extent to which other persons, such as during a confidential telephone conversation, can hear the audio delivered and / or exchanged according to the route. As described elsewhere herein, the latter audio session goal may be achieved by reproducing white noise or other masking noise between the route destination and one or more other persons in the audio environment, increasing the volume of other audio reproduced near one or more other persons in the audio environment, and the like.

[0458] Return now Figure 9A In this example, the routing initiation request can specify a priority via elements 909 and 910. In some examples, the priority can be indicated via an integer from a limited number of integers (e.g., 3, 4, 5, 6, 7, 8, 9, 10, etc.). In some examples, 1 can indicate the highest priority.

[0459] According to this example, the route initiation request can optionally specify a confirmation via element 911. For example, the route initiation request can indicate "tell Michael Richard that dinner is ready and get confirmation." In some examples, in response, the audio session manager can attempt to determine Michael's location. For example, the CHASM can infer that Michael is in the garage because that is where Michael's voice was last detected. Therefore, the audio session manager can cause the announcement "Dinner is ready; please confirm that you heard this message" to be played via one or more loudspeakers in the garage. If Michael responds, the audio session manager can cause the response to be reported / replayed to Richard. If (e.g., after ten seconds) Michael does not respond to the garage notification, the audio session manager can cause the announcement to be played at Michael's second most likely location (e.g., where Michael spends a lot of time or where he was last heard before the previous garage utterance). Assume that location is Michael's bedroom. If (e.g., after ten seconds) Michael does not respond to the notification in Michael's bedroom, the audio session manager can cause the announcement to be played via many loudspeakers in the environment, subject to other restrictions, such as "do not wake the baby."

[0460] Figure 10 The flow of a request to modify a route according to an example is shown. For example, a route modification request can be transmitted by an application and received by an audio session manager. For example, the route modification request can be triggered based on a user selecting an application and interacting with the application.

[0461] In this example, ID 1002 refers to a persistent unique audio session number or code that the audio session manager previously provided to the app in response to a route initiation request. According to this example, a connection mode change can be made via element 1003 and elements 1004A, 1004B, or 1004C. Alternatively, if a connection mode change is not desired, elements 1004A, 1004B, and 1004C can be bypassed.

[0462] According to this example, one or more audio session targets may be changed via elements 1005, 1006A, and 1006B. Alternatively, if no audio session target changes are desired, elements 1005, 1006A, and 1006B may be bypassed.

[0463] In this example, routing priority may be changed via elements 1007 and 1008. Alternatively, if routing priority changes are not desired, elements 1007 and 1008 may be bypassed.

[0464] According to this example, element 1009 or element 1011 can be used to make a change in the confirmation requirement. For example, element 1009 indicates that if confirmation was not previously required for the route, then confirmation can be added. Conversely, element 1011 indicates that if confirmation was previously required for the route, then confirmation can be removed. The semicolon in element 1010 indicates the end of the request to modify the route.

[0465] Figure 11A and Figure 11B Additional examples of the flow of requests for modifying routes are shown. Figure 11C An example of a process for deleting a route is shown. For example, a route modification or deletion request may be transmitted by an application and received by the audio session manager. For example, a route modification request may be triggered based on a user having selected an application and interacting with the application. Figure 11A and Figure 11B In the example, "sink" refers to the routing destination. As with other flowcharts disclosed in this article, Figures 11A to 11C The operations shown in are not necessarily performed in the order indicated. For example, in some embodiments, the routing ID may be specified earlier in the process (eg, at the beginning of the process).

[0466] Figure 11A A process 1100A for adding a source or destination is shown. In some instances, one or more sources or destinations may be added. In this example, a routing source may be added via elements 1101 and 1102A. According to this example, a routing destination may be added via elements 1101 and 1102B. In this example, the route for which the routing source or destination is added is indicated via elements 1103 and 1104. According to this example, a person may be added as a source or destination via element 1105A. In this example, a service may be added as a source or destination via element 1105B. According to this example, a location may be added as a source or destination via element 1105C. Element 1106 indicates the end of the process for adding one or more sources or destinations.

[0467] Figure 11BA process 1100B for removing a source or destination is shown. In some instances, one or more sources or destinations can be removed. In this example, a routing source can be removed via elements 1107 and 1108A. According to this example, a routing destination can be removed via elements 1107 and 1108B. In this example, the routing for which a routing source or destination is removed is indicated via elements 1109 and 1110. According to this example, a person can be removed as a source or destination via element 1111A. In this example, a service can be removed as a source or destination via element 1111B. According to this example, a location can be removed as a source or destination via element 1111C. Element 1112 indicates the end of the process for removing one or more sources or destinations.

[0468] Figure 11C The process for deleting a route is shown. Here, element 1113 indicates deletion. The route ID specified via element 1114 indicates the route to be deleted. Element 1115 indicates the end of the process for removing one or more sources or destinations.

[0469] Figure 12 is a flow diagram including blocks of an audio session management method according to some embodiments. According to this example, method 1200 is an audio session management method for an audio environment having multiple audio devices. As with other methods described herein, the blocks of method 1200 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 1200 may be performed simultaneously. In addition, some embodiments of method 1200 may include more or fewer blocks than shown and / or described. The blocks of method 1200 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6 The control system 610 shown and described below, or one of the other disclosed control system examples.

[0470] In this example, block 1205 involves receiving a first route initiation request from a first device implementing a first application and implemented by an audio session manager (e.g., CHASM) to initiate a first route for a first audio session. According to this example, the first route initiation request indicates a first audio source and a first audio environment destination. Here, the first audio environment destination corresponds to at least a first person in the audio environment. However, in this example, the first audio environment destination does not indicate an audio device.

[0471] According to some examples, the first routing initiation request may indicate at least a first area of ​​the audio environment as a first routing source or a first routing destination. In some instances, the first routing initiation request may indicate at least a first service as a first audio source.

[0472] In this embodiment, block 1210 involves establishing, by a device implementing an audio session manager, a first route corresponding to a first route initiation request. In this example, establishing the first route involves determining a first location of at least a first person in an audio environment, determining at least one audio device for a first phase of a first audio session, and initiating or scheduling the first audio session.

[0473] According to some examples, the first routing initiation request may include a first audio session priority. In some instances, the first routing initiation request may include a first connection mode. The first connection mode may be, for example, a synchronous connection mode, a transactional connection mode, or a scheduled connection mode. In some examples, the first routing initiation request may indicate more than one connection mode.

[0474] In some embodiments, the first routing initiation request may include an indication of whether confirmation from at least the first person is required. In some examples, the first routing initiation request may include a first audio session objective. The first audio session objective may include, for example, intelligibility, audio quality, spatial fidelity, and / or inaudibility.

[0475] As described elsewhere herein, in some embodiments, a route can have an associated audio session identifier, which in some embodiments can be a persistently unique audio session identifier. Thus, some embodiments of method 1200 can involve (e.g., by an audio session manager) determining a first persistently unique audio session identifier for a first route and transmitting the first persistently unique audio session identifier to a first device (a device executing a first application).

[0476] In some embodiments, establishing the first route may involve causing at least one device in the environment to establish at least a first media stream corresponding to the first route, the first media stream including a first audio signal. Some embodiments of method 1200 may involve causing the first audio signal to be rendered as a first rendered audio signal. In some examples, method 1200 may involve the audio session manager causing another device in the audio environment to render the first audio signal as the first rendered audio signal. However, in some embodiments, the audio session manager may be configured to receive the first audio signal and render the first audio signal as the first rendered audio signal.

[0477] As described elsewhere herein, in some embodiments, an audio session manager (e.g., CHASM) can monitor conditions of the audio environment, such as the location and / or orientation of one or more people in the audio environment, the location of audio devices in the audio environment, etc. For example, for a "don't wake the baby" use case, an audio session manager (e.g., Figure 7The optimizer 702 of the audio session manager can determine or at least estimate where the baby is. The audio session manager can know where the baby is due to an explicit statement from the user transmitted in "scripted language" from the associated application (e.g., "Don't wake the baby. The baby is in bedroom 1."). Alternatively or additionally, the audio session manager can determine where the baby is based on previous explicit input or based on previous detection of the baby's cry (e.g., as described below). In some examples, the audio session manager can receive this constraint (e.g., via an "inaudibility" audio session objective) and can enforce the constraint, for example, by ensuring that the sound pressure level at the baby's location is less than a threshold decibel level (e.g., 50 dB).

[0478] Some examples of method 1200 may involve determining a first orientation of a first person for a first phase of an audio conversation. According to some such examples, rendering a first audio signal as a first rendered audio signal may involve determining a first reference spatial pattern corresponding to a first position and a first orientation of the first person, and determining a first relative activation of loudspeakers in an audio environment corresponding to the first reference spatial pattern. Some detailed examples are described below.

[0479] In some instances, the audio session manager can determine that the first person has changed position and / or orientation. Some examples of method 1200 can involve determining at least one of a second position or a second orientation of the first person, determining a second reference spatial pattern corresponding to at least one of the second position or the second orientation, and determining a second relative activation of a loudspeaker in the audio environment corresponding to the second reference spatial pattern.

[0480] As described elsewhere in this disclosure, in some instances, the audio manager may be tasked with establishing and implementing more than one route at a time. Some examples of method 1200 may involve receiving, from a second device implementing a second application and a device implementing the audio session manager, a second route initiation request for initiating a second route for a second audio session. The second route initiation request may indicate a second audio source and a second audio environment destination. In some examples, the second audio environment destination may correspond to at least a second person in the audio environment. However, in some instances, the second audio environment destination may not indicate any specific audio device associated with the second route.

[0481] Some such examples of method 1200 can involve, by a device implementing the audio session manager, establishing a second route corresponding to a second route initiation request. In some instances, establishing the second route can involve determining a first location of at least a second person in the audio environment, determining at least one audio device for a first phase of a second audio session, and initiating the second audio session.

[0482] According to some examples, establishing the second route can involve establishing at least a second media stream corresponding to the second route. The second media stream can include a second audio signal. Some such examples of method 1200 can involve causing the second audio signal to be rendered as a second rendered audio signal.

[0483] Some examples of method 1200 may involve modifying a rendering process for a first audio signal based at least in part on at least one of a second audio signal, a second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. For example, modifying the rendering process for the first audio signal may involve shifting the rendering of the first audio signal away from a rendering position of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may involve modifying the loudness of one or more of the first rendered audio signal in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

[0484] Figure 13 is a flow diagram including blocks of an audio session management method according to some embodiments. According to this example, method 1300 is an audio session management method for an audio environment having multiple audio devices. As with other methods described herein, the blocks of method 1300 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 1300 may be performed simultaneously. In addition, some embodiments of method 1300 may include more or fewer blocks than shown and / or described. The blocks of method 1300 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6 The control system 610 shown and described below, or one of the other disclosed control system examples.

[0485] In this example, block 1305 involves receiving a first route initiation request from a first device implementing a first application and implemented by an audio session manager (e.g., CHASM) to initiate a first route for a first audio session. According to this example, the first route initiation request indicates a first audio source and a first audio environment destination. Here, the first audio environment destination corresponds to at least a first zone of an audio environment. However, in this example, the first audio environment destination does not indicate an audio device.

[0486] According to some examples, the first routing initiation request may indicate at least a first person in the audio environment as a first routing source or a first routing destination. In some instances, the first routing initiation request may indicate at least a first service as a first audio source.

[0487] In this embodiment, block 1310 involves establishing, by a device implementing an audio session manager, a first route corresponding to a first route initiation request. In this example, establishing the first route involves determining at least one audio device in a first zone of an audio environment for a first phase of a first audio session, and initiating or scheduling the first audio session.

[0488] According to some examples, the first routing initiation request may include a first audio session priority. In some instances, the first routing initiation request may include a first connection mode. The first connection mode may be, for example, a synchronous connection mode, a transactional connection mode, or a scheduled connection mode. In some examples, the first routing initiation request may indicate more than one connection mode.

[0489] In some embodiments, the first routing initiation request may include an indication of whether confirmation from at least the first person is required. In some examples, the first routing initiation request may include a first audio session objective. The first audio session objective may include, for example, intelligibility, audio quality, spatial fidelity, and / or inaudibility.

[0490] Some implementations of method 1300 may involve determining (eg, by an audio session manager) a first persistently unique audio session identifier for a first route, and transmitting the first persistently unique audio session identifier to a first device (a device executing a first application).

[0491] In some embodiments, establishing the first route may involve causing at least one device in the environment to establish at least a first media stream corresponding to the first route, the first media stream including a first audio signal. Some embodiments of method 1300 may involve causing the first audio signal to be rendered as a first rendered audio signal. In some examples, method 1300 may involve the audio session manager causing another device in the audio environment to render the first audio signal as the first rendered audio signal. However, in some embodiments, the audio session manager may be configured to receive the first audio signal and render the first audio signal as the first rendered audio signal.

[0492] As described elsewhere herein, in some implementations, an audio session manager (eg, CHASM) can monitor conditions of an audio environment, such as the location of one or more audio devices in the audio environment.

[0493] Some examples of method 1300 may involve performing a first loudspeaker automatic positioning process to automatically determine a first position of each audio device among a plurality of audio devices in a first area of ​​an audio environment at a first time. In some such examples, the rendering process may be based at least in part on the first position of each audio device. Some such examples may involve storing the first position of each audio device in a data structure associated with the first route.

[0494] In some instances, the audio session manager may determine that at least one audio device in the first zone has a changed position. Some such examples may involve executing a second loudspeaker automatic positioning process that automatically determines the changed position, and updating a rendering process based at least in part on the changed position. Some such embodiments may involve storing the changed position in a data structure associated with the first route.

[0495] In some instances, the audio session manager may determine that at least one additional audio device has been moved to the first area. Some such examples may involve performing a second loudspeaker automatic positioning process to automatically determine an additional audio device position of the additional audio device, and updating a rendering process based at least in part on the additional audio device position. Some such embodiments may involve storing the additional audio device position in a data structure associated with the first route.

[0496] As described elsewhere herein, in some examples, the first routing initiation request may indicate at least the first person as the first routing source or the first routing destination. Some examples of method 1300 may involve determining a first orientation of the first person for a first phase of the audio session. According to some such examples, causing the first audio signal to be rendered as a first rendered audio signal may involve determining a first reference spatial pattern corresponding to a first position and a first orientation of the first person, and determining a first relative activation of a loudspeaker in the audio environment corresponding to the first reference spatial pattern. Some detailed examples are described below.

[0497] In some instances, the audio session manager can determine that the first person has changed position and / or orientation. Some examples of method 1300 can involve determining at least one of a second position or a second orientation of the first person, determining a second reference spatial pattern corresponding to at least one of the second position or the second orientation, and determining a second relative activation of a loudspeaker in the audio environment corresponding to the second reference spatial pattern.

[0498] As described elsewhere in this disclosure, in some instances, the audio manager may be tasked with establishing and implementing more than one route at a time. Some examples of method 1300 may involve receiving, from a second device implementing a second application and a device implementing the audio session manager, a second route initiation request to initiate a second route for a second audio session. The second route initiation request may indicate a second audio source and a second audio environment destination. In some examples, the second audio environment destination may correspond to at least a second person in the audio environment. However, in some instances, the second audio environment destination may not indicate any specific audio device associated with the second route.

[0499] Some such examples of method 1300 can involve, by a device implementing the audio session manager, establishing a second route corresponding to a second route initiation request. In some instances, establishing the second route can involve determining a first location of at least a second person in the audio environment, determining at least one audio device for a first phase of a second audio session, and initiating the second audio session.

[0500] According to some examples, establishing the second route can involve establishing at least a second media stream corresponding to the second route. The second media stream can include a second audio signal. Some such examples of method 1300 can involve causing the second audio signal to be rendered as a second rendered audio signal.

[0501] Some examples of method 1300 may involve modifying a rendering process for a first audio signal based at least in part on at least one of a second audio signal, a second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. Modifying the rendering process for the first audio signal may, for example, involve shifting the rendering of the first audio signal away from a rendering position of the second rendered audio signal. Alternatively or additionally, modifying the rendering process for the first audio signal may involve modifying the loudness of one or more of the first rendered audio signal in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

[0502] Figure 14 is a flow diagram including blocks of an audio session management method according to some embodiments. According to this example, method 1400 is an audio session management method for an audio environment having multiple audio devices. As with other methods described herein, the blocks of method 1400 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 1400 may be performed simultaneously. In addition, some embodiments of method 1400 may include more or fewer blocks than shown and / or described. The blocks of method 1400 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6The control system 610 shown and described below, or one of the other disclosed control system examples.

[0503] In this example, in block 1405, Figure 4 The application 410 uses an orchestration language to instruct the CHASM 401. Block 1405 may involve, for example, the application 410 sending a route initiation request or a route modification request to the CHASM 401.

[0504] According to this example, CHASM 401 determines optimal media engine control information in response to instructions received from application 410. In this example, the optimal media engine control information is based at least in part on the audio session priority indicated in the instructions from application 410, the listener's location within the audio environment, and the availability of audio devices within the audio environment. In some instances, the optimal media engine control information can be based at least in part on media engine capabilities determined by CHASM 401, for example, via device property descriptors shared by one or more related audio devices. According to some examples, the optimal media engine control information can be based at least in part on the listener's orientation.

[0505] In this example, block 1415 involves sending control information to one or more audio device media engines. The control information may be the same as described above with reference to Figure 5 The audio session management control signal described in

[15] corresponds to the audio session management control signal described in

[15] .

[0506] According to this example, block 1420 represents CHASM 401 monitoring conditions within the audio environment, and possible further communications from application 410 regarding this particular route, to determine whether there have been any significant changes, such as a change in route priority, a change in the location of one or more audio devices, a change in the location of the listener, etc. If so, the process returns to block 1410 and the process of block 1410 is performed based on the one or more new parameters. If not, CHASM 401 continues the monitoring process of block 1420.

[0507] Figure 15 is a flow chart including blocks for an automatic setup process for one or more audio devices that are newly introduced to an audio environment, in accordance with some embodiments. In this example, some or all of the audio devices are new. As with other methods described herein, the blocks of method 1500 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 1500 may be performed simultaneously. Furthermore, some embodiments of method 1500 may include more or fewer blocks than shown and / or described. The blocks of method 1500 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6The control system 610 shown and described below, or one of the other disclosed control system examples.

[0508] In this example, the new audio devices are turned on and powered on in block 1505. In the example of block 1510, each of the new audio devices enters discovery mode to search for other audio devices, and specifically, to search the audio environment for CHASM. If an existing CHASM is found, the new audio device can be configured to communicate with the CHASM to share information about the capabilities of each new audio device, etc. with the CHASM.

[0509] However, according to this example, no existing CHASM is found. Therefore, in this example of block 1510, one of the new audio devices configures itself as a CHASM. In this example, the new audio device with the most available computing power and / or the greatest connectivity configures itself as the new CHASM 401.

[0510] In this example, in block 1515, each of the new non-CHASM audio devices communicates with another new audio device that is the newly designated CHASM 401. According to this example, the new CHASM 401 launches the "Settings" application, which in this example is Figure 4 In this example, the setup application 412 is configured to interact with the user, such as via audio and / or visual prompts, to guide the user through the setup process.

[0511] According to this example, in block 1520 , setup application 412 sends an instruction in an orchestration language to CHASM 401 indicating “set up all new devices” and having the highest priority level.

[0512] In this example, in block 1525, CHASM 401 interprets the instructions from the setup application 412 and determines that a new acoustic mapping calibration is required. According to this example, the acoustic mapping process is initiated in block 1525 and completed in block 1530 via communication between CHASM 401 and the media engine of the new non-CHASM audio device, which in this instance is Figure 4 441 and 442 of the media engines 440, 441, and 442. As used herein, the term "acoustic mapping" includes an estimation of all discoverable loudspeaker positions in an audio environment. The acoustic mapping process may, for example, involve an automatic loudspeaker positioning process, such as those described in detail below. In some instances, the acoustic mapping process may involve a process for discovering loudspeaker capability information and / or individual loudspeaker dynamic processing information.

[0513] According to this example, in block 1535, CHASM 401 sends a confirmation that the setup process has been completed to application 412. In this example, in block 1540, application 412 indicates to the user that the setup process has been completed.

[0514] Figure 16 is a flowchart including blocks of a process for installing a virtual assistant application according to some embodiments. In some instances, method 1600 may be performed after the setup process of method 1500. In this example, method 1600 involves installing a virtual assistant application in Figure 4 The virtual assistant application is installed in the context of the audio environment shown. As with other methods described herein, the blocks of method 1600 do not have to be executed in the order indicated. In some embodiments, one or more of the blocks of method 1600 can be executed simultaneously. In addition, some embodiments of method 1600 may include more or fewer blocks than shown and / or described. The blocks of method 1600 can be executed by one or more devices, which can be (or can include) a control system, such as Figure 6 The control system 610 shown and described below, or one of the other disclosed control system examples.

[0515] In this example, a new application 411 called "Virtual Assistant Liaison" or VAL is installed by a user in block 1605. According to some examples, block 1605 may involve downloading the application 411 from one or more servers to an audio device such as a cell phone via the Internet.

[0516] According to this embodiment, in block 1610, application 411 instructs CHASM 401 in orchestration language to continuously listen for the new wake word "Hey Val" with the highest priority and as a persistent audio session. In this example, in block 1615, CHASM 401 interprets the instruction from application 411 and instructs media engines 440, 441, and 442 to configure their wake word detectors to listen for the wake word "Hey Val" and to call back to CHASM 401 whenever the wake word "Hey Val" is detected. In this embodiment, in block 1620, media engines 440, 441, and 442 continue to listen for the wake word.

[0517] In this example, CHASM 401 receives a callback from media engines 440 and 441 indicating that the wake word "Hey Val" has been detected in block 1625. In response, CHASM 401 instructs media engines 440, 441, and 442 to listen for commands during a threshold time interval (5 seconds in this example) after the wake word is initially detected, and if a command is detected, to "duck" or lower the audio volume in the area where the command was detected.

[0518] According to this example, in block 1630, media engines 440, 441, and 442 each detect a command and send speech audio data and probabilities corresponding to the detected command to CHASM 401. In this example, in block 1630, CHASM 401 forwards the speech audio data and probabilities corresponding to the detected command to application 411.

[0519] In this embodiment, in block 1635, application 411 receives speech audio data and probabilities corresponding to the detected command and forwards this data to the cloud-based speech recognition application for processing. In this example, in block 1635, the cloud-based speech recognition application sends the results of the speech recognition process to application 411, which in this example include one or more words corresponding to the command. Here, in block 1635, application 411 instructs CHASM 401 in the orchestration language to end the speech recognition session. According to this example, CHASM 401 instructs media engines 440, 441, and 442 to stop listening for commands.

[0520] Figure 17 is a flow chart including blocks of an audio session management method according to some embodiments. According to this example, method 1700 is for Figure 4 In some examples, method 1700 may be performed after the setup process of method 1500. In some examples, method 1700 may be performed after the setup process of method 1500. Figure 16 The method 1700 may be performed before or after the process for installing the virtual assistant application described herein. As with other methods described herein, the blocks of method 1700 do not have to be performed in the order indicated. In some embodiments, one or more of the blocks of method 1700 may be performed simultaneously. In addition, some embodiments of method 1700 may include more or fewer blocks than shown and / or described. The blocks of method 1700 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6 The control system 610 shown and described below, or one of the other disclosed control system examples.

[0521] In this example, in block 1705, a user provides input to a music application running on a device in an audio environment. In this instance, the music application is Figure 4 According to this example, the application 410 is running on the smartphone and provides input via the smartphone's user interface (such as a touch and / or gesture sensor system).

[0522] According to this example, in block 1710, application 410 instructs CHASM 401 via a route initiation request in this example in an orchestration language to initiate routing from the cloud-based music service to a user interacting with application 410 via a smartphone. In this example, the route initiation request indicates a synchronization mode and an audio session target with the highest music reproduction quality, does not request confirmation and has a priority of 4, using the user's current favorite playlist of the cloud-based music service.

[0523] In this example, in block 1715, CHASM 401 determines which audio devices of the audio environment will be involved in the routing based on the instructions received in block 1710. This determination can be based at least in part on a previously determined acoustic map of the audio environment, which audio devices are currently available, the capabilities of the available audio devices, and the estimated current location of the user. In some examples, the determination in block 1715 can be based at least in part on the estimated current orientation of the user. In some embodiments, a nominal or initial listening level can also be selected in block 1715. This level can be based at least in part on the estimated proximity of the user to one or more audio devices, the ambient noise level in the user's area, etc.

[0524] According to this example, in block 1720, CHASM 401 sends control information to the selected audio device media engine, which in this example is media engine 441, to obtain a media bitstream corresponding to the route requested by application 410. In this example, CHASM 401 provides the HTTP address of the cloud-based music provider, e.g., the HTTP address of a particular server hosted by the cloud-based music provider, to media engine 441. According to this embodiment, in block 1725, media engine 441 obtains the media bitstream from the cloud-based music provider (in this example, from one or more allocated server locations).

[0525] In this example, block 1730 involves playing back music corresponding to the media stream obtained in block 1725. According to this example, CHASM 401 has determined that at least loudspeaker 461, and in some examples, loudspeaker 460 and / or loudspeaker 462, are involved in the playback of the music. In some such examples, CHASM 401 has provided instructions to media engine 441 to render audio data from the media stream and to provide rendered speaker feed signals to media engine 440 and / or media engine 442.

[0526] Figure 18Ais a block diagram of a minimal version of an embodiment. N program streams (N≥2) are depicted, the first of which is explicitly marked as a space, whose corresponding set of audio signals is fed through a corresponding renderer, each renderer being individually configured to play back its corresponding program stream through a common set of M arbitrarily spaced loudspeakers (M≥2). A renderer may also be referred to herein as a "rendering module". The rendering module and mixer 1830a may be implemented via software, hardware, firmware, or some combination thereof. In this example, the rendering module and mixer 1830a are implemented via the control system 610a described above with reference to Figure 6 According to some embodiments, the functionality of the rendering module and mixer 1830a can be implemented at least in part based on instructions from a device implementing what is referred to herein as an audio session manager (e.g., CHASM). In some such examples, the functionality of the rendering module and mixer 1830a can be implemented at least in part based on instructions from the device implementing what is referred to herein as an audio session manager (e.g., CHASM). Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 1830a.

[0527] exist Figure 18AIn the example shown in , each of the N renderers outputs a set of M loudspeaker feeds summed across all N renderers for simultaneous playback on the M loudspeakers. According to this embodiment, information about the layout of the M loudspeakers within the listening environment is provided to all renderers (indicated by the dotted line fed back from the loudspeaker frame) so that the renderers can be appropriately configured for playback through the speakers. Depending on the specific embodiment, this layout information may or may not be sent from one or more of the speakers themselves. According to some examples, the layout information may be provided by one or more smart speakers configured to determine the relative position of each of the M loudspeakers in the listening environment. Some such automatic positioning methods may be based on a direction of arrival (DOA) method or a time of arrival (TOA) method. In other examples, the layout information may be determined by another device and / or input by a user. In some examples, loudspeaker specification information about the capabilities of at least some of the M loudspeakers within the listening environment may be provided to all renderers. Such loudspeaker specification information may include impedance, frequency response, sensitivity, power rating, number and location of individual drivers, etc. According to this example, information from the rendering of one or more of the additional program streams is fed into the renderer of the primary spatial stream so that the rendering can be dynamically modified based on the information. This information is represented by the dotted line from rendering block 2 to rendering block N back to rendering block 1.

[0528] Figure 18B Another (more capable) embodiment with additional features is depicted. In this example, the rendering module and mixer 1830b are implemented via the control system 610b, which is the control system described above with reference to Figure 6 According to some embodiments, the functionality of the rendering module and mixer 1830b may be implemented at least in part based on instructions from a device implementing what is referred to herein as an audio session manager (e.g., CHASM). In some such examples, the functionality of the rendering module and mixer 1830b may be implemented at least in part based on instructions from the device implementing what is referred to herein as an audio session manager (e.g., CHASM). Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 1830b.

[0529] exist Figure 18BIn the figure, the dashed lines running up and down between all N renderers represent the idea that any one of the N renderers can contribute to the dynamic modification of any one of the remaining N-1 renderers. In other words, the rendering of any one of the N program streams can be dynamically modified based on a combination of one or more renderings of any of the remaining N-1 program streams. Additionally, any one or more of the program streams can be spatial mixes, and the rendering of any program stream (whether or not it is spatial) can be dynamically modified based on any of the other program streams. For example, as described above, loudspeaker layout information can be provided to the N renderers. In some examples, loudspeaker specification information can be provided to the N renderers. In some embodiments, the microphone system 620a can include a set of K microphones (K≥1) within the listening environment. In some examples, one or more microphones can be attached to or associated with one or more of the loudspeakers. These microphones can feed both their captured audio signals (represented by solid lines) and additional configuration information (e.g., their positions) (represented by dashed lines) back to the set of N renderers. Any of the N renderers can then be dynamically modified based on this additional microphone input. Various examples are provided in this article.

[0530] Examples of information derived from microphone input and subsequently used to dynamically modify any of the N renderers include, but are not limited to:

[0531] Detection of specific words or phrases spoken by the system user.

[0532] • An estimate of the location of one or more users of the system.

[0533] • An estimate of the loudness of any combination of N program streams at a particular location in the listening space.

[0534] An estimate of the loudness of other ambient sounds in the listening environment (such as background noise).

[0535] Figure 19 is an overview of the Figure 6 、 Figure 18A or Figure 18B 1900. As with other methods described herein, the blocks of method 1900 do not have to be performed in the order indicated. In addition, this method may include more or fewer blocks than shown and / or described. The blocks of method 1900 may be performed by one or more devices, which may be (or may include) a control system, such as Figure 6 、 Figure 18A and Figure 18B10, control system 610a, or control system 610b shown and described above, or one of the other disclosed control system examples. In some embodiments, the blocks of method 1900 may be performed at least in part based on instructions from a device implementing what is referred to herein as an audio session manager (e.g., CHASM). In some such examples, the blocks of method 1900 may be performed at least in part based on instructions from the device described above with reference to Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 1900. In an alternative example, a device implementing an audio session manager may also implement the blocks of method 1900.

[0536] In this embodiment, block 1905 involves receiving a first audio program stream via the interface system. In this example, the first audio program stream includes a first audio signal scheduled to be reproduced by at least some speakers of the environment. Here, the first audio program stream includes first spatial data. According to this example, the first spatial data includes channel data and / or spatial metadata. In some examples, block 1905 involves a first rendering module of the control system receiving the first audio program stream via the interface system.

[0537] According to this example, block 1910 involves rendering a first audio signal for reproduction via a speaker of an environment, thereby generating a first rendered audio signal. For example, as described above, some examples of method 1900 involve receiving loudspeaker layout information. For example, as described above, some examples of method 1900 involve receiving loudspeaker specification information. In some examples, the first rendering module may generate the first rendered audio signal based at least in part on the loudspeaker layout information and / or the loudspeaker specification information.

[0538] In this example, block 1915 involves receiving a second audio program stream via the interface system. In this embodiment, the second audio program stream includes a second audio signal scheduled to be reproduced by at least some speakers of the environment. According to this example, the second audio program stream includes second spatial data. The second spatial data includes channel data and / or spatial metadata. In some examples, block 1915 involves a second rendering module of the control system receiving the second audio program stream via the interface system.

[0539] Depending on the embodiment, block 1920 involves rendering the second audio signal for reproduction via a speaker of the environment, thereby generating a second rendered audio signal. In some examples, the second rendering module may generate the second rendered audio signal based at least in part on the received loudspeaker layout information and / or the received loudspeaker specification information.

[0540] In some instances, some or all of the speakers in an environment can be positioned arbitrarily. For example, at least some of the speakers of an environment can be placed in locations that do not correspond to any standard-mandated speaker layout (e.g., Dolby 5.1, Dolby 7.1, Hamasaki 22.2, etc.). In some such examples, at least some of the speakers of an environment can be placed in convenient locations relative to furniture, walls, etc. of the environment (e.g., where there is space to accommodate the speakers), but not in any standard-mandated speaker layout.

[0541] Thus, in some embodiments, box 1910 or box 1920 may involve flexible rendering to arbitrarily positioned speakers. Some such embodiments may involve center of mass amplitude translation (CMAP), flexible virtualization (FV), or a combination of the two. From a high level, these two techniques render a set of one or more audio signals (each audio signal has an associated desired perceptual spatial position) for playback through a set of two or more speakers, where the relative activation of the set of speakers is a function of a model of the perceptual spatial position of the audio signal played back by the speakers and the proximity of the desired perceptual spatial position of the audio signal to the positions of the speakers. The model ensures that the listener hears the audio signal near its expected spatial position, and the proximity term controls which speakers are used to achieve this spatial impression. Specifically, the proximity term favors activating speakers that are close to the desired perceptual spatial position of the audio signal. For both CMAP and FV, this functional relationship can be conveniently derived from a cost function, which is written as the sum of two terms, one for spatial aspects and one for proximity:

[0542]

[0543] Here, gather represents the positions of a set of M loudspeakers, denotes the desired perceptual spatial location of the audio signal, and g denotes the M-dimensional vector of loudspeaker activations. For CMAP, each activation in the vector represents a gain for each loudspeaker, while for FV, each activation represents a filter (in the second case, g can equivalently be viewed as a vector of complex values ​​at a particular frequency, and different g are computed across multiple frequencies to form the filter). The optimal vector of activations is found by minimizing the cost function across the activations:

[0544]

[0545] Under certain definitions of the cost function, although g opt The relative levels between the components of g are appropriate, but it is difficult to control the absolute level of the optimal activation produced by the above minimization. To solve this problem, one can perform optSubsequent normalization of in order to control the absolute level of activation. For example, it may be desirable to normalize the vector to have unit length, which conforms to the commonly used constant power shift rule:

[0546]

[0547] The exact behavior of the flexible rendering algorithm is determined by the two terms C in the cost function. spatial and C proximity For CMAP, C spatial is obtained from a model that places the perceptual spatial location of an audio signal played from a set of loudspeakers at the associated activation gains g generated by those loudspeakers. i The centroid of the weighted loudspeaker positions (elements of vector g):

[0548]

[0549] Equation 3 is then manipulated into a spatial cost representing the squared error between the desired audio position and the audio position produced by the activated loudspeakers:

[0550]

[0551] In the case of FV, the spatial term of the cost function is defined differently. The goal is to produce the same spatial information at the left and right ears as the audio object positions. The corresponding binaural response b. Conceptually, b is a 2×1 vector of filters (one filter for each ear), but it is more convenient to think of it as a 2×1 vector of complex values ​​at a specific frequency. Continuing with this representation at a specific frequency, the desired binaural response can be obtained from a set of HRTF indices at the object position:

[0552]

[0553] Meanwhile, the 2×1 binaural response e produced by the loudspeaker at the listener’s ears is modeled as a 2×M acoustic transfer matrix H multiplied by an M×1 vector g of complex loudspeaker activation values:

[0554] e=Hg (6)

[0555] The acoustic transmission matrix H is based on the set of loudspeaker positions Finally, the spatial component of the cost function is defined as the squared error between the desired binaural response (Equation 5) and the binaural response produced by the loudspeakers (Equation 6):

[0556]

[0557] Conveniently, both the spatial terms of the cost functions for CMAP and FV defined in Equations 4 and 7 can be rearranged as matrix quadratic functions as a function of the speaker activation g:

[0558]

[0559] Where A is an M×M square matrix, B is a 1×M vector, and C is a scalar. The rank of the matrix A is 2, and therefore when M>2, there are an infinite number of speaker activations g for which the spatial error term is equal to zero. The second term C of the cost function is introduced pr o ximity This uncertainty is eliminated and a specific solution is produced that has perceptually beneficial properties compared to other possible solutions. For both CMAP and FV, C pr o ximity is constructed so that its location Far from the desired audio signal location The activation of speakers located close to the desired location is penalized more than the activation of speakers located close to the desired location. This construction produces a sparse optimal set of speaker activations, where only speakers close to the location of the desired audio signal are significantly activated, and in fact leads to a spatial reproduction of the audio signal that is perceptually more robust to listener movement around the set of speakers.

[0560] To this end, the second term C of the cost function pr o ximity can be defined as the distance-weighted sum of the squared absolute values ​​of the speaker activations. This is concisely expressed in matrix form as:

[0561]

[0562] where D is the diagonal matrix of distance penalties between the desired audio position and each speaker:

[0563]

[0564] The distance penalty function can take many forms, but the following is a useful parameterization

[0565]

[0566] in, is the Euclidean distance between the desired audio position and the loudspeaker position, and α and β are adjustable parameters. The parameter α indicates the global strength of the penalty; d0 corresponds to the spatial extent of the distance penalty (loudspeakers at approximately d0 distance or further will be penalized), and β describes the abruptness of the onset of the penalty at distance d0.

[0567] Combining the two terms of the cost functions defined in Equation 8 and Equation 9a yields the overall cost function

[0568] C(g)=g * Ag+Bg+C+g * Dg=g * (A+D)g+Bg+C (10)

[0569] Setting the derivative of this cost function with respect to g equal to zero and solving for g yields the optimal speaker activation solution:

[0570]

[0571] Typically, the optimal solution in Equation 11 may produce speaker activations with negative values. For CMAP construction of flexible renderers, such negative activations may not be desired, so Equation (11) can be minimized while keeping all activations positive.

[0572] Pairing a flexible rendering approach (implemented in accordance with some embodiments) with a group of wireless smart speakers (or other smart audio devices) can result in a very capable and easy to use spatial audio rendering system. When considering interactions with such a system, it becomes apparent that dynamic modification of the spatial rendering may be desirable in order to optimize for other objectives that may arise during use of the system. To achieve this goal, one class of embodiments enhances an existing flexible rendering algorithm (in which speaker activation is a function of the previously disclosed spatial terms and proximity terms) with one or more additional dynamically configurable functions that depend on one or more properties of the audio signal being rendered, the speaker group, and / or other external inputs. According to some embodiments, the cost function for the existing flexible rendering given in Equation 1 is augmented with these one or more additional dependencies according to the following equation

[0573]

[0574] In Equation 12, the term represents the additional cost item, where A set of one or more properties representing an audio signal being rendered (e.g., an object-based audio program), a set of one or more properties representing the speakers on which the audio is being rendered, and Represents one or more additional external inputs. Each Returns the cost as a function of one or more properties of the audio signal, the loudspeaker, and / or the external input (generally represented by the set It should be understood that the set At least from or An element of any one of .

[0575] Examples include, but are not limited to:

[0576] The desired perceived spatial location of the audio signal;

[0577] the level of the audio signal (which may vary over time); and / or

[0578] The frequency spectrum of the audio signal (possibly varying over time).

[0579] Examples include, but are not limited to:

[0580] The location of the loudspeakers in the listening space;

[0581] The frequency response of the loudspeaker;

[0582] ·Playback level limit for loudspeakers;

[0583] Parameters of the dynamics processing algorithms within the loudspeaker, such as limiter gain;

[0584] A measurement or estimate of the acoustic transmission from each loudspeaker to the other loudspeakers;

[0585] Measurement of the performance of an echo canceller on a loudspeaker; and / or

[0586] • The relative synchronization of the loudspeakers with respect to each other.

[0587] Examples include, but are not limited to:

[0588] The location of one or more listeners or speakers in the playback space;

[0589] A measurement or estimate of the acoustic transmission from each loudspeaker to the listening position;

[0590] The measurement or estimation of the acoustic transmission from a speaker to a set of loudspeakers;

[0591] The location of some other landmark in the replay space; and / or

[0592] A measurement or estimate of the acoustic transmission from each loudspeaker to some other landmark in the playback space;

[0593] With the new cost function defined in Equation 12, the optimal group activation can be found by minimization with respect to g as previously specified in Equations 2a and 2b and possible post-normalization.

[0594] Similar to the proximity cost defined in Equations 9a and 9b, the new cost function term It is also convenient to express each of as a weighted sum of the squared absolute values ​​of the speaker activations:

[0595]

[0596] Among them, W j is a weight describing the cost associated with activating speaker i for item j The diagonal matrix of :

[0597]

[0598] Combining Equations 13a and 13b with the matrix quadratic versions of the CMAP and FV cost functions given in Equation 10 yields a potentially beneficial implementation of the generalized extended cost function (of some embodiments) given in Equation 12:

[0599] C(g)=g * Ag+Bg+C+g * Dg+∑ j g * W j g=g * (A+D+∑ j W j )g+Bg+C (14)

[0601] Under this definition of the new cost function term, the overall cost function is still a matrix quadratic equation and can be expressed by

[0602] Differentiation of Equation 14 finds the optimal group activation gopt to produce

[0603]

[0604] The weight term w ij Each of the loudspeakers is considered as a given continuous penalty value for each loudspeaker in In one example embodiment, the penalty value is the distance from the object (to be rendered) to the loudspeaker under consideration. In another example embodiment, the penalty value indicates that a given loudspeaker cannot reproduce some frequencies. Based on the penalty value, the weight term w ij Can be parameterized as:

[0605]

[0606] Among them, α j represents the prefactor (which takes into account the global strength of the weight term), where τ j represents the penalty threshold (about or above which the weight term becomes significant), and where fj (x) represents a monotonically increasing function. For example, In the case of , the weight term has the following form:

[0607]

[0608] Among them, α j , β j , τ j are adjustable parameters that indicate the global strength of the penalty, the abruptness of the onset of the penalty, and the range of the penalty, respectively. Care should be taken when setting these adjustable values ​​so that the cost term C j Relative to any other additional cost items and C spatial and C proximity The relative influence of α is appropriate to achieve the desired outcome. For example, as a rule of thumb, if a particular penalty is expected to significantly dominate other penalties, its strength α should be set to j A setting of approximately ten times the next largest penalty strength may be appropriate.

[0609] In the case where all loudspeakers are penalized, it is often convenient to subtract the minimum penalty from all weight terms in post-processing so that at least one of the loudspeakers is not penalized:

[0610] w ij →w′ ij =w ij -min i (w ij ) (18)

[0611] As described above, many possible use cases can be implemented using the new cost function terms described herein (and similar new cost function terms employed according to other embodiments). Next, more specific details are described using three examples: moving audio toward a listener or speaker, moving audio away from a listener or speaker, and moving audio away from a landmark.

[0612] In the first example, what is referred to herein as "attractive force" is used to pull the audio toward a certain location, which in some examples can be the location of the listener or speaker, a landmark, a piece of furniture, etc. This location can be referred to herein as an "attractive location" or "attractor location." As used herein, "attractive force" is a factor that favors the activation of relatively higher loudspeakers closer to the attractive location. According to this example, the weight w ij It takes the form of Equation 17, where the continuous penalty value p ij The distance between the i-th speaker and the fixed attractor The distance is given, and the threshold τ j is given by the maximum of these distances for all loudspeakers:

[0613]

[0614]

[0615] To illustrate the use case of "pulling" the audio towards the listener or speaker, α is specifically set to j =20,β j =3, and Set to the vector corresponding to the listener / speaker position of 180 degrees. j , β j and These values ​​of are examples only. In other embodiments, j can be in the range of 1 to 100 and β j Can be in the range of 1 to 25.

[0616] In a second example, a "repelling force" is used to "push" the audio away from a certain location, which can be the listener's location, the speaker's location, or other locations such as landmarks, furniture, etc. This location may be referred to herein as a "repelling force location" or "repelling location." As used herein, a "repelling force" is a factor that favors the activation of relatively lower loudspeakers closer to the repelling force location. According to this example, relative to a fixed repelling location, the repelling force is a factor that favors the activation of relatively lower loudspeakers closer to the repelling force location. Define p ij and τ j , similar to the attraction in Equation 19:

[0617]

[0618]

[0619] To illustrate the use case of "pushing" the audio away from the listener or speaker, α is specifically set to j =5,β j =2, and Set to the vector corresponding to the listener / speaker position of 180 degrees. j , β j and These values ​​are examples only.

[0620] Return now Figure 19In this example, block 1925 involves modifying a rendering process for the first audio signal based at least in part on at least one of the second audio signal, the second rendered audio signal, or characteristics thereof to produce a modified first rendered audio signal. Various examples of modifying a rendering process are disclosed herein. For example, “characteristics” of a rendered signal may include estimated or measured loudness or audibility at an intended listening position, whether in silence or in the presence of one or more additional rendered signals. Other examples of characteristics include parameters associated with the rendering of the signal, such as the intended spatial location of the constituent signals of the associated program stream, the location of the loudspeakers on which the signals are rendered, the relative activation of the loudspeakers according to the intended spatial location of the constituent signals, and any other parameters or states associated with a rendering algorithm used to generate the rendered signal. In some examples, block 1925 may be performed by a first rendering module.

[0621] According to this example, block 1930 involves modifying a rendering process for a second audio signal based at least in part on the first audio signal, the first rendered audio signal, or at least one of its characteristics to produce a modified second rendered audio signal. In some examples, block 1930 can be performed by a second rendering module.

[0622] In some embodiments, modifying the rendering process for the first audio signal can involve offsetting the rendering of the first audio signal away from the rendering position of the second rendered audio signal and / or modifying the loudness of one or more of the first rendered audio signals in response to the loudness of the second audio signal or one or more of the second rendered audio signals. Alternatively or additionally, modifying the rendering process for the second audio signal can involve offsetting the rendering of the second audio signal away from the rendering position of the first rendered audio signal and / or modifying the loudness of one or more of the second rendered audio signals in response to the loudness of the first audio signal or one or more of the first rendered audio signals. Some examples are provided below with reference to Figure 3 and below.

[0623] However, other types of rendering process modifications are within the scope of the present disclosure. For example, in some instances, modifying the rendering process for the first audio signal or the second audio signal may involve performing spectral modifications, audibility-based modifications, or dynamic range modifications. Depending on the specific example, these modifications may be related to loudness-based rendering modifications or may be unrelated. For example, in the above-mentioned case of rendering a primary spatial stream in an open-plan living area and rendering a secondary stream including cooking cues in an adjacent kitchen, it may be desirable to ensure that the cooking cues remain audible in the kitchen. This can be accomplished by estimating the loudness of the rendered cooking cues stream in the kitchen without interference from the first signal, then estimating the loudness in the presence of the first signal in the kitchen, and finally dynamically modifying the loudness and dynamic range of the two streams across multiple frequencies to ensure audibility of the second signal in the kitchen.

[0624] exist Figure 19 In the example shown in FIG, block 1935 involves mixing at least the modified first rendered audio signal and the modified second rendered audio signal to produce a mixed audio signal. Block 1935 may, for example, be performed by Figure 18B The mixer 1830b shown in FIG.

[0625] According to this example, block 1940 involves providing the mixed audio signal to at least some speakers of the environment.Some examples of method 1900 involve playing back the mixed audio signal by speakers.

[0626] like Figure 19 As shown, some embodiments may provide more than two rendering modules. Some such embodiments may provide N rendering modules, where N is an integer greater than 2. Thus, some such embodiments may include one or more additional rendering modules. In some such examples, each of the one or more additional rendering modules may be configured to receive an additional audio program stream via the interface system. The additional audio program stream may include an additional audio signal scheduled for reproduction by at least one speaker of the environment. Some such embodiments may involve rendering the additional audio signal for reproduction via the at least one speaker of the environment to produce an additional rendered audio signal, and modifying the rendering process for the additional audio signal based at least in part on at least one of the first audio signal, the first rendered audio signal, the second audio signal, the second rendered audio signal, or characteristics thereof to produce a modified additional rendered audio signal. According to some such examples, the mixing module may be configured to mix the modified additional rendered audio signal with at least the modified first rendered audio signal and the modified second rendered audio signal to produce a mixed audio signal.

[0627] As mentioned above Figure 6 and Figure 18B As described, some embodiments may include a microphone system comprising one or more microphones in a listening environment. In some such examples, a first rendering module may be configured to modify a rendering process for a first audio signal based at least in part on a first microphone signal from the microphone system. Depending on the particular embodiment, the "first microphone signal" may be received from a single microphone or from two or more microphones. In some such embodiments, a second rendering module may be configured to modify a rendering process for a second audio signal based at least in part on the first microphone signal.

[0628] As mentioned above Figure 18B In some instances, the positions of one or more microphones may be known and provided to the control system. According to some such embodiments, the control system may be configured to estimate the first sound source position based on the first microphone signal and modify the rendering process for at least one of the first audio signal or the second audio signal based at least in part on the first sound source position. For example, the first sound source position may be estimated based on a triangulation process based on DOA data from each microphone in three or more microphones or microphone groups with known positions. Alternatively or additionally, the first sound source position may be estimated based on the amplitude of the signals received from two or more microphones. It may be assumed that the microphone that produces the highest amplitude signal is closest to the first sound source position. In some such examples, the first sound source position may be set to the position of the nearest microphone. In some such examples, the first sound source position may be associated with the position of a region, wherein the region is selected by processing the signals from the two or more microphones using a pre-trained classifier (such as a Gaussian mixture model).

[0629] In some such embodiments, the control system can be configured to determine whether the first microphone signal corresponds to ambient noise. Some such embodiments can involve modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to ambient noise. For example, if the control system determines that the first microphone signal corresponds to ambient noise, modifying the rendering process for the first audio signal or the second audio signal can involve increasing the level of the rendered audio signal so that the perceived loudness of the signal in the presence of noise at the intended listening position is substantially equal to the perceived loudness of the signal in the absence of noise.

[0630] In some examples, the control system can be configured to determine whether the first microphone signal corresponds to human speech. Some such embodiments may involve modifying a rendering process for at least one of the first audio signal or the second audio signal based at least in part on whether the first microphone signal corresponds to human speech. For example, if the control system determines that the first microphone signal corresponds to human speech (such as a wake-up word), modifying the rendering process for the first audio signal or the second audio signal may involve reducing the loudness of the rendered audio signal reproduced by a speaker near the first sound source position compared to the loudness of the rendered audio signal reproduced by a speaker away from the first sound source position. Alternatively or in addition, modifying the rendering process for the first audio signal or the second audio signal may involve modifying the rendering process to offset the expected position of the constituent signals of the associated program stream away from the first sound source position and / or penalize the use of speakers near the first sound source position compared to speakers away from the first sound source position.

[0631] In some embodiments, if the control system determines that the first microphone signal corresponds to human speech, the control system can be configured to reproduce the first microphone signal in one or more speakers near a location of the environment that is different from the location of the first sound source. In some such examples, the control system can be configured to determine whether the first microphone signal corresponds to the crying of a child. According to some such embodiments, the control system can be configured to reproduce the first microphone signal in one or more speakers near a location of the environment that corresponds to the estimated location of a caregiver (such as a parent, relative, guardian, childcare provider, teacher, nurse, etc.). In some examples, the process of estimating the estimated location of the caregiver can be triggered by a voice command such as "<wake word>, don't wake the baby". The control system will be able to estimate the location of the speaker (caregiver) through triangulation based on the DOA information provided by three or more local microphones, etc., based on the location of the nearest smart audio device that is implementing the virtual assistant. According to some embodiments, the control system will have prior knowledge of the location of the baby's room (and / or the listening device therein) and will then be able to perform appropriate processing.

[0632] According to some such examples, the control system can be configured to determine whether the first microphone signal corresponds to a command. In some instances, if the control system determines that the first microphone signal corresponds to a command, the control system can be configured to determine a reply to the command and control at least one speaker near the first sound source location to reproduce the reply. In some such examples, the control system can be configured to return to an unmodified rendering process for the first audio signal or the second audio signal after controlling the at least one speaker near the first sound source location to reproduce the reply.

[0633] In some embodiments, the control system can be configured to execute commands. For example, the control system can be or can include a virtual assistant configured to control audio equipment, televisions, home appliances, etc. according to commands.

[0634] exist Figure 6 、 Figure 18A and Figure 18B With this definition of a minimal and more capable multi-stream rendering system as shown in , dynamic management of simultaneous playback of multiple program streams can be achieved for many useful scenarios. Figure 20 and Figure 21 Describe multiple examples.

[0635] Let's first examine the example discussed previously involving simultaneous playback of a spatial movie soundtrack in a living room and cooking tips in a connected kitchen. The spatial movie soundtrack is an example of the "first audio program stream" mentioned above, and the cooking tips audio is an example of the "second audio program stream" mentioned above. Figure 20 and Figure 21 An example of a floor plan of a connected living space is shown. In this example, the living space 2000 includes a living room at the upper left, a kitchen at the lower center, and a bedroom at the lower right. The boxes and circles 2005a to 2005h distributed across the living space represent a set of eight loudspeakers placed in locations convenient for the space, but not adhering to any standard prescribed layout (arbitrary placement). Figure 20 In FIG, only the spatial movie soundtrack is played back, and taking into account the loudspeaker capabilities and layout, all loudspeakers in the living room 2010 and kitchen 2015 are utilized to create an optimized spatial reproduction around a listener 2020a sitting on a sofa 2025, facing a television 2030. This optimal reproduction of the movie soundtrack is visually represented by a cloud 2035a that lies within the range of the active loudspeakers.

[0636] exist Figure 21 , cooking tips are simultaneously rendered and played back for a second listener 2020b on a single loudspeaker 2005g in kitchen 2015. The reproduction of this second program stream is visually represented by a cloud 2140 emanating from loudspeaker 2005g. Figure 20 If the cooking tips are played back simultaneously with the cooking tips (shown in the example above), the audio from the movie soundtrack emanating from the speakers in or near kitchen 2015 would interfere with the second listener's ability to understand the cooking tips. Instead, in this example, the rendering of the spatial movie soundtrack is dynamically modified based on the rendering of the cooking tips. Specifically, the rendering of the movie soundtrack is moved away from the speakers near the rendering location of the cooking tips (kitchen 2015), where the shift is visually indicated by Figure 21In some embodiments, if the cooking tips playback stops while the movie soundtrack is still playing, the rendering of the movie soundtrack can be dynamically moved back to the cooking tips playback. Figure 20 This dynamic shifting of the rendering of a spatial movie soundtrack can be achieved by a number of disclosed methods.

[0637] Many spatial audio mixes consist of multiple constituent audio signals designed to be played back at specific locations in the listening space. For example, Dolby 5.1 and 7.1 surround sound mixes consist of 6 and 8 signals, respectively, meant to be played back on speakers in specified, canonical positions around the listener. Object-based audio formats such as Dolby Atmos consist of constituent audio signals with associated metadata that describes the potentially time-varying 3D positions in the listening space where the audio is to be rendered. Assuming that the renderer of a spatial movie soundtrack is capable of rendering individual audio signals at any position with respect to an arbitrary set of loudspeakers, then offsetting the intended positions of the audio signals within the spatial mix can be achieved. Figure 20 and Figure 21 . For example, the 2D or 3D coordinates associated with the audio signal can be pushed away from the position of the speakers in the kitchen, or alternatively pulled towards the upper left corner of the living room. The result of this shift is that the speakers near the kitchen are less used because the shifted position of the spatially mixed audio signal is now further away from that position. While this approach does achieve the goal of making the second audio stream more intelligible to the second listener, it does so at the cost of significantly changing the intended spatial balance of the movie soundtrack for the first listener.

[0638] A second approach for achieving a dynamic shift in spatial rendering can be achieved through the use of a flexible rendering system. In some such embodiments, the flexible rendering system can be CMAP, FV, or a hybrid of the two, as described above. Some such flexible rendering systems attempt to reproduce a spatial mix in which all of the constituent signals of the spatial mix are considered to originate from their intended locations. In some examples, while doing this for each signal in the mix, priority is given to activating loudspeakers that are close to the expected location of that signal. In some embodiments, additional terms can be dynamically added to the optimization of the rendering that penalize the use of certain loudspeakers based on other criteria. For example, a so-called "repulsive force" can be dynamically placed at the location of the kitchen to highly penalize the use of loudspeakers near that location and effectively push the rendering of a spatial movie soundtrack away. As used herein, the term "repulsive force" can refer to a factor corresponding to relatively low loudspeaker activation in a particular location or area of ​​the listening environment. In other words, the phrase "repulsive force" can refer to a factor that favors activating loudspeakers that are relatively far away from the particular location or area corresponding to the "repulsive force." However, according to some such embodiments, the renderer may still attempt to use the remaining, less penalized speakers to reproduce the intended spatial balance of the mix. As such, this technique can be considered a superior method for achieving dynamic shifting of renderings compared to methods that simply offset the intended positions of the mix's component signals.

[0639] Can be used Figure 18A A minimal version of the multi-stream renderer described in

[15] implements the described scenario of shifting the rendering of the spatial movie soundtrack away from the cooking cues in the kitchen. However, this can be achieved by adopting Figure 18BThe more capable system depicted in

[15] achieves improvements to the scenario. While shifting the rendering of the spatial movie soundtrack does improve the intelligibility of the cooking instructions in the kitchen, the movie soundtrack may still be noticeably audible in the kitchen. Depending on the temporal conditions of both streams, the cooking instructions may be masked by the movie soundtrack; for example, loud moments in the movie soundtrack mask softer moments in the cooking instructions. To address this issue, a rendering modification that dynamically modifies the cooking instructions based on the rendering of the spatial movie soundtrack can be added. For example, a method can be implemented to dynamically alter the audio signal across frequency and time to maintain its perceived loudness in the presence of an interfering signal. In this scenario, an estimate of the perceived loudness of the shifted movie soundtrack at the kitchen location can be generated and fed into such a process as an interfering signal. The temporal and frequency variation levels of the cooking instructions can then be dynamically modified to maintain their perceived loudness above the interfering signal, thereby better maintaining intelligibility for the second listener. The required estimate of the loudness of the movie soundtrack in the kitchen can be generated based on the rendered speaker feeds of the soundtrack, signals from microphones in or near the kitchen, or a combination thereof. This process of maintaining the perceived loudness of the cooking tips will typically increase the level of the cooking tips, and in some cases, the overall loudness may become objectionably high. To address this issue, yet another rendering modification can be employed. The interfering spatial movie soundtrack can be dynamically turned down based on the loudness in the kitchen being too loud due to the modified cooking tips. Finally, some external noise sources may interfere with the audibility of both program streams simultaneously; for example, a blender may be used in the kitchen while cooking. An estimate of the loudness of this ambient noise source in both the living room and the kitchen can be generated based on microphones connected to the rendering system. For example, this estimate can be added to an estimate of the loudness of the soundtrack in the kitchen to affect the loudness modification of the cooking tips. At the same time, the rendering of the soundtrack in the living room can be additionally modified based on the ambient noise estimate to maintain the perceived loudness of the soundtrack in the living room in the presence of this ambient noise, thereby better maintaining audibility for the listener in the living room.

[0640] As can be seen, this example use case of the disclosed multi-stream renderer employs a number of interrelated modifications to the two program streams in order to optimize their simultaneous playback. In summary, these modifications to the streams can be listed as follows:

[0641] Space movie soundtrack

[0642] o Shift the spatial rendering away from the kitchen based on the cooking cues being rendered in the kitchen

[0643] o Dynamically reduce the loudness based on the loudness of cooking cues rendered in the kitchen

[0644] o Dynamically increase the loudness based on an estimate of how loud an intrusive blender noise from the kitchen would be in the living room

[0645] Cooking Tips

[0646] o Dynamically increase loudness based on a combined estimate of the loudness of both the movie soundtrack and the blender noise in the kitchen

[0647] The second example use case of the disclosed multi-stream renderer involves the simultaneous playback of a spatial program stream (such as music) and the response of a smart voice assistant to certain user queries. For existing smart speakers, where playback is typically constrained to mono or stereo playback on a single device, interaction with a voice assistant typically consists of the following stages:

[0648] 1) Play music

[0649] 2) The user says the voice assistant wake-up word

[0650] 3) The smart speaker recognizes the wake-up word and significantly turns down (dodges) the music

[0651] 4) The user speaks a command to the smart assistant (i.e., “play the next song”)

[0652] 5) The smart speaker recognizes the command, confirms the command by playing a voice response (i.e., "OK, play the next song") mixed in with the ducked music through the speaker, and then executes the command

[0653] 6) The smart speaker turns the music back up to its original volume

[0654] Figure 22 and Figure 23 An example of a multi-stream renderer providing simultaneous playback of a spatial music mix and a voice assistant response is shown. Some embodiments provide improvements to the above chain of events when playing spatial audio on multiple programmed smart speakers. Specifically, the spatial mix can be shifted away from one or more of the speakers selected as suitable for relaying responses from the voice assistant. Creating this space for the voice assistant response means that the spatial mix can be turned down less, or not at all, than in the existing cases listed above. Figure 22 and Figure 23 Describes the scene.

[0655] In this example, a modified chain of events may occur as follows:

[0656] 1) Streaming spatial music programs for users on multiple programmed smart speakers ( Figure 22 Cloud line 2035c).

[0657] 2) User 2020c speaks the voice assistant wake-up word.

[0658] 3) One or more smart speakers (e.g., speaker 2005d and / or speaker 2005f) recognize the wake word and use associated recordings from microphones associated with the one or more smart speakers to determine the location of user 2020c or which speaker(s) user 2020c is closest to.

[0659] 4) Shift the rendering of the spatial music mix away from the location determined in the previous step in anticipation of the voice assistant response program stream being rendered near that location ( Figure 23 Cloud line 2035d).

[0660] 5) The user speaks a command to the smart assistant (e.g., to a smart speaker running smart assistant / virtual assistant software).

[0661] 6) The smart speaker recognizes the command, synthesizes the corresponding response program stream, and renders the response near the user's location ( Figure 23 The cloud line 2340 in the figure).

[0662] 7) When the voice assistant response is complete, the rendering of the spatial music program stream will shift back to its original state ( Figure 22 Cloud line 2035c).

[0663] In addition to optimizing the simultaneous playback of the spatial music mix and the voice assistant response, shifting the spatial music mix can also improve the ability of one set of speakers to understand the listener in step 5. This is because the music has been moved away from the speakers near the listener, thereby improving the speech to the associated microphones at other ratios.

[0664] Similar to what was described for the previous scenario with the spatial movie mix and cooking prompts, the current scenario can be further optimized beyond the optimization provided by shifting the rendering of the spatial mix based on the voice assistant response. Shifting the spatial mix by itself may not be enough to make the voice assistant response fully intelligible to the user. A simple solution is to also lower the spatial mix by a fixed amount, albeit less than what is required for the current state. Alternatively, the loudness of the voice assistant response program stream can be dynamically boosted based on the loudness of the spatial music mix program stream to maintain audibility of the response. As an extension, if this boosting of the response stream becomes too loud, the loudness of the spatial music mix can also be dynamically cut.

[0665] Figure 24 、 Figure 25 and Figure 26 A third example use case of the disclosed multi-stream renderer is illustrated. This example involves managing the simultaneous playback of a spatial music mix program stream and a comfort noise program stream while trying to ensure that a baby in an adjacent room stays asleep but is able to hear if the baby cries. Figure 24A starting point is depicted where a spatial music mix (represented by cloud 2035e) is being played optimally across all speakers in the living room 2010 and kitchen 2015 for the many people at the party. Figure 25 In the video, baby 2510 is now attempting to sleep in the adjacent bedroom 2505, depicted in the lower right corner. To help ensure this, the spatial music mix is ​​dynamically shifted away from the bedroom to minimize leakage into the bedroom (as depicted by cloud 2035f), while still maintaining a reasonable experience for the people at the party. Simultaneously, a second program stream containing soothing white noise (represented by cloud 2540) plays from speakers 2005h in the baby's room to mask any remaining leakage from the music in the adjacent room. In some examples, to ensure complete masking, the loudness of this white noise stream can be dynamically modified based on an estimate of the loudness of the spatial music leaking into the baby's room. This estimate can be generated based on a rendered speaker feed of the spatial music, a signal from a microphone in the baby's room, or a combination thereof. Additionally, if the loudness of the spatial music mix becomes too loud, it can be dynamically attenuated based on the loudness-modified noise. This is similar to the loudness processing between the spatial movie mix and the cooking tips in the first scene. Finally, microphones in the infant's room (e.g., microphones associated with speaker 2005h, which in some embodiments can be a smart speaker) can be configured to record audio from the infant (eliminating sounds that may be picked up from the spatial music and white noise), and then if crying is detected (through machine learning, via pattern matching algorithms, etc.), the combination of these processed microphone signals can serve as a third program stream that can be played back simultaneously near a listener 2020d (who can be a parent or other caregiver) in the living room 2010. Figure 26 The reproduction of this additional stream is depicted by cloud 2650. In this case, the spatial music mix can be additionally shifted away from the speakers playing the baby's crying near the parents (e.g., relative to the Figure 25 The program stream of the baby's cry can be modified in loudness based on the spatial music stream so that the baby's cry remains audible to listener 2020d. The interrelated modifications considered in this example to optimize the simultaneous playback of the three program streams can be summarized as follows:

[0666] Spatial music mixing in the living room

[0667] o Relocate spatial rendering away from the infant’s room to reduce transmission into that room

[0668] o Dynamically reduce the loudness based on the loudness of the white noise rendered in the baby's room

[0669] o As the baby's crying sound is being rendered on speakers near the parent, shift the spatial rendering away from the parent

[0670] White noise

[0671] o Dynamically increase loudness based on an estimate of the loudness of the music streaming into the baby's room

[0672] Recording baby's crying

[0673] o Dynamically increase loudness based on an estimate of the loudness of the music mix at the parent or other caregiver's location.

[0674] Next, examples of how some of the mentioned embodiments may be implemented are described.

[0675] exist Figure 18A In , each of the rendering blocks 1...N can be implemented as the same instance of any single-stream renderer (such as the previously mentioned CMAP, FV or hybrid renderer). Constructing a multi-stream renderer in this way has some convenient and useful properties.

[0676] First, if rendering is done in this layered arrangement, and each of the single-stream renderer instances is configured to operate in the frequency / transform domain (e.g., QMF), then the mixing of the streams can also occur in the frequency / transform domain, and the inverse transform only needs to be run once for M channels. This is a significant efficiency improvement compared to running N×M inverse transforms and mixing in the time domain.

[0677] Figure 27 Shown Figure 18A . In this example, a quadrature mirror analysis filter bank (QMF) is applied to each of the program streams 1 to N before it is received by the corresponding rendering module in rendering modules 1 to N. According to this example, rendering modules 1 to N operate in the frequency domain. After mixer 1830a mixes the outputs of rendering modules 1 to N, inverse synthesis filter bank 2735a converts the mixture to the time domain and provides the mixed speaker feed signals in the time domain to loudspeakers 1 to M. In this example, the quadrature mirror filter bank, rendering modules 1 to N, mixer 1830a and inverse filter bank 2735a are components of control system 610c.

[0678] Figure 28 Shown Figure 18B Frequency domain / transform domain example of a multi-stream renderer as shown in Figure 27In the embodiment of the present invention, a quadrature mirror filter bank (QMF) is applied to each program stream 1 to N before each program stream is received by the corresponding rendering module in the rendering module 1 to N. According to this example, the rendering modules 1 to N operate in the frequency domain. In this embodiment, the time domain microphone signal from the microphone system 620b is also provided to the quadrature mirror filter bank, so that the rendering modules 1 to N receive the microphone signal in the frequency domain. After the mixer 1830b mixes the outputs of the rendering modules 1 to N, the inverse filter bank 2735b converts the mixture into the time domain and provides the mixed speaker feed signal in the time domain to the loudspeakers 1 to M. In this example, the quadrature mirror filter bank, the rendering modules 1 to N, the mixer 1830b and the inverse filter bank 2735b are components of the control system 610d.

[0679] refer to Figure 29 Another example embodiment is described. As with the other figures provided herein, Figure 29 The types and quantities of elements shown in the drawings are provided as examples only. Other implementations may include more, fewer and / or different types and quantities of elements. Figure 29 A plan view of a listening environment, which in this example is a living space, is depicted. According to this example, the environment 2000 includes a living room 2010 at the upper left, a kitchen 2015 at the lower center, and a bedroom 2505 at the lower right. The boxes and circles distributed across the living space represent a set of loudspeakers 2005a to 2005h (at least some of which may be smart speakers in some embodiments) placed in locations convenient for the space, but not adhering to any standard prescribed layout (arbitrary placement). In some examples, the loudspeakers 2005a to 2005h may be coordinated to implement one or more disclosed embodiments. For example, in some embodiments, the loudspeakers 2005a to 2005h may be coordinated according to commands from a device implementing an audio session manager, which in some examples may be a CHASM. In some such examples, the disclosed audio processing (including but not limited to the disclosed flexible rendering functionality) may be based at least in part on the disclosed audio processing from the above references. Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 In some embodiments, the one or more smart audio devices in the environment 2000 may also include one or more cameras. The one or more smart audio devices may be single-purpose audio devices or virtual assistants. In some such examples, the one or more cameras (which may be Figure 6The camera of the optional sensor system 630 can reside in or on the television 2030, in a mobile phone, or in a smart speaker (e.g., one or more of microphones 2005b, 2005d, 2005e, or 2005h). Although cameras 2911a through 2911e are not shown in every depiction of the listening environments presented in this disclosure, in some embodiments, each of the listening environments (including but not limited to environment 2000) can still include one or more cameras.

[0680] Figure 30 、 Figure 31 、 Figure 32 and Figure 33 Shown for Figure 29 An example of flexible rendering of spatial audio in reference space mode for multiple different listening positions and orientations in a living space is shown in FIG. Figures 30 to 33 This functionality is depicted at four example listening positions. In each example, arrow 3005 pointing to person 3020a indicates the position of the front sound field (the position person 3020a is facing). In each example, arrow 3010a indicates the left surround field and arrow 3010b indicates the right surround field.

[0681] exist Figure 30 In , for the person 3020a sitting on the living room sofa 2025, a reference spatial model has been determined (e.g., by a device implementing an audio session manager) and spatial audio has been flexibly rendered. Figure 30 In the example shown, an optimized spatial reproduction of the audio data is created around listener 3020a using all of the loudspeakers in living room 2010 and kitchen 2015, taking into account loudspeaker capabilities and layout. This optimal reproduction is visually represented by cloud 3035 located within the range of the active loudspeakers.

[0682] According to some embodiments, a control system (e.g., a system configured to implement an audio session manager) Figure 6 The control system 610) can be configured to be configured according to the interface system (such as Figure 6 The reference spatial pattern data may include data from a microphone system (e.g., an interface system 605) to determine an assumed listening position and / or an assumed orientation of the reference spatial pattern. Some examples are described below. In some such examples, the reference spatial pattern data may include data from a microphone system (e.g., Figure 6 Microphone data of microphone system 120).

[0683] In some such examples, the reference spatial pattern data may include microphone data corresponding to a wake word and voice command (e.g., "[wake word], make the TV the front soundstage"). Alternatively or additionally, the microphone data may be used to triangulate the position of the user based on the sound of the user's voice, for example, via direction of arrival (DOA) data. For example, three or more of the loudspeakers 2005a through 2005e may use microphone data to triangulate the position of person 3020a sitting on living room couch 2025 based on the sound of person 3020a's voice via DOA data. The orientation of person 3020a may be assumed based on the position of person 3020a: if person 3020a is in Figure 30 , it can be assumed that person 3020a is facing television 2030.

[0684] Alternatively or additionally, the image may be acquired based on the image from a camera system (e.g. Figure 6 The position and orientation of person 3020a is determined using image data from sensor system 130).

[0685] In some examples, the position and orientation of person 3020a can be determined based on user input obtained via a graphical user interface (GUI). According to some such examples, the control system can be configured to control a display device (e.g., a display device of a cellular phone) to present a GUI that allows person 3020a to input the position and orientation of person 3020a.

[0686] exist Figure 31 In , for a person 3020a sitting in a living room reading chair 3115, a reference spatial pattern has been determined and spatial audio has been flexibly rendered. Figure 32 In , for the person 3020a standing next to the kitchen counter 330, the reference spatial pattern has been determined and the spatial audio has been flexibly rendered. Figure 33 In Figure 3000, a reference spatial model has been determined and spatial audio has been flexibly rendered for a person 3020a seated at a breakfast table 340. It can be observed that the front sound field orientation, indicated by arrow 3005, does not necessarily correspond to any specific loudspeaker within environment 2000. As the listener's position and orientation change, the loudspeaker's responsibility for rendering the various components of the spatial mix also changes.

[0687] for Figures 30 to 33 For any person 3020a in the figure, he or she hears the expected spatial mix for each of the positions and orientations shown. However, for other listeners in the space, the experience may be suboptimal. Figure 34 An example of reference spatial mode rendering is shown when two listeners are at different locations in the listening environment. Figure 34A reference spatial mode rendering is depicted for person 3020a on a couch and person 3020b standing in the kitchen. The rendering is optimal for person 3020a, but person 3020b, given his / her position, will hear primarily signals from the surround field and very little from the front sound field. In this and other situations where multiple people may be moving around in a space in an unpredictable manner (e.g., a party), a rendering mode that is more suitable for such a distributed audience is needed. Reference is made to pages 27 through 43 of U.S. Provisional Patent Application 62 / 705,351, filed June 23, 2020, and entitled “ADAPTABLE SPATIAL AUDIO PLAYBACK.” Figure 4 An example of such a distributed spatial rendering mode is described in Figures B through 9, which are incorporated by reference in their entirety.

[0688] Figure 35 An example of a GUI for receiving user input regarding the position and orientation of a listener is shown. According to the example, the user has previously identified a number of possible listening positions and corresponding orientations. The loudspeaker positions corresponding to each position and corresponding orientation have been entered and stored during the setup process. Some examples are disclosed herein. A detailed example of an automatic positioning process for an audio device is described below. For example, a listening environment layout GUI may have been provided, and the user may have been prompted to touch a location corresponding to a possible listening position and a speaker position and name the possible listening position. In this example, Figure 35 At the time shown, the user has provided user input regarding the user's position to the GUI 3500 by touching the virtual button "Living Room Sofa." Because there are two possible forward-facing positions given the L-shaped sofa 2025, the user is prompted to indicate which direction the user is facing.

[0689] Figure 36 An example of the geometric relationship between three audio devices in an environment is shown. In this example, environment 3600 is a room that includes a television 3601, a sofa 3603, and five audio devices 3605. According to this example, audio devices 3605 are located in positions 1 to 5 of environment 3600. In this embodiment, each of audio devices 3605 includes a microphone system 3620 having at least three microphones and a speaker system 3625 including at least one speaker. In some embodiments, each microphone system 3620 includes a microphone array. According to some embodiments, each of audio devices 3605 may include an antenna system including at least three antennas.

[0690] As with other examples disclosed herein, Figure 36The types, quantities, and arrangements of the elements shown in the drawings are for example only. Other embodiments may have different types, quantities, and arrangements of elements, for example, more or fewer audio devices 3605, audio devices 3605 in different locations, etc.

[0691] In this example, the vertices of triangle 3610a are at positions 1, 2, and 3. Here, triangle 3610a has sides 12, 23a, and 13a. According to this example, the angle between sides 12 and 23 is θ2, the angle between sides 12 and 13a is θ1, and the angle between sides 23a and 13a is θ3. As described in more detail below, these angles can be determined based on DOA data.

[0692] In some embodiments, only the relative lengths of the triangle sides may be determined. In alternative embodiments, the actual lengths of the triangle sides may be estimated. According to some such embodiments, the actual lengths of the triangle sides may be estimated based on TOA data, for example, based on the time of arrival of a sound generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. Alternatively or in addition, the lengths of the triangle sides may be estimated based on electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. For example, the lengths of the triangle sides may be estimated based on the signal strength of the electromagnetic waves generated by an audio device located at one triangle vertex and detected by an audio device located at another triangle vertex. In some embodiments, the lengths of the triangle sides may be estimated based on the phase shift of the detected electromagnetic waves.

[0693] Figure 37 Shown Figure 36 Another example of the geometric relationship between three audio devices in the environment shown. In this example, triangle 3610b has vertices at positions 1, 3, and 4. Here, triangle 3610b has sides 13b, 14, and 34a. According to this example, the angle between sides 13b and 14 is θ4, the angle between sides 13b and 34a is θ5, and the angle between sides 34a and 14 is θ6.

[0694] By comparison Figure 36 and Figure 37 , it can be observed that the length of side 13a of triangle 3610a should be equal to the length of side 13b of triangle 3610b. In some embodiments, it can be assumed that the side length of a triangle (e.g., triangle 3610a) is correct, and the length of the side shared by the adjacent triangle will be constrained to be that length.

[0695] Figure 38 Shown Figure 36 and Figure 37The two triangles depicted in the figure do not correspond to audio equipment and other features of the environment. Figure 38 An estimate of the side lengths and angular orientations of triangles 3610a and 3610b is shown. Figure 38 In the example shown, the length of side 13b of triangle 3610b is constrained to be the same as the length of side 13a of triangle 3610a. The lengths of the other sides of triangle 3610b are scaled proportionally to the resulting change in the length of side 13b. The resulting triangle 3610b' is Figure 38 Shown adjacent to triangle 3610a.

[0696] According to some embodiments, the side lengths of other triangles adjacent to triangles 3610a and 3610b may be determined in a similar manner until all of the audio device positions in environment 3600 have been determined.

[0697] Some examples of audio device locations can be performed as follows. Each audio device can report the DOA of each other audio device in the environment (e.g., room) based on the sounds produced by each other audio device in the environment (e.g., according to instructions from a device that is implementing an audio session manager (such as CHASM)). The Cartesian coordinates of the i-th audio device can be represented as x i =[x i ,y i ] T , where the superscript T indicates the vector transpose. Given M audio devices in an environment, then i = {1...M}.

[0698] Figure 39 An example of estimating the inner angle of a triangle formed by three audio devices is shown. In this example, the audio devices are i, j, and k. The DOA of a sound source emitted from device j as observed by device i can be expressed as θ ji The DOA of the sound source emitted from device k observed by device i can be expressed as θ ki .exist Figure 39 In the example shown, θ ji and θ ki is measured from axis 3905a, the orientation of the axis is arbitrary and the axis may, for example, correspond to the orientation of audio device i. The interior angle a of triangle 3910 may be expressed as a=θ ki -θ ji It can be observed that the calculation of the interior angle a is independent of the orientation of the axis 3905a.

[0699] exist Figure 39 In the example shown, θ ij and θ kjis measured from axis 3905b, the orientation of the axis is arbitrary and the axis may correspond to the orientation of audio device j. The interior angle b of triangle 3910 may be expressed as b = θ ij -θ kj Similarly, in this example, θ jk and θ ik is measured from axis 3905c. The interior angle c of triangle 3910 can be expressed as c = θ jk -θ ik .

[0700] In the presence of measurement errors, a+b+c≠180°. Robustness can be improved by predicting each angle from the other two angles and averaging them, for example, as follows:

[0701]

[0702] In some embodiments, edge lengths (A, B, C) can be calculated by applying the sine rule (up to scaling errors). In some examples, an edge length can be assigned an arbitrary value, such as 1. For example, by setting A = 1 and placing the vertex Placed at the origin, the positions of the remaining two vertices can be calculated as follows:

[0703]

[0704] However, any rotation may be acceptable.

[0705] According to some embodiments, the process of triangle parameterization can be repeated for all possible subsets of three audio devices in the environment, In some examples, T l The lth triangle may be represented. Depending on the implementation, the triangles may not be enumerated in any particular order. Due to possible errors in DOA and / or side length estimation, triangles may overlap and may not be perfectly aligned.

[0706] Figure 40 is an overview of the Figure 6 4000. As with other methods described herein, the blocks of method 4000 need not be performed in the order indicated. Furthermore, this method may include more or fewer blocks than those shown and / or described. In this embodiment, method 4000 involves estimating the position of a speaker in an environment. The blocks of method 4000 may be performed by one or more devices, which may be (or may include) Figure 6According to some embodiments, the blocks of method 4000 may be performed at least in part by a device implementing an audio session manager (e.g., CHASM) and / or according to instructions from a device implementing an audio session manager. In some such examples, the blocks of method 4000 may be performed at least in part by the above-referenced Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 The blocks of method 4000 may be as described above with reference to CHASM 208C, CHASM 208D, CHASM 307, and / or CHASM 401. Figure 15 The method 1500 is described as being performed as part of a setup process.

[0707] In this example, block 4005 involves obtaining direction of arrival (DOA) data for each of a plurality of audio devices. In some examples, the plurality of audio devices may include all audio devices in the environment, e.g. Figure 36 All audio devices 3605 are shown.

[0708] However, in some instances, the plurality of audio devices may include only a subset of all audio devices in the environment. For example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0709] Depending on the particular implementation, DOA data can be obtained in various ways. In some instances, determining DOA data can involve determining DOA data for at least one audio device from the plurality of audio devices. For example, determining DOA data can involve receiving microphone data from each of a plurality of audio device microphones corresponding to a single audio device from the plurality of audio devices, and determining DOA data for the single audio device based at least in part on the microphone data. Alternatively or additionally, determining DOA data can involve receiving antenna data from one or more antennas corresponding to the single audio device from the plurality of audio devices, and determining DOA data for the single audio device based at least in part on the antenna data.

[0710] In some such examples, a single audio device itself may determine the DOA data. According to some such embodiments, each of a plurality of audio devices may determine its own DOA data. However, in other embodiments, another device, which may be a local device or a remote device, may determine the DOA data for one or more audio devices in the environment. According to some embodiments, a server may determine the DOA data for one or more audio devices in the environment.

[0711] According to this example, block 4010 involves determining an interior angle of each of a plurality of triangles based on the DOA data. In this example, each of the plurality of triangles has vertices corresponding to audio device positions of three of the audio devices. Some such examples are described above.

[0712] Figure 41 An example is shown in which each audio device in the environment is a vertex of multiple triangles. The sides of each triangle correspond to the distance between two audio devices in audio devices 3605.

[0713] In this embodiment, block 4015 involves determining the length of each side of each of the triangles. (The sides of a triangle may also be referred to herein as "edges.") According to this example, the side lengths are based at least in part on the interior angles. In some instances, the side lengths can be calculated by determining a first length of a first side of the triangle and determining the lengths of a second side and a third side of the triangle based on the interior angles of the triangle. Some such examples are described above.

[0714] According to some such embodiments, determining the first length may involve setting the first length to a predetermined value. However, in some examples, the first length may be determined based on arrival time data and / or received signal strength data. In some embodiments, the arrival time data and / or received signal strength data may correspond to sound waves from a first audio device in the environment detected by a second audio device in the environment. Alternatively or additionally, the arrival time data and / or received signal strength data may correspond to electromagnetic waves (e.g., radio waves, infrared waves, etc.) from a first audio device in the environment detected by a second audio device in the environment.

[0715] According to this example, block 4020 involves performing a forward alignment process that aligns each triangle in the plurality of triangles in a first order. According to this example, the forward alignment process generates a forward alignment matrix.

[0716] According to some such examples, it is expected that the triangle has edges (x i , x j ) are aligned in a way that is equal to adjacent edges, e.g. Figure 38 As shown and described above. Let ε be the size of In some such embodiments, block 4020 may involve traversing ε and aligning common edges of triangles in forward order by forcing edges to coincide with edges of previously aligned edges.

[0717] Figure 42 An example of a portion of the forward alignment process is provided. Figure 42 The numbers 1 to 5 in bold are Figure 36 、 Figure 37 and Figure 41 corresponds to the audio device locations shown in . Figure 42 The order of the forward alignment process shown in and described herein is only an example.

[0718] In this example, with Figure 38 Likewise, the length of side 13b of triangle 3610b is forced to coincide with the length of side 13a of triangle 3610a. Figure 42 The resulting triangle 3610b' is shown in , where the same interior angles are maintained. According to this example, the length of side 13c of triangle 3610c is also forced to coincide with the length of side 13a of triangle 3610a. Figure 42 The resulting triangle 3610c' is shown in , where the same interior angles are maintained.

[0719] Next, in this example, the length of side 34b of triangle 3610d is forced to coincide with the length of side 34a of triangle 3610b'. Additionally, in this example, the length of side 23b of triangle 3610d is forced to coincide with the length of side 23a of triangle 3610a. Figure 42 The resulting triangle 3610d' is shown in FIG, where the same interior angles are maintained. According to some such examples, Figure 41 The remaining triangles shown in FIG. 36 can be processed in the same manner as triangles 3610b, 3610c, and 3610d.

[0720] The results of the forward alignment process can be stored in a data structure. According to some such examples, the results of the forward alignment process can be stored in a forward alignment matrix. For example, the results of the forward alignment process can be stored in a matrix , where N indicates the total number of triangles.

[0721] When the DOA data and / or the initial side length determination contain errors, multiple estimates of the audio device position will occur. During the forward alignment process, the errors will typically increase.

[0722] Figure 43 An example of multiple audio device position estimates that have occurred during the forward alignment process is shown. In this example, the forward alignment process is based on a triangle with seven audio device positions as its vertices. Here, due to additional errors in the DOA estimates, the triangles are not perfectly aligned. Figure 43 The positions of numbers 1 through 7 shown in FIG correspond to the estimated audio device positions resulting from the forward alignment process. In this example, the audio device position estimate labeled "1" agrees, but the audio device position estimates for audio devices 6 and 7 show large differences, as indicated by the relatively large areas where numbers 6 and 7 are located.

[0723] return Figure 40 In this example, block 4025 involves a reverse alignment process of aligning each of the plurality of triangles in a second order that is reversed from the first order. According to some embodiments, the reverse alignment process may involve traversing ε as previously described but in a reverse order. In alternative examples, the reverse alignment process may not be the exact reverse order of the forward alignment process. According to this example, the reverse alignment process produces a reverse alignment matrix, which may be represented herein as

[0724] Figure 44 An example of a portion of the reverse alignment process is provided. Figure 44 The numbers 1 to 5 in bold are Figure 36 、 Figure 37 and Figure 41 corresponds to the audio device locations shown in . Figure 44 The order of the reverse alignment process shown in and described herein is only an example.

[0725] exist Figure 44 In the example shown in , triangle 3610e is based on audio device locations 3, 4, and 5. In this embodiment, the side lengths (or "edges") of triangle 3610e are assumed to be correct, and the side lengths of adjacent triangles are forced to be consistent with them. According to this example, the length of side 45b of triangle 3610f is forced to be consistent with the length of side 45a of triangle 3610e. Figure 44 The resulting triangle 3610f' is shown in , where the interior angles remain the same. In this example, the length of side 35b of triangle 3610c is forced to coincide with the length of side 35a of triangle 3610e. Figure 44 The resulting triangle 3610c is shown in FIG, where the interior angles remain the same. According to some such examples, Figure 5 The remaining triangles shown in can be processed in the same manner as triangles 3610c and 3610f until the reverse alignment process has included all remaining triangles.

[0726] Figure 45 An example of multiple estimates of the audio device position that have occurred during the reverse alignment process is shown. In this example, the reverse alignment process is based on the same method as described above with reference to Figure 43 Describes a triangle with the same seven audio device positions as its vertices. Figure 45 The positions of numbers 1 through 7 shown in the figure correspond to the estimated audio device positions resulting from the inverse alignment process. Here again, due to the additional error in the DOA estimates, the triangles are not perfectly aligned. In this example, the estimated positions of audio devices labeled 6 and 7 agree, but the estimates for audio device positions of audio devices 1 and 2 show greater discrepancies.

[0727] return Figure 40 Block 4030 involves generating a final estimate of the position of each audio device based at least in part on the value of the forward alignment matrix and the value of the reverse alignment matrix. In some examples, generating the final estimate of the position of each audio device can involve translating and scaling the forward alignment matrix to generate a translated and scaled forward alignment matrix, and translating and scaling the reverse alignment matrix to generate a translated and scaled reverse alignment matrix.

[0728] For example, by moving the centroid to the origin and enforcing a unit Frobenius norm (e.g., and ) to fix pan and zoom.

[0729] According to some such examples, generating a final estimate of the position of each audio device can also involve generating a rotation matrix based on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix. The rotation matrix can include multiple estimated audio device positions for each audio device. For example, the optimal rotation between the forward alignment and the reverse alignment can be found by singular value decomposition. In some such examples, generating the rotation matrix can involve performing singular value decomposition on the translated and scaled forward alignment matrix and the translated and scaled reverse alignment matrix, for example, as follows:

[0730]

[0731] In the aforementioned equations, U represents the matrix The left singular vector of and V represents the matrix The right singular vector of . ∑ represents the singular value matrix. The above equation produces the rotation matrix R = VU T Matrix product VU T Generate a rotation matrix such that is optimally rotated to Align.

[0732] According to some examples, when determining the rotation matrix R=VU T Afterwards, the alignments can be averaged, for example, as follows:

[0733]

[0734] In some embodiments, generating a final estimate of the position of each audio device may also involve averaging the estimated audio device positions of each audio device to generate a final estimate of the position of each audio device. Various disclosed embodiments have been shown to be robust even when the DOA data and / or other calculations include significant errors. For example, due to overlapping vertices from multiple triangles, Contains the same node The final estimate is generated by averaging across common nodes.

[0735] Figure 46 A comparison of the estimated audio device position and the actual audio device position is shown. Figure 46 In the example shown, the audio device position is the same as in the above reference Figure 43 and Figure 45 The audio device positions estimated during the forward and reverse alignment processes described correspond to the positions of the audio devices. In these examples, the error in the DOA estimates has a standard deviation of 15 degrees. Nevertheless, the final estimate of each audio device position (where each final estimate is Figure 46 ) and the actual audio device positions (where each actual audio device position is in Figure 46 ) corresponds well to the

[0736] Most of the preceding discussion has dealt with automatic positioning of audio devices. The following discussion expands upon some of the methods briefly described above for determining listener position and listener angular orientation. In the preceding description, the term "rotation" is used in essentially the same manner as the term "orientation" in the following description. For example, the "rotation" mentioned above may refer to a global rotation of the final loudspeaker geometry, rather than to a rotation of the final loudspeaker geometry as in the preceding description. Figure 40 and the rotation of the individual triangles during the process described below.This global rotation or orientation may be resolved with reference to the listener angular orientation (eg, by the direction the listener is looking, the direction the listener's nose is pointing, etc.).

[0737] Various satisfactory methods for estimating the listener position are described below. However, estimating the listener angular orientation can be challenging. Some related methods are described in detail below.

[0738] Determining the listener position and the listener angular orientation can enable some desirable features, such as directionally positioned audio devices relative to the listener. Knowing the listener position and angular orientation allows determining, for example, which speakers are in front, which are in the back, which are near the center (if any), etc. relative to the listener's environment.

[0739] After establishing an association between the audio device position and the listener's position and orientation, some embodiments may involve providing the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data to an audio rendering system. Alternatively or additionally, some embodiments may involve an audio data rendering process based at least in part on the audio device position data, the audio device angular orientation data, the listener position data, and the listener angular orientation data.

[0740] Figure 47 is an overview of the Figure 6 4700 is a flowchart of an example of a method performed by an apparatus of the apparatus shown. As with other methods described herein, the blocks of method 4700 do not have to be performed in the order indicated. In addition, this method may include more or fewer blocks than shown and / or described. In this example, the blocks of method 4700 are performed by a control system, which may be (or may include) Figure 6 . As described above, in some embodiments, the control system 610 may reside in a single device, while in other embodiments, the control system 610 may reside in two or more devices. According to some embodiments, the blocks of method 4700 may be performed at least in part by a device that implements an audio session manager (e.g., CHASM) and / or according to instructions from a device that implements an audio session manager. In some such examples, the blocks of method 4700 may be performed at least in part by the above-referenced Figure 2C 、 Figure 2D 、 Figure 3C and Figure 4 According to some embodiments, the blocks of method 4700 may be as described above with reference to CHASM 208C, CHASM 208D, CHASM 307, and / or CHASM 401. Figure 12 The method 1200 may be performed as part of the process described, for example, determining a first position and a first orientation of a first person, determining at least one of a second position or a second orientation of the first person, etc.

[0741] In this example, block 4705 involves obtaining direction of arrival (DOA) data for each of a plurality of audio devices in the environment. In some examples, the plurality of audio devices may include all audio devices in the environment, such as Figure 36 All audio devices 3605 are shown.

[0742] However, in some instances, the plurality of audio devices may include only a subset of all audio devices in the environment. For example, the plurality of audio devices may include all smart speakers in the environment, but not one or more of the other audio devices in the environment.

[0743] Depending on the particular implementation, DOA data can be obtained in various ways. In some instances, determining the DOA data can involve determining DOA data for at least one audio device from the plurality of audio devices. In some examples, the DOA data can be obtained by controlling each of a plurality o...

Claims

1. An audio session management method for an audio environment having multiple audio devices, the audio session management method comprising: receiving, from a first device implementing a first application and by a device implementing an audio session manager, a first route initiation request to initiate a first route for a first audio session, the first route initiation request indicating a first audio source and a first audio environment destination, the first audio environment destination corresponding to at least a first person in the audio environment, the first audio environment destination not indicating an audio device; Establishing, by the device implementing the audio session manager, a first route corresponding to the first route initiation request, wherein establishing the first route involves: determining a first position of the at least first person in the audio environment; determining at least one audio device for a first phase of the first audio session; and Initiating or scheduling the first audio session includes: causing the at least one audio device to establish at least a first media stream corresponding to the first route, the first media stream comprising a first audio signal; and causing the first audio signal to be rendered as a first rendered audio signal; receiving, from a second device implementing a second application and by the device implementing the audio session manager, a second route initiation request to initiate a second route for a second audio session, the second route initiation request indicating a second audio source and a second audio environment destination, the second audio environment destination corresponding to at least a second person in the audio environment, and the second audio environment destination not indicating an audio device; and Establishing, by the device implementing the audio session manager, a second route corresponding to the second route initiation request, wherein establishing the second route involves: Initiating the second audio session includes: establishing at least a second media stream corresponding to the second route, the second media stream comprising a second audio signal; and causing the second audio signal to be rendered as a second rendered audio signal, wherein a rendering process for the first audio signal is modified based at least in part on at least one of the second audio signal, the second rendered audio signal, or a characteristic of the second rendered audio signal to produce a modified first rendered audio signal; and Wherein, a rendering process for the second audio signal is modified based at least in part on the first audio signal, the first rendered audio signal, or at least one of a characteristic of the first rendered audio signal to produce a modified second rendered audio signal.

2. The audio session management method according to claim 1, wherein: The first routing initiation request includes a first audio session priority.

3. The audio session management method according to claim 1 or claim 2, wherein: The first routing initiation request includes a first connection mode.

4. The audio session management method according to claim 3, wherein: The first connection mode is a synchronous connection mode, a transaction connection mode or a scheduling connection mode.

5. The audio session management method according to claim 1 or claim 2, wherein: The first routing initiation request includes an indication of whether confirmation from the at least first person will be required.

6. The audio session management method according to claim 1 or claim 2, wherein: The first routing initiation request includes a first audio session target.

7. The audio session management method according to claim 6, wherein: The first audio session objective includes one or more of intelligibility, audio quality, spatial fidelity, audibility, inaudibility, or privacy.

8. The audio session management method of claim 1 or claim 2, further comprising determining a first persistent unique audio session identifier for the first route and transmitting the first persistent unique audio session identifier to the first device.

9. The audio session management method of claim 1, further comprising determining a first orientation of the first person for the first phase of the audio session, wherein: Causing the first audio signal to be rendered as a first rendered audio signal involves: determining a first reference spatial pattern corresponding to the first position and the first orientation of the first person; as well as A first relative activation of loudspeakers in the audio environment corresponding to the first reference spatial pattern is determined.

10. The audio session management method according to claim 9, further comprising: determining at least one of a second position or a second orientation of the first person for a second phase of the first audio conversation; determining a second reference spatial pattern corresponding to at least one of the second position or the second orientation; as well as A second relative activation of loudspeakers in the audio environment corresponding to the second reference spatial pattern is determined.

11. The audio session management method according to claim 1, in, Establishing the second route involves, before initiating the second audio session: determining a first position of the at least a second person in the audio environment; and At least one audio device is determined for the first phase of the second audio session.

12. The audio session management method according to claim 1, wherein: Modifying the rendering process for the first audio signal involves performing one or more of the following operations: offsetting the rendering of the first audio signal away from the rendering position of the second rendered audio signal, or The loudness of one or more of the first rendered audio signals is modified in response to the loudness of one or more of the second audio signal or the second rendered audio signal.

13. The audio session management method according to claim 1 or claim 2, wherein: The first routing initiation request indicates at least a first area of ​​the audio environment as a first routing source or a first routing destination.

14. The audio session management method according to claim 1 or claim 2, wherein: The first routing initiation request indicates at least a first service as the first audio source.

15. An audio session management system, comprising: processor; and A memory storing instructions for controlling the processor to execute the method according to any one of claims 1 to 14.

16. A non-transitory medium having software encoded thereon, the software comprising instructions for controlling one or more devices to perform the method of any one of claims 1 to 14.

Images