Audio device based environment sensing

Abstract

Some disclosed methods involve causing one or more loudspeakers in an audio environment to emit sound and receiving, from one or more microphones in the audio environment, microphone signals corresponding to an acoustic response of the audio environment to the emitted sound. Some disclosed methods involve detecting, by a control system, a change in an acoustic response of an audio environment. Some disclosed methods involve altering one or more aspects of media processing for media played back by one or more devices in an audio environment based at least in part on a change in an acoustic response of the audio environment. Alternatively or additionally, some disclosed methods may involve changing illumination in the environment, changing gain of one or more microphone signals, locking or unlocking one or more devices, or a combination thereof, based at least in part on a change in the acoustic response.

Description

[0001] Cross - reference to related applications

[0002] This application claims the priority benefits of Spanish Patent Application No. P202231001 filed on November 18, 2022, European Patent Application No. 23152124.6 filed on January 18, 2023, and US Provisional Patent Application No. 63 / 480,920 filed on January 20, 2023. Each of these applications is incorporated herein by reference in its entirety. Technical field

[0003] The present disclosure relates to devices, systems, and methods for environmental sensing based on signals from one or more audio devices, and responses to such environmental sensing. Background art

[0004] Methods, devices, and systems for environmental sensing are known. Although existing devices, systems, and methods for environmental sensing provide benefits, improved systems and methods would still be desirable.

[0005] Symbols and terms

[0006] Throughout this disclosure, including in the claims, the terms "speaker", "loudspeaker", and "audio reproduction transducer" are used synonymously to denote any sound - generating transducer (or group of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. The speaker can be implemented to include multiple transducers (e.g., a woofer and a tweeter), which can be driven by a single common speaker feed or multiple speaker feeds. In some examples, the (multiple) speaker signals can undergo different processing in different circuit branches coupled to different transducers.

[0007] Throughout this disclosure, including in the claims, the expression of performing an operation on a signal or data (e.g., filtering, scaling, transforming, or applying gain to a signal or data) is used in a broad sense to denote performing an operation directly on the signal or data or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or pre - processing before the operation is performed on it).

[0008] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem implementing a decoder can be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, where the subsystem generates M inputs and the other X - M inputs are received from an external source) can also be referred to as a decoder system.

[0009] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to denote a system or device that is programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chip sets), digital signal processors that are programmed and / or otherwise configured to perform pipelined processing on audio or other sound data, programmable general-purpose processors or computers, and programmable microprocessor chips or chip sets.

[0010] Throughout this disclosure, including in the claims, the term "couples" or "coupled" is used to mean a direct or indirect connection. Thus, if a first device is coupled to a second device, the connection can be achieved by a direct connection or by an indirect connection via other devices and connections.

[0011] As used herein, an "intelligent device" is an electronic device that can operate interactively and / or autonomously to some extent and is typically configured to communicate with one or more other devices (or networks) via various wireless protocols such as Bluetooth, Zigbee, near field communication, Wi-Fi, Li-Fi, 3G, 4G, 5G, etc. Some well-known types of intelligent devices are smart phones, smart cars, smart thermostats, smart doorbells, smart locks, smart refrigerators, phablets and tablet computers, smart watches, smart bracelets, smart keychains, and smart audio devices. The term "intelligent device" can also refer to a device that exhibits some properties of pervasive computing such as artificial intelligence.

[0012] As used herein, the expression "smart audio device" is used to denote an intelligent device that is a single-purpose audio device or a multi-purpose audio device (e.g., an audio device that implements at least some aspects of a virtual assistant function). A single-purpose audio device is a device that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera) and is largely or primarily designed to achieve a single purpose (e.g., a television (TV)). For example, although a TV can typically play (and is considered capable of playing) audio from program material, in most instances, a modern TV runs some operating system on which applications (including applications for watching TV) run locally. In this sense, a single-purpose audio device having (multiple) speakers and (multiple) microphones is typically configured to run local applications and / or services to directly use the (multiple) speakers and (multiple) microphones. Some single-purpose audio devices can be configured to be combined together to play audio over a certain area or user-configured area.

[0013] A common type of multi-purpose audio device is an intelligent audio device (such as a "smart speaker") that implements at least some aspects of a virtual assistant function, although other aspects of the virtual assistant function can be implemented by one or more other devices such as one or more servers, and the multi-purpose audio device is configured to communicate with the one or more servers. Such a multi-purpose audio device may be referred to herein as a "virtual assistant". A virtual assistant is a device (e.g., a smart speaker or a voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker and / or at least one camera). In some examples, a virtual assistant may provide the ability to use multiple devices (different from the virtual assistant) for cloud-enabled applications or otherwise applications that are not fully implemented in or on the virtual assistant itself. In other words, at least some aspects of the virtual assistant function (e.g., the speech recognition function) may be implemented (at least in part) by one or more servers or other devices, and the virtual assistant may communicate with the one or more servers or other devices via a network (such as the Internet). Virtual assistants may sometimes work together, e.g., in a discretely and conditionally defined manner. For example, two or more virtual assistants may work together in the sense that one of them (e.g., the virtual assistant that is most certain to have heard the wake word) responds to the wake word. In some embodiments, connected virtual assistants may form a constellation that may be managed by a main application, which may be (or implement) a virtual assistant.

[0014] As used herein, "wake word" is used in a broad sense to denote any sound (e.g., a word spoken by a human or other sound) in response to the detection of which (the "hearing" of the sound) an intelligent audio device is configured to wake up (using at least one microphone included in or coupled to the intelligent audio device, or at least one other microphone). In this context, "waking up" means that the device enters a state of waiting (in other words, listening) for a voice command. In some instances, what is referred to herein as a "wake word" may include more than one word, e.g., a phrase.

[0015] As used herein, the expression "wake word detector" refers to a device (or software including instructions for configuring a device) configured to continuously search for an alignment between real-time sound (e.g., speech) features and a training model. Generally, whenever the wake word detector determines that the probability of detecting a wake word exceeds a predefined threshold, a wake word event is triggered. For example, the threshold may be a predefined threshold adjusted to give a reasonable trade-off between false acceptance rate and false rejection rate. After a wake word event, the device may enter a state (which may be referred to as a "wake" state or an "attentiveness" state) in which the device listens for commands and passes the received commands to a larger, more computationally intensive recognizer.

[0016] As used herein, the terms "program stream" and "content stream" refer to a collection of one or more audio signals and, in some instances, a collection of video signals, at least a portion of which are intended to be heard together. Examples include selections of music, movie soundtracks, movies, television shows, the audio portions of television shows, podcasts, live voice calls, synthetic voice responses from smart assistants, and the like. In some instances, a content stream may include multiple versions of at least a portion of an audio signal, e.g., the same dialogue in more than one language. In such instances, only one version (e.g., the version corresponding to a single language) is intended to be reproduced at a time for the audio data or portions thereof. SUMMARY OF THE INVENTION

[0017] At least some aspects of the present disclosure may be implemented via one or more audio processing methods. In some instances, the (s) method(s) may be implemented at least in part by a control system and / or via instructions (e.g., software) stored on one or more non-transitory media. Some methods may involve causing one or more loudspeakers in an audio environment to emit sound by a control system. According to some examples, the sound emitted by one or more loudspeakers may not be perceptible to humans. Some methods may involve receiving, by a control system, microphone signals corresponding to the acoustic response of the audio environment to the sound emitted by one or more loudspeakers from one or more microphones in the audio environment. Some methods may involve detecting, by a control system, a change in the acoustic response of the audio environment. Some methods may involve changing, at least in part based on the change in the acoustic response of the audio environment, one or more aspects of media processing for media played back by one or more devices in the audio environment.

[0018] Some examples may involve a control system estimating a change in the acoustic response of an audio environment corresponding to the presence of one or more people in the audio environment. Some such examples may involve a control system estimating the location of one or more people. According to some examples, changing one or more aspects of media processing may involve changing a rendering process for sound played back by one or more audio devices in an audio environment, at least in part based on the location of one or more people. In some examples, estimating the location of one or more people may be at least in part based on microphone signals received from a plurality of microphones in the audio environment. According to some examples, estimating the location of one or more people may be at least in part based on sound emitted by a plurality of loudspeakers in the audio environment. In some examples, changing one or more aspects of media processing may be at least in part based on the presence of one or more people in the audio environment.

[0019] Some examples may involve a control system estimating a change in the acoustic response of an audio environment corresponding to the arrival of one or more people in the audio environment or the departure of one or more people from the audio environment. When estimating that a change in the acoustic response of the audio environment corresponds to the arrival of one or more people in the audio environment, some examples may involve initiating or resuming media playback by one or more devices in the audio environment. When estimating that a change in the acoustic response of the audio environment corresponds to the departure of one or more people from the audio environment, some examples may involve stopping or pausing media playback by one or more devices in the audio environment.

[0020] In some examples, detecting a change in the acoustic response of an audio environment may involve energy analysis, correlation analysis, or a combination thereof. Alternatively or additionally, detecting a change in the acoustic response of an audio environment may involve time-based analysis, frequency-based analysis, or a combination thereof. Alternatively or additionally, detecting a change in the acoustic response of an audio environment may involve implementing a trained neural network. According to some examples, detecting a change in the acoustic response of an audio environment may involve implementing a machine learning classifier.

[0021] According to some examples, the acoustic response of an audio environment may be or may correspond to an impulse response. In some examples, media processing may involve audio processing, video processing, or a combination thereof.

[0022] In some examples, detecting a change in the acoustic response of an audio environment may involve estimating a current acoustic response of the audio environment and comparing the current acoustic response of the audio environment to a previous acoustic response of the audio environment. According to some examples, the method may involve a control system estimating a previous acoustic response of the audio environment before estimating the current acoustic response of the audio environment.

[0023] Some examples may involve changing one or more aspects of illumination in an audio environment based at least in part on a change in the acoustic response of the audio environment. Alternatively or additionally, some examples may involve changing the gain of one or more microphone signals based at least in part on a change in the acoustic response of the audio environment. Alternatively or additionally, some examples may involve locking or unlocking one or more devices based at least in part on a change in the acoustic response of the audio environment.

[0024] Some or all of the operations, functions, and / or methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read-only memory (ROM) devices, and the like. Thus, some innovative aspects of the subject matter described in this disclosure may be implemented via one or more non-transitory media having software stored thereon.

[0025] At least some aspects of the present disclosure may be implemented via an apparatus. For example, one or more devices (e.g., a system including one or more devices) may be capable of performing at least in part the methods disclosed herein. In some embodiments, the apparatus is or includes an audio processing system having an interface system and a control system. The control system may include one or more general single-chip or multi-chip processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or combinations thereof. The control system may be configured to implement some or all of the methods disclosed herein.

[0026] Details of one or more embodiments of the subject matter described in this specification are set forth in the following figures and description. Other features, aspects, and advantages will become apparent from the specification, the figures, and the claims. Note that the relative dimensions of the following figures may not be drawn to scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In the various figures, like reference numerals and names indicate like elements.

[0028] Figure 1A An example of an audio environment at a moment in time is shown.

[0029] Figure 1B Shows a representation Figure 1A of one aspect of the acoustic response of the audio environment.

[0030] Figure 2A Shows Figure 1A an example of the audio environment at another time.

[0031] Figure 2B shows an aspect of the acoustic response of an audio environment representative of Figure 2A the audio environment in a graph.

[0032] Figure 3 is a block diagram showing an example of components of an apparatus capable of implementing various aspects of the present disclosure.

[0033] Figure 4 is a flowchart outlining an example of the disclosed method. DETAILED DESCRIPTION

[0034] Some of the disclosed embodiments relate to environment awareness techniques based on microphone signals from one or more audio devices in an audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a conference room environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc.

[0035] According to some embodiments, one or more novel aspects may reside in audio processing apparatuses, systems, and methods. However, in some embodiments, one or more pre-existing loudspeakers and microphones may be used to obtain data for evaluating the state of the audio environment. Some such embodiments may utilize the already existing devices to capture the acoustic footprint of the audio environment in a user-unperceivable manner. By monitoring the acoustic response of the audio environment, some of the disclosed embodiments may be configured to estimate conditions in the audio environment, such as the presence of one or more persons, the arrival or departure of one or more persons, etc.

[0036] Alternatively or additionally, one or more novel aspects may reside in the disclosed response to changes in the audio environment. In some embodiments, one or more other types of pre-existing devices, such as one or more loudspeakers, televisions, laptop computers, cellular telephones, smart speakers, car audio systems, lights, etc., may be controlled based on detected changes in the acoustic response of the audio environment.

[0037] For example, some of the disclosed methods may involve changing one or more aspects of media processing for media played back by one or more devices in the audio environment, at least in part based on changes in the acoustic response of the audio environment. In some such examples, changing one or more aspects of media processing may involve changing the rendering process for sounds played back by one or more audio devices in the audio environment. Alternatively or additionally, some of the disclosed methods may involve changing the lighting in the environment, changing the gain of one or more microphone signals, locking or unlocking one or more devices, initiating or resuming media playback, stopping or pausing media playback, or a combination thereof, at least in part based on changes in the acoustic response.

[0038] Figure 1A shows an example of an audio environment at a particular moment. As with other figures provided herein, Figure 1A the types and quantities of the elements shown are provided only as examples. Other embodiments may include more, fewer, and / or different types and quantities of elements.

[0039] According to this example, the audio environment 100 includes audio devices 110A, 110B, 110C, and 110D. In this example, each audio device 110A - 110D includes a respective microphone among microphones 120A, 120B, 120C, and 120D and a respective loudspeaker among loudspeakers 121A, 121B, 121C, and 121D. According to some examples, each audio device 110A - 110D may be a smart audio device, such as a smart speaker.

[0040] In this example, the control system of the devices in the audio environment 100 is configured to cause the loudspeakers 121A - 121D to emit sounds 122A, 122B, 122C, and 122D, respectively. In some examples, the sounds 122A - 122D may be outside the frequency range audible to humans (e.g., 20 - 20,000 Hz). However, in other examples, the sounds 122A - 122D may be within the frequency range audible to humans. Even if the sounds 122A - 122D are within the frequency range audible to humans, depending on the specific implementation, the sounds 122A - 122D may or may not be perceptible to a person. Further examples and details are provided below.

[0041] According to this example, the control system is configured to receive microphone signals corresponding to the acoustic response of the audio environment to sounds emitted by one or more of the audio devices 110A - 110D from one or more microphones in the audio environment. In this example, the control system is configured to determine the acoustic response of the audio environment 100 based on the microphone signals. In some examples, the control system may be configured to determine the acoustic response of the audio environment 100 at different times (such as times corresponding to consecutive measurements of the acoustic response). According to some such examples, the control system may be configured to detect changes in the acoustic response of the audio environment.

[0042] According to some examples, the control system may be configured to change one or more aspects of media processing for media (such as sounds played back by one or more of the audio devices 110A - 110D, video, sound, or a combination thereof played back by a television, etc.) played back by one or more devices in the audio environment at least partially based on changes in the acoustic response of the audio environment.

[0043] In some examples, the control system can be a control system of an orchestrating device, such as the so-called smart home hub in this document. In some examples, the orchestrating device can be one of the audio devices 110A - 110D. The orchestrating device can be an instance of the device 300 described below with reference to Figure 3 In some such examples, the orchestrating device can be configured to cause two or more of the loudspeakers 121A - 121D to emit sound. According to some such examples, the orchestrating device can be configured to receive microphone signals corresponding to the acoustic response of the audio environment to the sound emitted by two or more of the audio devices 110A - 110D from the microphones of two or more of the audio devices 110A - 110D. In some such examples, the orchestrating device can be configured to determine the acoustic response of the audio environment 100 based on the microphone signals.

[0044] According to some such examples, the orchestrating device can be configured to determine a change in the acoustic response of the audio environment 100 based on the microphone signals received at multiple times (such as times corresponding to consecutive measurements of the acoustic response). In some examples, the orchestrating device can be configured to change one or more aspects of media processing for media played back by one or more devices in the audio environment (such as sound played back by one or more of the audio devices 110A - 110D, video, sound, or a combination thereof played back by a television, etc.) at least partially based on the change in the acoustic response of the audio environment.

[0045] Figure 1B shows a graph representing Figure 1A an aspect of the acoustic response of the audio environment. According to this example, the acoustic response is the impulse response of the audio environment 100. In this example, the impulse response is determined by the control system based on the sound emitted by a single audio device (such as audio device 110A) and based on the microphone signals from the same audio device. In this example, the vertical axis represents the sound pressure measured by a microphone or a microphone array, and the horizontal axis represents the distance in centimeters. In this example, the distance in centimeters represented by the vertical axis is obtained by multiplying the time-based impulse response values (which correspond to the microphone signals received by one or more microphones of the audio device) by the speed of sound. Thus, Figure 1B the distance indicated in

[0046] Figure 2A shows Figure 1AAn example of the audio environment at another time. According to this example, the change in the acoustic response of the audio environment 100 is caused by the person 205 entering the audio environment 100. In this example, the person 205 enters the audio environment 100 through the door 210, and in this example, the door remains open. Therefore, the change in the acoustic response of the audio environment 100 may be due to the presence of the person 205 (whether the person 205 is speaking or not) and the change in the position of the door 210.

[0047] Figure 2B shows a graph representing Figure 2A the acoustic response of the audio environment. According to this example, similar to Figure 1B the acoustic response is the impulse response of the audio environment 100 and is determined by the control system based on the sound emitted by the same audio device used to determine the Figure 1B acoustic response shown in Figure 1B and based on the microphone signal from the same audio device. Therefore, Figure 2B the difference between the acoustic response shown in Figure 1A and the acoustic response shown in

[0048] corresponds to one or more changes in the audio environment 100 relative to the state shown in

[0049] As described in more detail below, in some alternative examples, the acoustic response of the audio environment 100 or another audio environment can be determined based on the sounds emitted by two or more audio devices in the audio environment and the microphone signals from these audio devices.

[0050] Figure 3 is a block diagram showing an example of the components of a device capable of implementing various aspects of the present disclosure. Similar to other figures provided herein, Figure 3The types and quantities of the components shown are provided only as examples. Other embodiments may include more, fewer, and / or different types and quantities of components. According to some examples, the apparatus 300 may be configured to perform at least some of the methods disclosed herein. In some embodiments, the apparatus 300 may be or may include one or more components of an audio system. For example, in some embodiments, the apparatus 300 may be an audio device, such as a smart audio device. In other examples, the apparatus 300 may be a mobile device (such as a cellular phone), a laptop computer, a tablet computer device, a smart home hub, a television, or other types of devices.

[0051] According to some examples, the apparatus 300 may be or may include an orchestration device configured to provide control signals to two or more devices in an audio environment. In some examples, the control signals may be provided by the orchestration device to coordinate aspects of audio playback, such as controlling the playback of multiple audio devices according to an orchestrated rendering process. Some examples are disclosed herein. In some examples, the orchestration device may be or may include what is referred to herein as a smart home hub. In other examples, the orchestration device may be or may include a smart audio device, a mobile device (such as a cellular phone), a laptop computer, a tablet computer device, a television, or other types of devices.

[0052] According to some alternative embodiments, the apparatus 300 may be or may include a server. In some such examples, the apparatus 300 may be or may include an encoder. In some examples, the apparatus 300 may be or may include a decoder. Thus, in some instances, the apparatus 300 may be a device configured for use within an audio environment, such as a home audio environment, while in other instances, the apparatus 300 may be a device configured for use in the "cloud", e.g., a server.

[0053] In this example, the apparatus 300 includes an interface system 305 and a control system 310. In some embodiments, the interface system 305 may be configured to communicate with one or more other devices in the audio environment. In some examples, the audio environment may be a home audio environment. In other examples, the audio environment may be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. In some embodiments, the interface system 305 may be configured to exchange control information and associated data with the audio devices in the audio environment. In some examples, the control information and associated data may relate to one or more software applications being executed by the apparatus 300.

[0054] In some embodiments, the interface system 305 may be configured to receive a content stream or to provide a content stream. The content stream may include audio data. The audio data may include, but is not limited to, audio signals. In some instances, the audio data may include spatial data such as channel data and / or spatial metadata. For example, the metadata may be provided by a device that may be referred to herein as an "encoder". In some examples, the content stream may include video data and audio data corresponding to the video data.

[0055] The interface system 305 may include one or more network interfaces and / or one or more external device interfaces (such as one or more Universal Serial Bus (USB) interfaces). According to some embodiments, the interface system 305 may include one or more wireless interfaces. The interface system 305 may include one or more devices for implementing a user interface, such as one or more microphones, one or more speakers, a display system, a touch sensor system, and / or a gesture sensor system. Thus, although some such devices are shown separately in Figure 3 , in some examples, such devices may correspond to aspects of the interface system 305.

[0056] In some examples, the interface system 305 may include one or more interfaces between the control system 310 and a memory system (such as the optional memory system 315 shown in Figure 3 . However, in some instances, the control system 310 may include a memory system. In some embodiments, the interface system 305 may be configured to receive input from one or more microphones in the environment. In some embodiments, the interface system 305 may be configured to provide control signals from an orchestration device to one or more other devices in the audio environment.

[0057] For example, the control system 310 may include a general-purpose single-chip or multi-chip processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic, and / or discrete hardware components.

[0058] In some embodiments, the control system 310 may reside in more than one device. For example, in some embodiments, a portion of the control system 310 may reside in a device within one of the environments depicted herein, and another portion of the control system 310 may reside in a device outside the environment, such as a server, a mobile device (such as a smart phone or a tablet computer), etc. In other examples, a portion of the control system 310 may reside in a device within one of the environments depicted herein, and another portion of the control system 310 may reside in one or more other devices of the environment. For example, the functions of the control system may be distributed across multiple smart audio devices of the environment, or may be shared by an orchestration device (such as a device that may be referred to herein as a smart home hub) and one or more other devices of the environment. In other examples, a portion of the control system 310 may reside in a device (such as a server) that implements a cloud-based service, and another portion of the control system 310 may reside in another device (such as another server, a memory device, etc.) that implements a cloud-based service. In some examples, the interface system 305 may also reside in more than one device.

[0059] In some embodiments, the control system 310 may be configured to at least partially perform the methods disclosed herein. According to some examples, the control system 310 may be configured to cause one or more loudspeakers in an audio environment to emit sound. In some such examples, the control system 310 may be configured to receive microphone signals corresponding to the acoustic response of the audio environment to the emitted sound from one or more microphones in the audio environment. In some such examples, the control system 310 may be configured to detect a change in the acoustic response of the audio environment. For example, detecting a change in the acoustic response of the audio environment may involve estimating the current acoustic response of the audio environment and comparing the current acoustic response of the audio environment with a previous acoustic response of the audio environment. In some such examples, the control system 310 may have previously estimated the previous acoustic response.

[0060] According to some examples, the control system 310 may be configured to change one or more aspects of media processing for media played back by one or more devices in the audio environment at least partially based on a change in the acoustic response of the audio environment. Alternatively or additionally, the control system 310 may be configured to change the lighting in the environment, change the gain of one or more microphone signals, lock or unlock one or more devices, or a combination thereof at least partially based on a change in the acoustic response.

[0061] Some or all of the methods described herein may be performed by one or more devices according to instructions (e.g., software) stored on one or more non-transitory media. Such non-transitory media may include memory devices as described herein, including but not limited to random access memory (RAM) devices, read only memory (ROM) devices, etc. One or more non-transitory media may reside, for example, in Figure 3 the optional memory system 315 and / or the control system 310 shown in Figure 3 . Thus, various innovative aspects of the subject matter described in this disclosure may be implemented in one or more non-transitory media on which software is stored. For example, the software may include instructions for controlling at least one device to perform some or all of the methods disclosed herein. For example, the software may be executed by one or more components of a control system such as

[0062] the control system 310. Figure 3 In some examples, the device 300 may include the optional microphone system 320 shown in

[0063] . The optional microphone system 320 may include one or more microphones. According to some examples, the optional microphone system 320 may include a microphone array. In some examples, the microphone array may be configured to determine, for example, the direction of arrival (DOA) and / or the time of arrival (TOA) information according to instructions from the control system 310. In some instances, the microphone array may be configured to perform receive-side beamforming, for example, according to instructions from the control system 310. In some embodiments, one or more microphones may be part of or associated with another device (such as a speaker of a speaker system, a smart audio device, etc.). In some examples, the device 300 may not include the microphone system 320. However, in some such embodiments, the device 300 may still be configured to receive microphone data of one or more microphones in the audio environment via the interface system 310. In some such embodiments, a cloud-based implementation of the device 300 may be configured to receive microphone data or data corresponding to the microphone data from one or more microphones in the audio environment via the interface system 310.

[0063] According to some embodiments, the device 300 may include Figure 3 the optional loudspeaker system 325 shown in

[0064] . The optional loudspeaker system 325 may include one or more loudspeakers, which may also be referred to herein as "speakers", or more generally as "audio reproduction transducers". In some examples (e.g., cloud-based implementations), the device 300 may not include the loudspeaker system 325.

[0064] In some embodiments, the device 300 may include Figure 3The optional sensor system 330 shown in []. The optional sensor system 330 may include one or more touch sensors, gesture sensors, motion detectors, etc. According to some embodiments, the optional sensor system 330 may include one or more cameras. In some embodiments, the camera may be a stand-alone camera. In some examples, one or more cameras of the optional sensor system 330 may reside in a smart audio device, which may be a single-purpose audio device or a virtual assistant. In some such examples, one or more cameras of the optional sensor system 330 may reside in a television, a mobile phone, or a smart speaker. In some examples, the device 300 may not include the sensor system 330. However, in some such embodiments, the device 300 may still be configured to receive sensor data of one or more sensors (such as a camera) in the audio environment via the interface system 310.

[0065] In some embodiments, the device 300 may include Figure 3 The optional display system 335 shown in []. The optional display system 335 may include one or more displays, such as one or more light-emitting diode (LED) displays. In some instances, the optional display system 335 may include one or more organic light-emitting diode (OLED) displays. In some examples, the optional display system 335 may include one or more displays of a smart audio device. In other examples, the optional display system 335 may include a television display, a laptop computer display, a mobile device display, or another type of display. In some examples where the device 300 includes the display system 335, the sensor system 330 may include a touch sensor system and / or a gesture sensor system that is proximate to one or more displays of the display system 335. According to some such embodiments, the control system 310 may be configured to control the display system 335 to present one or more graphical user interfaces (GUIs).

[0066] According to some such examples, the device 300 may be or may include a smart audio device, such as a smart speaker. In some such embodiments, the device 300 may be or may include a wake word detector. For example, the device 300 may be configured to (at least partially) implement a virtual assistant.

[0067] According to some examples, the disclosed method may at least include an acoustic response acquisition process and an acoustic analysis process. The acoustic response acquisition process generally involves determining one or more instances of the acoustic response of an audio environment according to some objective criterion or criteria. In some examples, the acoustic response may be represented by the impulse response of the audio environment, or by another quantity that is very similar to the impulse response of the audio environment, such as a band-limited impulse response. Thus, the present disclosure may sometimes refer to the acoustic response of an audio environment as the impulse response (IR), but the inventors envision various types of acoustic response representations.

[0068] According to some examples, the acoustic analysis process involves analyzing the results of measuring one or more instances of the acoustic response of an audio environment. Various examples are described below.

[0069] In some examples, the method may further include determining a response that is at least partially based on the acoustic response or change in acoustic response of the audio environment (as determined by the acoustic analysis process). Some such examples may also involve implementing the response, instructing one or more devices to implement the response, or a combination thereof. According to some such examples, implementing the response may involve changing one or more aspects of media processing for media played back by one or more devices in the audio environment, at least partially based on a change in the acoustic response of the audio environment. Media processing may involve audio processing, video processing, or a combination thereof. Alternatively or additionally, implementing the response may involve changing one or more aspects of the lighting in the environment, changing the gain of one or more microphone signals, locking or unlocking one or more devices, or a combination thereof, at least partially based on a change in the acoustic response of the audio environment.

[0070] Acoustic Response Acquisition

[0071] In some examples, the acoustic response acquisition process may involve using one loudspeaker and one microphone. According to some alternative examples, the acoustic response acquisition process may involve using multiple loudspeakers, multiple microphones, or a combination thereof. In the case of "one loudspeaker and one microphone", generally only one acoustic response will be measured for each moment or each time interval. In the case of involving multiple loudspeakers, multiple microphones, or a combination thereof, n×m acoustic responses (such as impulse responses (IR)) will be measured, where n represents the number of loudspeakers and m represents the number of microphones.

[0072] Transparent acoustic response acquisition (in other words, acquiring the acoustic response in a way that is imperceptible to humans) has potential advantages. One potential advantage is that it can improve the experience of any listener who may be in the audio environment. Another related advantage is the ability to perform relatively more frequent acoustic response measurements, provided that the corresponding acoustic response acquisition does not annoy or distract any listener in the audio environment.

[0073] Thus, in some examples, the impulse response or a quantity similar to the impulse response can be obtained by methods imperceptible to humans in the audio environment. Such methods include, but are not limited to, the following methods:

[0074] 1. Methods relying on inaudible signals, such as sounds outside the frequency range perceptible to humans;

[0075] 2. Methods relying on hidden or masked signals; and

[0076] 3. Methods relying on reproducing normal audio content, examples of which are described below.

[0077] Inaudible signal

[0078] The swept-sine method involves using a sinusoidal excitation whose frequency increases exponentially with time. The response to the excitation can be recorded by a microphone, and the impulse response can be extracted by deconvolution techniques. In other words, the acoustic response can be determined by deconvolving the recorded response with the input swept-sine signal.

[0079] If the swept-sine signal only includes frequencies above 20KHz, the vast majority of adults cannot hear the generated signal, so the signal is transparent to typical adults. For example, if the bandwidth of the swept-sine wave is limited between 20 and 24KHz, adults generally cannot hear the generated signal. However, in some alternative examples, the swept-sine signal can have different minimum frequencies, different maximum frequencies, or a combination thereof.

[0080] Hidden or masked signal

[0081] The maximum length sequence (MLS) method is based on excitation by a periodic pseudo-random signal that has the same spectrum as white noise but more favorable statistical properties. Using MLS techniques, the impulse response can be obtained by circular cross-correlation. The MLS method benefits from a good signal-to-noise ratio and can obtain the impulse response in the presence of noise or other signals.

[0082] The MLS method allows the impulse response to be calculated based on a low-level MLS signal without the need for other sound sources to be silent. If audio content is being played back, the MLS signal may be completely masked by the emitted sound and thus cannot be perceived.

[0083] Test signals can be hidden in the audio content played back in the audio environment according to other similar strategies. Some such examples can involve monitoring the time-frequency response and playback amplitude of the hidden test signal and the playback signal, and ensuring that the hidden test signal is always masked by the playback signal according to the psychoacoustic masking principle.

[0084] Normal audio content

[0085] It is also possible to measure the impulse response of normal audio content emitted from a loudspeaker. Broadly speaking, the emitted audio signal is spectrally divided by the recorded signal to produce the impulse response. However, in practice, problems may occur when certain frequencies are missing or when there is correlated content in multiple channels. There are several strategies to solve this problem. For example, an adaptive filter can be used to extract the impulse response from the emitted audio content.

[0086] Acoustic analysis

[0087] In some disclosed acoustic analysis processes, the acquired (multiple) acoustic responses are analyzed to look for changes in the acoustic response of the audio environment. The acquired (multiple) acoustic responses can be monitored and studied. The acoustic analysis process can involve various techniques, depending on the specific implementation. Such techniques can include one or more of the following list of examples.

[0088] Energy analysis

[0089] The energy analysis method is well-suited for determining the state of the audio environment in a given area of space. Given the measured impulse response IR i (t), the ratio of the energy detected within a certain time window to the total detected energy can be expressed as follows:

[0090]

[0091] In the foregoing equation, τ represents the center time, and T represents the time window. The time T can be selected according to the distance from the monitored environmental event:

[0092] τ = d / c

[0093] In the foregoing equation, d represents the distance from the monitored environmental event, and c represents the speed of sound, taking into account the flight time between the speaker and the location of the acoustic event as well as the flight time from the location of the acoustic event to the microphone. For example, d can represent the distance from the audio device measuring the acoustic response to the doorway through which a person can enter or leave the audio environment. The time T corresponds to the monitored spatial extent: relatively small values of T correspond to relatively small detection areas.

[0094] According to some examples, a threshold of the energy ratio can be selected or determined. In some such examples, a value greater than the threshold will indicate the presence of an environmental event (such as a person entering the audio environment), while a value less than the threshold will indicate the absence of an environmental event. Some examples can involve more than one threshold. For example, some examples can involve a smaller threshold indicating a possible detection of a specific event (such as the presence of a person), while a larger threshold indicates a relatively more certain detection of that event. In another example, a smaller threshold can indicate the detection of one event, such as the opening of door 210 in FIG. 2, while a larger threshold can indicate the detection of another event, such as the presence of person 205 within the audio environment 100, the presence of person 205 within a specific distance of the audio device, etc. According to some examples, the threshold can be determined based on one or more previous occurrences of the same event. In some examples, the occurrence of an event corresponding to the threshold can be determined or verified based on input from another type of sensor (such as camera data), based on input from a person, etc. For example, if the presence of a person or a cat is estimated based on microphone data, then in some examples, camera data can be used to determine whether the estimate is correct.

[0095] In some examples, the values of τ and T may not be unique, but multiple sets of values (τ, T) can be analyzed, with each set associated with a different threshold. Each pair of values can correspond to a different energy analysis. This method allows the control system to simultaneously explore the presence of environmental events at multiple spatial locations.

[0096] Some energy-based examples can be at least partially based on an analysis of a frequency range. In addition to the analysis involving a time window, such frequency-based examples can also be implemented. Some examples are described below.

[0097] Correlation analysis

[0098] Correlation analysis is well-suited for detecting changes in the environment, such as the movement of a person or the opening of a door. Given two successive impulse responses IR i (t) and IR i+1 (t), the correlation c i between these two impulse responses can be expressed as follows:

[0099]

[0100] In the foregoing equation, τ represents the central time of the correlation, and T represents the degree of correlation. Both τ and T can be selected as described above. In some examples, the detection value d i can be expressed as follows:

[0101]

[0102] A smaller detected value indicates that the acoustic response (and presumably the state of the audio environment) is stable, while a larger detected value indicates that the acoustic response (and presumably the state of the audio environment) has changed suddenly near the detection time. In some examples, the indication of an audio environment change can be based on a threshold method, such as the method described above in the energy analysis section.

[0103] As described above, the values of τ and T may not be unique, but there can be multiple sets of values (τ, T) being analyzed, each set associated with a different threshold. Each pair of values can correspond to a different correlation analysis.

[0104] In some other examples, the values of τ and T can be large enough such that not only the direct sound from the sound source to the location of the environmental change can be monitored, but also the entire reverberation tail (including multiple room reflections) can be monitored. This method is well-suited for monitoring an entire room, including blind spots that are not in the direct line of sight of the transmitter.

[0105] Time-frequency spread

[0106] Both the energy analysis and the correlation analysis described above are directly or indirectly based on the time domain. Some energy-based examples and correlation-based examples can be at least partially based on an analysis of the frequency range.

[0107] In addition to the analysis involving time windows, some such frequency-based examples can be implemented. For example, some implementations can involve energy analysis, correlation analysis, or a combination thereof, which are similar to those described above but involve time-frequency representations (spectrograms), where the integration (or summation) is performed over a certain time-frequency region.

[0108] Events caused by relatively small objects or creatures (such as a cat or a small dog) may only affect the acoustics of the room at higher frequencies (roughly above 500 Hz), while events caused by an adult may affect the acoustics of the room at lower frequencies (roughly above 100 Hz). Thus, time-based and frequency-based examples can more accurately estimate the current environmental state and can help eliminate false alarms. For example, based only on time-based analysis, the control system may determine that a person has entered the audio environment when in fact a cat or a small dog has entered the audio environment.

[0109] Multiple transducers

[0110] Some embodiments relate to combining acoustic responses caused and detected by multiple transducers. In such cases, the above energy ratios and detection values can be represented as matrices, such as a matrix having the same number of rows as the acoustic transmitters and the same number of columns as the acoustic receivers. In some examples, there may be n×m acoustic responses (such as impulse responses (IRs)), where n represents the number of loudspeakers and m represents the number of microphones. Potential advantages of measuring acoustic responses using multiple audio devices include: improved robustness of the measured acoustic responses and the ability to use the measured acoustic responses for localization, such as for estimating the position of audio devices in an audio environment, the position of one or more persons in the audio environment, etc.

[0111] Example of a home environment with smart speakers

[0112] In this example, a home audio environment will be considered, which includes s smart speakers, each of which is capable of emitting and detecting sound. When a person enters the room, regardless of whether the person makes any sound, the acoustic effect of the room changes, and these changes can be detected by the system.

[0113] In some multi-device examples, s×s impulse responses (IRs) will be measured. We can label these impulse responses as IRij(t), where the first index i indicates the acoustic receiver device and the second index j indicates the acoustic transmitter device.

[0114] Obviously, measuring multiple IRs can improve the robustness of the method for monitoring the acoustic state of an audio environment. In addition, measuring multiple IRs can also improve the localization ability, such as the ability to estimate the position of a person in an audio environment. The following discussion includes various methods of using acoustic responses measured by multiple devices in an audio environment.

[0115] Person detection

[0116] In some examples, it is possible to detect whether there is a person in the room via a correlation-based method. For example, each acoustic response can be correlated with the acoustic responses acquired during a previous time interval, such as by using the correlation method described above.

[0117] For person detection, one advantage of using multiple acoustic responses is robustness. In some examples, the presence of a person can be determined only if at least a threshold number, fraction, percentage, etc. of all acoustic responses indicate the presence of a person in the audio environment. In one such example, the presence of a person can be determined only if at least half of all acoustic responses indicate the presence of a person in the audio environment. In another such example, the presence of a person can be determined only if at least 3 / 4 of all acoustic responses indicate the presence of a person in the audio environment. Other examples can involve higher or lower fractions, percentages, etc.

[0118] Person location using Self-IR

[0119] Define Self-IR as the IR measured based on the sound emitted and captured by the same audio device. The Self-IR can be expressed as IR ii (t), where i corresponds to the index of the relevant audio device.

[0120] The presence of people in the room usually causes a change in the Self-IR of each audio device. Usually, in each Self-IR, interference occurs at time τ, where τ = 2d / c, where d represents the distance between the person and the device, and c represents the speed of sound. The factor 2 takes into account the time required for the emitted sound to travel to the person and be reflected back by the person's body and then return to the audio device.

[0121] In some examples, the control system can independently detect the time τ for each audio device, for example, by performing a variant of energy analysis. According to some such examples, a set of predefined times τ can be analyzed, and the one that results in the highest energy ratio can be selected. Once the time τ has been determined for each audio device i , the corresponding distance τ can be found through the following formula i = 2d i / c. After the distance d between the person and each audio device is known i , the position of the person in the audio environment can be found by using triangulation, cost function minimization methods, etc.

[0122] Person location using Cross-IR

[0123] Define Cross-IR as the IR measured based on the sound emitted by one audio device and captured by another audio device. The Cross-IR can be expressed as IR ij (t), where i ≠ j. By monitoring the total energy in each Cross-IR, the approximate position of a person can be detected. Whenever a temporary drop in energy is detected in one of the Cross-IRs IR ij (t), this may mean that someone or something (such as a person) is obstructing the line of sight between device i and device j. Therefore, the position of the person, animal, or object obstructing the sound can be determined to be on the straight line connecting device i and device j.

[0124] In some instances, a temporary drop in energy may be detected in both the first Cross-IR and the second Cross-IR. In some such instances, the position of the person, animal, or object can be determined to be at the intersection of the straight line connecting device i and device j (the device corresponding to the first Cross-IR) and the straight line connecting device i' and device j' (the device corresponding to the second Cross-IR).

[0125] In some examples, the mutual IR-based positioning method can replace or supplement the self-IR-based positioning method. Combining these techniques can provide additional robustness and reliability. For example, using the mutual IR method, sometimes the position of a person can be determined to be on the straight line connecting device i and device j, so the search space of the self-IR technique can be restricted to the area around this straight line.

[0126] Conference room example

[0127] The relevant configuration can include 1 loudspeaker and m microphones. In some such examples, this configuration can be implemented in a conference room. In some such examples, m impulse responses IR i (t) can be obtained, with one IR corresponding to each microphone.

[0128] In a simple case, there may be a microphone in front of each seat in the conference room. By using the energy analysis method, it should be possible to determine (or at least make a logical inference) whether someone is sitting in a specific seat or the seat is empty. According to some examples, different actions or responses can be triggered in each case. For example, if the control system determines that a certain seat is unoccupied, the control system can mute the corresponding microphone feed. In other configurations, there may not be a corresponding microphone for each seat position, but the microphones can be distributed in the room or concentrated in a microphone array. Even in this case, by knowing the seat positions, it should be possible to determine whether a given seat is occupied.

[0129] Machine learning methods

[0130] In some examples, the acoustic responses measured by multiple devices can be combined with various analyses (such as energy, correlation, etc. in time representation and / or frequency representation) and multiple analysis parameters (e.g., τ, T...). This additional information has the potential to make the method more powerful, but it increases the difficulty and computational overhead required to determine or estimate potential events in the audio environment. In this case, simple threshold methods may no longer be practical or feasible.

[0131] Instead, machine learning methods can be applied in some data-rich examples. In some such examples, a feature set can be generated. For example, each feature can correspond to a specific combination of a given transducer (or a specific audio device), a given analysis, and a given parameter set. In some such examples, a control system implementing a machine learning algorithm (such as a machine learning classifier) can learn to predict the environmental state from the input feature set.

[0132] According to some examples, a control system implementing a machine learning algorithm can be trained to identify the presence and / or location of a person based on a collection of all acoustic responses. As in the single-device case, at least two approaches are viable. In one approach, a feature set can be generated. For example, each feature can correspond to a specific combination of a given transducer, a given analysis, and a given set of parameters. A control system implementing a machine learning algorithm can learn to predict the presence and / or location of a person, for example, based on "ground truth" feedback from a person, from other sensors in the audio environment such as one or more cameras, etc.

[0133] Another approach can involve end-to-end machine learning. For example, a control system implementing a machine learning algorithm can be provided with a set of impulse responses (time, frequency, or a combination thereof), and can automatically learn to predict changes in the environment based on the input impulse responses without any artificially designed features. An example of such a machine learning algorithm is a deep neural network.

[0134] Figure 4 is a flowchart outlining an example of the disclosed method. As with other methods described herein, the blocks of method 400 need not be performed in the indicated order. Additionally, such a method can include more or fewer blocks than those shown and / or described. In this example, method 400 is an audio processing method.

[0135] Method 400 can be performed by a device or system of device 300 as shown and described above. In some such examples, device 300 includes at least Figure 3 the control system 310 as shown and described above. In some examples, the blocks of method 400 can be performed by one or more devices within the audio environment, e.g., by an audio system controller (such as a device that can be referred to herein as a smart home hub) or by another component of the audio system (such as a smart speaker (such as Figure 3 one or more of the audio devices 110A - 110D in Figure 1A , one or more of its components, etc.), a television, a television control module, a laptop computer, a mobile device (such as a cellular phone), etc.). In some embodiments, the audio environment can include one or more rooms of a home environment. In other examples, the audio environment can be another type of environment, such as an office environment, a car environment, a train environment, a street or sidewalk environment, a park environment, etc. However, in alternative embodiments, at least some of the blocks of method 400 can be performed by one or more devices (such as one or more servers) configured to implement a cloud-based service.

[0136] In this example, block 405 involves causing one or more loudspeakers in an audio environment to emit sound by a control system. According to some examples, the sound emitted by one or more loudspeakers may not be perceivable by humans. For example, the sound emitted by one or more loudspeakers may not be within the frequency range audible to most humans. In some examples, the sound emitted by one or more loudspeakers may be hidden or masked, such as by implementing a maximum length sequence (MLS) method.

[0137] According to this example, block 410 involves the control system receiving, from one or more microphones in the audio environment, microphone signals corresponding to the acoustic response of the audio environment to the sound emitted by one or more loudspeakers. According to some examples, block 410 may involve receiving signals from a single microphone or from a single microphone array. However, in other examples, block 410 may involve receiving signals from microphones or microphone arrays in two or more audio devices.

[0138] In this example, block 415 involves the control system detecting a change in the acoustic response of the audio environment. In some examples, detecting a change in the acoustic response of the audio environment may involve obtaining or estimating the current acoustic response of the audio environment and comparing the current acoustic response of the audio environment with a previous acoustic response of the audio environment. For example, the previous acoustic response may have been obtained or estimated by the control system and may have been previously stored in a memory accessible to the control system. According to some examples, the acoustic response of the audio environment may be or may correspond to an impulse response.

[0139] In some examples, detecting a change in the acoustic response of the audio environment may involve energy analysis, correlation analysis, or a combination thereof. According to some examples, detecting a change in the acoustic response of the audio environment may involve time-based analysis, frequency-based analysis, or a combination thereof. In some examples, detecting a change in the acoustic response of the audio environment may involve implementing a trained neural network. According to some examples, detecting a change in the acoustic response of the audio environment may involve implementing a machine learning classifier.

[0140] According to this example, block 420 involves changing one or more aspects of media processing for media played back by one or more devices in the audio environment, at least in part based on a change in the acoustic response of the audio environment. In some examples, media processing may involve audio processing, video processing, or a combination thereof. Alternatively or additionally, method 400 may involve changing one or more aspects of lighting in the audio environment, changing the gain of one or more microphone signals, locking or unlocking one or more devices, or a combination thereof, at least in part based on a change in the acoustic response of the audio environment.

[0141] Some examples may involve estimating a change in the acoustic response of an audio environment corresponding to the presence of one or more people in the audio environment. Some such examples may involve estimating the position of one or more people by a control system. According to some examples, estimating the position of one or more people may be at least partially based on microphone signals received from a plurality of microphones in the audio environment (such as one or more microphones in each of a plurality of audio devices in the audio environment). In some such examples, estimating the position of one or more people may be at least partially based on sounds emitted by a plurality of loudspeakers in the audio environment (such as one or more loudspeakers in each of a plurality of audio devices in the audio environment). According to some examples, the position of one or more people may be a home audio environment position (such as a chair or a sofa), a car seat position, a conference room position (such as a conference room seat position), etc.

[0142] In some examples, changing one or more aspects of media processing may be at least partially based on the presence of one or more people in the audio environment. In some examples, changing one or more aspects of media processing may involve changing a rendering process for sounds played back by one or more audio devices in the audio environment at least partially based on the position of one or more people.

[0143] For example, referring to Figure 2A , some examples may involve rendering audio played back from audio devices 110A - 110D at least partially based on the position of person 205 in audio environment 100. In some such examples, the audio played back from audio devices 110A - 110D may be rendered to "push" the audio away from the position of person 205, such that the loudspeaker activation is relatively low at positions relatively close to person 205. Such examples may be beneficial if person 205 is speaking or likely to speak a wake word or a voice command, is on a phone call, etc.

[0144] In another example, changing a rendering process for sounds played back by one or more audio devices in the audio environment may involve "pulling" the audio towards the position of one or more people in the audio environment. Again referring to Figure 2A , the audio played back from audio devices 110A - 110D may be rendered to "pull" the audio towards the position of person 205, such that the loudspeaker activation is relatively high at positions relatively close to person 205.

[0145] Alternatively or additionally, changing a rendering process for sounds played back by one or more audio devices in the audio environment may involve optimizing the spatial reproduction of audio in the audio environment at least partially based on the position of one or more people in the audio environment. For example, again referring to Figure 2A, the audio can be rendered at least in part based on the estimated position of the person 205, the estimated orientation of the person 205, or a combination thereof to attempt to optimize the spatial reproduction of the audio played back from the audio devices 110A-110D.

[0146] Some examples of push, pull, and rendering optimizations are disclosed in U.S. Patent Publication No. 2022 / 0272454, titled "Managing Playback of Multiple Streams of Audio over Multiple Speakers", which is hereby incorporated by reference, particularly pages 15 to 20 and Figures 2F to 5C.

[0147] Some examples can involve the control system estimating a change in the acoustic response of the audio environment corresponding to one or more people arriving at or leaving the audio environment. In some instances, the control system can determine or estimate a change in the acoustic response of the audio environment corresponding to one or more people arriving at the audio environment. In some such examples, method 400 can involve initiating or resuming media playback by one or more devices in the audio environment. In some instances, the control system can determine or estimate a change in the acoustic response of the audio environment corresponding to one or more people leaving the audio environment. In some such examples, method 400 can involve stopping or pausing media playback by one or more devices in the audio environment.

[0148] Alternatively or additionally, method 400 can involve dimming the lighting in the audio environment if the change in acoustic response is interpreted as corresponding to a person leaving the audio environment; brightening the lighting in the audio environment if the change in acoustic response is interpreted as corresponding to a person entering the audio environment; or a combination thereof. In another example, method 400 can involve increasing the gain of the microphone signals in one or more audio devices estimated to be near a person in the audio environment. In some examples, method 400 can involve locking one or more devices if the change in acoustic response is interpreted as corresponding to a person leaving the audio environment; unlocking one or more devices if the change in acoustic response is interpreted as corresponding to a person entering the audio environment; or a combination thereof.

[0149] Some aspects of the present disclosure include a system or device configured (e.g., programmed) to perform one or more examples of the disclosed methods, and a tangible computer-readable medium (e.g., a disk) storing code for implementing one or more examples of the disclosed methods or their steps. For example, some disclosed systems can be or include a programmable general-purpose processor, a digital signal processor, or a microprocessor, which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the disclosed methods or their steps. Such a general-purpose processor can be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform one or more examples of the disclosed methods (or their steps) in response to data asserted thereto.

[0150] Some embodiments can be implemented as a configurable (e.g., programmable) digital signal processor (DSP) that is configured (e.g., programmed and otherwise configured) to perform the required processing on the (multiple) audio signals, including the execution of one or more examples of the disclosed methods. Alternatively, embodiments of the disclosed system (or its elements) can be implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which can include an input device and a memory), which is programmed with software or firmware and / or otherwise configured to perform any of a variety of operations, including one or more examples of the disclosed methods. Alternatively, elements of some embodiments of the system of the present invention are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform one or more examples of the disclosed methods, and the system further includes other elements (e.g., one or more loudspeakers and / or one or more microphones). The general-purpose processor configured to perform one or more examples of the disclosed methods can be coupled to an input device (e.g., a mouse and / or a keyboard), a memory, and a display device.

[0151] Another aspect of the present disclosure is a computer-readable medium (e.g., a disk or other tangible storage medium) that stores code (e.g., an encoder executable to perform one or more examples of the disclosed methods or their steps) for performing one or more examples of the disclosed methods or their steps.

[0152] The various aspects of the present invention can be understood from the following enumerated example embodiments (EEE):

[0153] EEE 1. An audio processing method, comprising:

[0154] One or more loudspeakers in an audio environment emit sound under the control of a control system;

[0155] The control system receives microphone signals corresponding to the acoustic response of the audio environment to the sound emitted by the one or more loudspeakers from one or more microphones in the audio environment;

[0156] The control system detects a change in the acoustic response of the audio environment; and

[0157] At least in part based on the change in the acoustic response of the audio environment, one or more aspects of media processing for media played back by one or more devices in the audio environment are changed.

[0158] EEE 2. The method according to EEE 1, further comprising the control system estimating that the change in the acoustic response of the audio environment corresponds to the presence of one or more people in the audio environment.

[0159] EEE 3. The method according to EEE 2, further comprising the control system estimating the position of the one or more people.

[0160] EEE 4. The method according to EEE 3, wherein changing the one or more aspects of the media processing involves at least in part changing the rendering process for sound played back by one or more audio devices in the audio environment based on the position of the one or more people.

[0161] EEE 5. The method according to EEE 3 or EEE 4, wherein estimating the position of the one or more people is at least in part based on microphone signals received from multiple microphones in the audio environment.

[0162] EEE 6. The method according to EEE 5, wherein estimating the position of the one or more people is at least in part based on sound emitted by multiple loudspeakers in the audio environment.

[0163] EEE 7. The method according to any one of EEE 2 to 6, wherein changing the one or more aspects of the media processing is at least in part based on the presence of the one or more people in the audio environment.

[0164] EEE 8. The method according to any one of EEE 1 to 7, further comprising the control system estimating that the change in the acoustic response of the audio environment corresponds to one or more people arriving at the audio environment or one or more people leaving the audio environment.

[0165] EEE 9. The method as described in EEE 8, wherein the change in the acoustic response of the audio environment corresponds to the arrival of the one or more individuals at the audio environment, and the method further includes initiating or resuming media playback by one or more devices in the audio environment.

[0166] EEE 10. The method as described in EEE 8, wherein the change in the acoustic response of the audio environment corresponds to the departure of the one or more individuals from the audio environment, and the method further includes stopping or pausing media playback by one or more devices in the audio environment.

[0167] EEE 11. The method as described in any one of EEE 1 to 10, wherein detecting the change in the acoustic response of the audio environment involves energy analysis, correlation analysis, or a combination thereof.

[0168] EEE 12. The method as described in any one of EEE 1 to 11, wherein detecting the change in the acoustic response of the audio environment involves time-based and frequency-based analysis.

[0169] EEE 13. The method as described in any one of EEE 1 to 10, wherein detecting the change in the acoustic response of the audio environment involves implementing a trained neural network.

[0170] EEE 14. The method as described in any one of EEE 1 to 10, wherein detecting the change in the acoustic response of the audio environment involves implementing a machine learning classifier.

[0171] EEE 15. The method as described in any one of EEE 1 to 14, wherein the acoustic response of the audio environment is or corresponds to an impulse response.

[0172] EEE 16. The method as described in any one of EEE 1 to 15, wherein the sound emitted by the one or more loudspeakers is not perceivable by humans.

[0173] EEE 17. The method as described in any one of EEE 1 to 16, wherein the media processing involves audio processing, video processing, or a combination thereof.

[0174] EEE 18. The method as described in any one of EEE 1 to 17, wherein detecting the change in the acoustic response of the audio environment involves estimating the current acoustic response of the audio environment and comparing the current acoustic response of the audio environment with a previous acoustic response of the audio environment.

[0175] EEE 19. The method as described in EEE 18 further includes estimating the previous acoustic response of the audio environment by the control system and before estimating the current acoustic response of the audio environment.

[0176] EEE 20. The method as described in any one of EEE 1 to 19 further includes changing one or more aspects of the lighting in the audio environment at least in part based on a change in the acoustic response of the audio environment.

[0177] EEE 21. The method as described in any one of EEE 1 to 19 further includes changing the gain of one or more microphone signals at least in part based on a change in the acoustic response of the audio environment.

[0178] EEE 22. The method as described in any one of EEE 1 to 19 further includes locking or unlocking one or more devices at least in part based on a change in the acoustic response of the audio environment.

[0179] EEE 23. An apparatus configured to implement the method as described in any one of EEE 1 to 22.

[0180] EEE 24. A system configured to implement the method as described in any one of EEE 1 to 22.

[0181] EEE 25. One or more non-transitory media having stored thereon software, the software including instructions for controlling one or more devices to implement the method as described in any one of EEE 1 to 22.

[0182] Although specific embodiments of the present disclosure and applications of the present disclosure have been described herein, it will be apparent to those of ordinary skill in the art that many changes can be made to the embodiments and applications described herein without departing from the scope of the present disclosure as described and claimed herein. It should be understood that although certain forms of the present disclosure have been shown and described, the present disclosure is not limited to the specific embodiments or specific methods described and shown.

Claims

1. An audio processing method, comprising: causing loudspeakers of a plurality of devices in an audio environment to emit sound by a control system; receiving, by the control system, microphone signals corresponding to an acoustic response of the audio environment to the sound emitted by the loudspeakers from microphones of the plurality of devices; detecting, by the control system, a change in the acoustic response of the audio environment; and changing, at least in part based on the change in the acoustic response of the audio environment, one or more aspects of media processing for media played back by one or more of the plurality of devices in the audio environment.

2. The method according to claim 1, further comprising estimating, by the control system, that the change in the acoustic response of the audio environment corresponds to the presence of one or more persons in the audio environment.

3. The method according to claim 2, further comprising estimating, by the control system, the position of the one or more persons.

4. The method according to claim 3, wherein, Changing the one or more aspects of the media processing involves changing a rendering process for sound played back by one or more of the plurality of devices in the audio environment at least in part based on the position of the one or more persons.

5. The method according to any one of claims 2 to 4, wherein Changing the one or more aspects of the media processing is at least in part based on the presence of the one or more persons in the audio environment.

6. The method according to any one of claims 1 to 5, further comprising estimating, by the control system, that the change in the acoustic response of the audio environment corresponds to one or more persons arriving at the audio environment or one or more persons leaving the audio environment.

7. The method according to claim 6, wherein, When the change in the acoustic response of the audio environment corresponds to the one or more persons arriving at the audio environment, the method further comprises initiating or resuming media playback by one or more of the plurality of devices in the audio environment.

8. The method according to claim 6, wherein When the change in the acoustic response of the audio environment corresponds to the one or more persons leaving the audio environment, the method further comprises stopping or pausing media playback by one or more of the plurality of devices in the audio environment.

9. The method according to any one of claims 1 to 8, wherein, Detecting the change in the acoustic response of the audio environment involves energy analysis, correlation analysis, or a combination thereof.

10. The method according to any one of claims 1 to 9, wherein, Detecting the change in the acoustic response of the audio environment involves time-based and frequency-based analysis.

11. The method according to any one of claims 1 to 8, wherein Detecting the change in the acoustic response of the audio environment involves implementing a trained neural network.

12. The method according to any one of claims 1 to 8, wherein Detecting the change in the acoustic response of the audio environment involves implementing a machine learning classifier.

13. The method according to any one of claims 1 to 12, wherein The acoustic response of the audio environment is or corresponds to an impulse response.

14. The method according to any one of claims 1 to 13, wherein, The sound emitted by the loudspeakers is not perceptible to humans.

15. The method according to any one of claims 1 to 14, wherein, The media processing involves audio processing, video processing, or a combination thereof.

16. The method according to any one of claims 1 to 15, wherein, Detecting the change in the acoustic response of the audio environment involves estimating a current acoustic response of the audio environment and comparing the current acoustic response of the audio environment with a previous acoustic response of the audio environment.

17. The method according to claim 16, further comprising estimating, by the control system, the previous acoustic response of the audio environment before estimating the current acoustic response of the audio environment.

18. The method according to any one of claims 1 to 17, further comprising changing one or more aspects of lighting in the audio environment at least in part based on a change in the acoustic response of the audio environment.

19. The method according to any one of claims 1 to 17, further comprising changing the gain of one or more microphone signals of the plurality of devices at least in part based on a change in the acoustic response of the audio environment.

20. The method according to any one of claims 1 to 17, further comprising locking or unlocking one or more of the plurality of devices at least in part based on a change in the acoustic response of the audio environment.

21. An apparatus configured to implement the method according to any one of claims 1 to 20.

22. A system configured to implement the method according to any one of claims 1 to 20.

23. One or more non-transitory media having stored thereon software, the software comprising instructions for controlling one or more devices to implement the method according to any one of claims 1 to 20.

Images