Estimating user location in system comprising smart audio device

Abstract

The disclosure relates to estimating a user location in a system including a smart audio device. The present disclosure provides methods and systems for performing at least one audio activity (e.g., making a telephone call or playing music or other audio content) in an environment, including determining an estimated location of a user in the environment in response to a sound (e.g., a voice command) made by the user, and controlling the audio activity in response to determining the estimated user location. The environment may have regions indicated by a region map and the estimation of the user location may include estimating in which of the regions the user is located. The audio activity may be performed using microphones and speakers implemented in or coupled to a smart audio device.

Description

[0001] Information about divisional applications

[0002] This application is a divisional application of the Chinese invention patent application with application number 202080064412.5, application date July 28, 2020, and invention name “Estimating user position in a system including a smart audio device”.

[0003] CROSS REFERENCE TO RELATED APPLICATIONS

[0004] This application claims the benefit of U.S. Patent Application No. 16 / 929,215 filed on July 15, 2020 and U.S. Provisional Patent Application No. 62 / 880,118 filed on July 30, 2019, the entire texts of both applications are hereby incorporated by reference. Technical Field

[0005] The present invention relates to systems and methods for coordinating (or orchestrating) and implementing audio devices (e.g., smart audio devices), and to tracking a user's location in response to sounds emitted by the user and detected by (several) microphones comprising an audio device (e.g., smart audio device) system. Background Art

[0006] Currently, designers view audio devices as a single interface point for audio that can serve a mix of entertainment, communication, and information. Using audio for notifications and voice control has the advantage of avoiding visual or physical intrusion. As more systems compete for our pair of ears, the expanding device landscape becomes fragmented. As wearable augmented audio begins to become available, things do not seem to be moving towards the ideal pervasive audio personal assistant, and it is impossible to capture, connect, and communicate seamlessly with the many devices around us.

[0007] It would be useful to develop methods and systems to bridge devices (e.g., smart audio devices) and better manage location, context, content, timing, and user preferences. A set of standards, infrastructure, and APIs together can enable better unified access to the user's environment (e.g., the audio space around the user). We consider methods and systems that manage basic audio input and output and allow connected audio devices (e.g., smart audio devices) to implement specific activities (e.g., applications implemented by the system or its smart audio devices). Summary of the invention

[0008] In one class of embodiments, a method and system in which multiple audio devices (e.g., smart audio devices) are coordinated includes estimating (and typically also tracking) a user's position by responding to sounds emanating from the user and detected by microphone(s). Each microphone is included in a system including an audio device (e.g., smart audio device), and typically at least some of the microphones (of the plurality of microphones) are implemented in (or coupled to) smart audio devices of the system.

[0009] Some embodiments of the methods of the present invention include performing (and some embodiments of the systems of the present invention are configured to perform) at least one audio activity. Herein, an audio activity is an activity that includes detecting sound (using at least one microphone) and / or generating sound (by emitting sound from at least one speaker). Examples of audio activities include (but are not limited to) making a phone call (e.g., using at least one smart audio device) while detecting sound using at least one microphone (e.g., of at least one smart audio device), or playing music or other audio content (e.g., using at least one smart audio device). Some embodiments of the methods of the present invention include controlling (and some embodiments of the systems of the present invention are configured to control) at least one audio activity. This control of the audio activity may occur together with (or simultaneously with) the performance and / or control of at least one video activity (e.g., displaying a video), and each video activity may be controlled together with (or simultaneously with) the control of at least one audio activity.

[0010] In some embodiments, the method comprises the following steps:

[0011] performing at least one audio activity using a speaker set of a system implemented in an environment, wherein the system includes at least two microphones and at least two speakers, and the speaker set includes at least one of the speakers;

[0012] determining an estimated position of a user in the environment in response to a sound uttered by the user (e.g., a voice command, or an utterance that is not a voice command), wherein the sound uttered by the user is detected by at least one of the microphones of the system; and

[0013] Controlling the audio activity in response to determining the estimated location of the user includes by at least one of:

[0014] controlling at least one setting or state of the speaker group; or

[0015] The audio activity is caused to be performed using a modified speaker group, wherein the modified speaker group includes at least one speaker of the system, but wherein the modified speaker group is different from the speaker group.

[0016] Typically, at least some of the microphones and at least some of the speakers of the system are implemented in (or coupled to) a smart audio device.

[0017] In some embodiments, the method includes the steps of: performing at least one audio activity using a transducer set of a system, wherein the transducer set includes at least one microphone and at least one speaker, the system is implemented in an environment having zones, and the zones are indicated by a zone map; and determining an estimated location of a user in response to a sound (e.g., a voice command, or an utterance that is not a voice command) emitted by the user, including detecting the sound emitted by the user using at least one microphone of the system and estimating in which of the zones the user is located. Typically, the system includes microphones and speakers, and at least some of the microphones and at least some of the speakers are implemented in (or coupled to) smart audio devices. And typically, the method includes the following steps: controlling the audio activity in response to determining the estimated position of the user, including by at least one of: controlling at least one setting or state of the transducer group (for example, at least one microphone and / or at least one speaker of the transducer group); or causing the audio activity to be performed using a modified transducer group, wherein the modified transducer group includes at least one microphone and at least one speaker of the system, but wherein the modified transducer group is different from the transducer group.

[0018] In some embodiments of the inventive method, the step of controlling at least one audio activity is performed in response to determining both the estimated location of the user and at least one learned experience (e.g., a learned preference of the user). For example, this audio activity may be controlled in response to data indicative of at least one learned experience, wherein the learned experience has been determined (e.g., by a learning module of an embodiment of the inventive system) from at least one previous activity (which occurred prior to the controlling step). For example, the learned experience may have been determined from previous user commands (e.g., voice commands) asserted under conditions that are the same or similar to those present during the current audio activity, and the controlling step may be performed according to a probabilistic confidence based on the data indicative of the learned experience.

[0019] In some embodiments, a system including a coordinated plurality of smart audio devices is configured to track the location of a user in a home or other environment (e.g., within an area of ​​the environment), and determine a set of (several) best speakers and (several) best microphones of the system for implementing a current audio activity (or activities) being or to be performed by the system in view of the user's current location (e.g., the area in which the user is currently located). Tracking of the user's location may be performed in response to sounds (e.g., voice commands) emanating from the user and detected by at least one microphone (e.g., two or more microphones) of the system. Examples of the audio activities include (but are not limited to) conducting a phone call, watching a movie, listening to music, and listening to a podcast. The system may be configured to respond to a change in the user's location (e.g., movement of the user from one area to another), including by determining a new (updated) set of (several) best speakers and (several) best microphones for the activity or activities.

[0020] Aspects of the present invention include a system configured (e.g., programmed) to perform any embodiment of the present invention method or its steps, and a tangible non-transitory computer-readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., code executable for execution) for performing any embodiment of the present invention method or its steps in a non-transitory manner. For example, an embodiment of the present invention system (or one or more elements thereof) may be or include a programmable general-purpose processor, digital signal processor, or microprocessor (e.g., included in a smart phone or other smart audio device) that is programmed using software or firmware and / or otherwise configured to perform any of a variety of operations on data (including embodiments of the present invention method or its steps). This general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform an embodiment of the present invention method (or its steps) in response to data asserted thereto.

[0021] Symbols and nomenclature

[0022] Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to refer to any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), all driven by a single common speaker feed (the speaker feed may undergo different processing in different circuitry branches coupled to different transducers).

[0023] Throughout this disclosure, including in the claims, "wake-up word" is used in a broad sense to refer to any sound (e.g., a word uttered by a human, or some other sound) that a smart audio device is configured to wake up (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone) in response to detecting ("hearing") the sound. In this context, "waking up" means that the device enters a state in which it is waiting for (i.e., listening for) a voice command.

[0024] Throughout this disclosure, including in the claims, the expression "wake-up word detector" refers to a device (or software including instructions for configuring the device) configured to continuously search for alignment between real-time sound (e.g., speech) features and a trained model. Typically, a wake-up word event is triggered whenever the wake-up word detector determines that the probability of detecting a wake-up word exceeds a predefined threshold. For example, the threshold may be a predetermined threshold tuned to provide a good compromise between a false acceptance rate and a false rejection rate. After a wake-up word event, the device may enter a state (which may be referred to as an "awake" state or a "focused" state) in which the device listens for commands and passes received commands to a larger, more computationally intensive recognizer.

[0025] Throughout this disclosure, including in the claims, statements about performing an operation "on" a signal or data (e.g., filtering, scaling, transforming, or applying a gain to the signal or data) are used broadly to mean performing the operation directly on the signal or data, or on a processed version of the signal or data (e.g., a version of the signal that has undergone preliminary filtering or preprocessing before the operation is performed on the signal).

[0026] Throughout this disclosure, including in the claims, the expression "system" is used in a broad sense to refer to a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to a plurality of inputs, where the subsystem generates M inputs and the other XM inputs are received from external sources) may also be referred to as a decoder system.

[0027] Throughout this disclosure, including in the claims, the term "processor" is used in a broad sense to refer to a system or device that is programmable or otherwise configurable (e.g., using software or firmware) to perform operations on data (e.g., audio or video or other image data). Examples of processors include field programmable gate arrays (or other configurable integrated circuits or chipsets), digital signal processors that are programmed and / or otherwise configured to perform pipeline processing of audio or other sound data, programmable general purpose processors or computers, and programmable microprocessor chips or chipsets.

[0028] Throughout this disclosure, including in the claims, the terms "couple" or "coupled" are used to mean either a direct or indirect connection. Thus, if a first device couples to a second device, that connection may be through a direct connection or through an indirect connection via other devices and connections. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1A is a diagram of a system that may be implemented according to some embodiments of the present invention.

[0030] Figure 1B is a diagram of a system that may be implemented according to some embodiments of the present invention.

[0031] Figure 2 is a block diagram of a system implemented according to an embodiment of the present invention.

[0032] Figure 3 yes Figure 2 A block diagram of an exemplary embodiment of module 201.

[0033] Figure 4 yes Figure 2 A block diagram of another exemplary embodiment of module 201.

[0034] Figure 5 is a block diagram of a system implemented according to another embodiment of the present invention. DETAILED DESCRIPTION

[0035] Many embodiments of the present invention are technically feasible. Based on the present disclosure, how to implement the embodiments will be obvious to a person of ordinary skill in the art to which the present disclosure belongs. Some embodiments of the present system and method are described herein.

[0036] Examples of devices that implement audio input, output, and / or real-time interaction and are included in some embodiments of the present system include, but are not limited to, wearable devices, home audio, mobile devices, automotive and mobile computing devices, and smart speakers. Smart speakers may include network-connected speakers and microphones for cloud-based services. Other examples of devices included in some embodiments of the present system include, but are not limited to, speakers, microphones, and devices that include speaker(s) and / or microphone(s), such as lights, clocks, personal assistant devices, and / or trash cans.

[0037] In this article, we use the expression "smart audio device" to refer to a smart device that is a single-purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single-purpose audio device is a device (e.g., a TV or mobile phone) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker) and / or at least one speaker (and optionally also includes or is coupled to at least one microphone) and is largely or primarily designed to achieve a single purpose. Although TVs can generally play (and are considered to be able to play) audio from program material, in most cases, modern TVs run an operating system on which applications (including applications for watching TV) run locally. Similarly, audio input and output in a mobile phone can do a lot of things, but these are served by applications running on the phone. In this sense, a single-purpose audio device with (several) speakers and (several) microphones is typically configured to run local applications and / or services to directly use (several) speakers and (several) microphones. Some single-purpose audio devices can be configured to be grouped together to achieve audio playback on a zone or user-configured area.

[0038] A virtual assistant (e.g., a connected virtual assistant) is a device (e.g., a smart speaker or voice assistant integrated device) that includes or is coupled to at least one microphone (and optionally also includes or is coupled to at least one speaker) and can provide applications that are cloud-enabled in some sense or otherwise not implemented in or on the virtual assistant itself with the ability to utilize multiple devices (other than the virtual assistant). Virtual assistants can sometimes work together, for example, in very discrete and conditionally defined ways. For example, two or more virtual assistants can work together in the sense that one of them (i.e., the virtual assistant that is most confident of hearing the wake word) responds to the word. The connected devices can form a cluster that can be managed by one master application that can be (or implement) a virtual assistant.

[0039] Although the categories of single-purpose audio devices and virtual assistants are not strictly orthogonal, the speaker(s) and microphone(s) of an audio device (e.g., a smart audio device) may be assigned to functions enabled by or attached to (or implemented by) the smart audio device. However, it is generally not meaningful to add the speaker(s) and / or microphone(s) of an audio device that is considered individually (as distinct from the audio device) to a set.

[0040] In some embodiments, the coordinated system is or includes multiple smart audio devices (and optionally also video devices). The system (and / or one or more of its devices) is configured to implement (and execute) at least one application, including by tracking the user's location and selecting the best speaker(s) and the best microphone(s) for the application. For example, the application may be or include making a phone call, or listening to music or a podcast. In the case of a phone call, the application may involve selecting an appropriate microphone and speaker from a set of known available audio devices (i.e., included in or coupled to the set of audio devices) based on the user's location. In some embodiments, the user's location is determined from a voice command (or at least one other user utterance) and / or from an electronic positioning beacon (e.g., using Bluetooth technology). In some embodiments, once the best microphone(s) and the best speaker(s) are selected, and then the user moves, a new set of best microphone(s) and best speaker(s) is determined for the new user location.

[0041] Figure 1A and 1B Each of is a diagram of a system that may be implemented according to some embodiments of the invention. Figure 1B and Figure 1A The difference is that Figure 1A The user's location 101 in Figure 1B The location of the user in 113 is different.

[0042] exist Figure 1A and Figure 1B The marked components are:

[0043] 107: Area 1;

[0044] 112: Area 2;

[0045] 101: User (talker) location, in area 1;

[0046] 102: Direct local voice (sent by the user);

[0047] 103: A plurality of speakers in a smart audio device (e.g., a voice assistant device) located in zone 1;

[0048] 104: A plurality of microphones in a smart audio device (e.g., a voice assistant device) located in zone 1;

[0049] 105: Household appliances located in zone 1, such as lamps;

[0050] 106: A plurality of microphones located in a household appliance in zone 1;

[0051] 113: User (talker) location, in zone 2;

[0052] 108: A plurality of speakers in a smart audio device (e.g., a voice assistant device) located in zone 2;

[0053] 109: A plurality of microphones in a smart audio device (e.g., a voice assistant device) located in zone 2;

[0054] 110: A household appliance (eg, a refrigerator) located in zone 2; and

[0055] 111 : Multiple microphones located in household appliances in zone 2.

[0056] Figure 2 is a block diagram of a system implemented in an environment (e.g., home) according to an embodiment of the present invention. The system implements a "follow" mechanism to track the user's location. Figure 2 The marked components are:

[0057] 201: A subsystem (sometimes referred to as a module or "follower" module) configured to obtain input and (in response to the input) make a decision regarding the best microphone and speaker for a determined activity (e.g., indicated by input 206A);

[0058] 201A: data indicating the decision regarding the best speaker(s) for the system for the determined activity (determined in module 201) and / or the zone in which the user (e.g., talker) is currently located (i.e., one of the zones indicated by zone map 203);

[0059] 201B: Data indicating the decision regarding the best microphone(s) of the system for the determined activity (determined in module 201) and / or the area in which the user is currently located (i.e., one of the areas indicated by area map 203);

[0060] 202: User location subsystem (module), which is configured to determine the user (eg, talker, e.g. Figure 1A In some embodiments, the subsystem 202 is configured to estimate the area of ​​the user (e.g., based on a plurality of acoustic features derived from at least some of the microphones 205). In some such embodiments, the goal is not to estimate the exact geometric position of the user, but to form a robust estimate of the discrete area in which the user is located (e.g., in the presence of strong noise and residual echo);

[0061] 202A: information (data) indicating the current location of the user (talker) determined by module 202 and asserted to module 201;

[0062] 203: Area Map Subsystem, which provides an area map indicating the area of ​​the environment of the system (for example, if the system is in Figure 1A and 1B In the environment, then Figure 1A and 1B ), and a list of all microphones and speakers of the system grouped by their location in the area. In some embodiments, subsystem 203 is or includes a memory storing data indicating a map of the area;

[0063] 203A: Asserting (in some implementations of the system) to module 201 and / or module 202 information (data) about at least one area (of the area map) and a plurality of microphones and speakers contained in each such area of ​​the area map (e.g., each of at least a subset of the areas);

[0064] 204: a preprocessing subsystem coupled and configured to perform preprocessing on the output of microphone 205. Subsystem 204 may implement one or more microphone preprocessing subsystems (e.g., an echo management subsystem, a wake-up word detector, and / or a speech recognition subsystem, etc.);

[0065] 204A: pre-processed microphone signal(s) generated by and output from subsystem 204;

[0066] 205: Multiple microphones (e.g., including Figure 1A and 1B Microphones 104, 106, 109 and 111);

[0067] 206: A subsystem coupled and configured to implement at least one current audio activity (e.g., multiple currently ongoing audio activities). Each such audio activity (sometimes referred to herein as an "activity" for convenience) includes detecting sound (using at least one microphone) and / or generating sound (by emitting sound from at least one speaker). Examples of such audio activities include, but are not limited to, music playback (e.g., including steps of providing audio for rendering by subsystem 207), podcasts (e.g., including steps of providing audio for rendering by subsystem 207), and / or phone calls (e.g., including providing conference call audio for rendering by subsystem 207, and processing and / or transmitting each microphone signal provided to subsystem 204);

[0068] 206A: information (data) about the currently ongoing activity or activities implemented by subsystem 206, which is generated by subsystem 206 and asserted from subsystem 206 to module 201;

[0069] 207: A multi-channel speaker renderer subsystem coupled and configured to render audio generated or otherwise provided during execution of at least one current activity of the system (e.g., by generating speaker feeds for driving speakers 208). For example, subsystem 207 may be implemented to render audio for playback by a subset of speakers 208 (which may be implemented in or coupled to different smart audio devices) so that the sound emitted by the associated speakers can be perceived by the user in the user's current location (e.g., zone) according to data 201A (e.g., clearly, or in an optimal or desired manner);

[0070] 208: Multiple speakers (e.g., including Figure 1A and 1B 103 and 108 of 2011-03-1997); and

[0071] 401: From user (e.g., speaker, e.g. Figure 1A or 1B user) which is output from subsystem 204 and provided to module 201 in a typical implementation of the system.

[0072] Elements 201, 202, and 203 (or elements 202 and 203) may be collectively referred to as Figure 2 The user positioning and activity control subsystem of the system.

[0073] Figure 2 Elements of the system (and some other embodiments of the invention) may be implemented in or coupled to a smart audio device. For example, all or some of the speakers 208 and / or all or some of the microphones 205 may be implemented in or coupled to one or more smart audio devices, or at least some of the microphones and speakers may be implemented in a Bluetooth device connected to a Bluetooth transmitter / receiver (e.g., a smart phone). Also, for example, Figure 2 One or more other elements of the system (e.g., all or some of elements 201, 202, 203, 204, and 206) (and / or the following description) Figure 5 All or some of the elements 201, 202, 203, 204, 206, and 211 of the system may be implemented in or coupled to an intelligent audio device. In such example embodiments, the "follow" module 201 operates (and the other system elements operate) to coordinate (coordinate) the intelligent audio device by tracking the user's position in response to sounds (emitted by the user and detected by at least one microphone of the system). For example, this coordination includes coordinating the rendering of sounds emitted by (several) elements of the system and / or the processing of (several) outputs of (several) microphones of the system, and / or by the system (e.g., by element 206 of the system, such as by controlling Figure 5 At least one activity implemented by the activity manager 211 of the system or another activity manager of the system.

[0074] Typically, subsystems 202 and 203 are tightly integrated. Subsystem 202 may receive the output of all or some (e.g., two or more) of microphones 205 (e.g., implemented as asynchronous microphones). Subsystem 202 may implement a classifier, which is implemented in an intelligent audio device of the system in some instances. In other instances, the classifier may be implemented by another type of device of the system (e.g., an intelligent device that is not configured to provide audio), which is coupled and configured to communicate with the microphone. For example, at least some of microphones 205 may be discrete microphones (e.g., in a home appliance) that are not included in any intelligent audio device but are configured to communicate with a device that implements subsystem 202 as a classifier, and the classifier may be configured to estimate the area of ​​the user based on multiple acoustic features derived from the output signal of each microphone. In some such embodiments, the goal is not to estimate the exact geometric position of the user, but to form a robust estimate of the discrete area (e.g., in the presence of strong noise and residual echo).

[0075] In this document, the expression "geometric position" of an object or user or talker in an environment (referred to in the previous and following descriptions) refers to a position based on a coordinate system (e.g., a coordinate system with reference to GPS coordinates), with reference to the entire system environment (e.g., according to a Cartesian or polar coordinate system with its origin somewhere within the environment), or with reference to a specific device (e.g., a smart audio device) within the environment (e.g., according to a Cartesian or polar coordinate system with the device as its origin). In some embodiments, the subsystem 202 is configured to determine an estimate of the user's position in the environment without reference to the geometric position of the microphone 205.

[0076] The "follow" module 201 is coupled and configured to operate in response to a number of inputs (one or more of 202A, 203A, 206A, and 401) and produce one or both of outputs 201A and 201 B. Examples of the inputs are described in more detail next.

[0077] Input 203A may indicate information about each area of ​​the area map (sometimes referred to as an acoustic area), including but not limited to one or more of the following: a list of devices of the system (e.g., smart devices, microphones, speakers, etc.) positioned within each area, the size(s) of each area (e.g., in the same coordinate system as the geometric location unit), the geometric location of each area (e.g., kitchen, living room, bedroom, etc.) relative to the environment and / or relative to other areas, the geometric location of each device of the system (e.g., relative to its corresponding area and / or relative to others of the devices), and / or the name of each area.

[0078] Input 202A may be or include real-time information (data) about all or some of the following: the acoustic region in which the user (talker) is located, the talker's geometric position within this region, and how long the talker has been in this region. Input 202A may also include the user location module 202's confidence in the accuracy or correctness of any of the information mentioned in the previous sentence, and / or the history of the talker's movement (e.g., over the past N hours, where the parameter N is configurable).

[0079] Input 401 may be a voice command issued by a user (talker) or two or more voice commands, each of which has been detected by pre-processing subsystem 204 (eg, commands related or unrelated to the functionality of “follow” module 201 ).

[0080] Output 201A of module 201 is an instruction to a rendering subsystem (renderer) 207 to adapt processing according to the current (e.g., most recently determined) acoustic region of the talker. Output 201B of module 201 is an instruction to a pre-processing subsystem 204 to adapt processing according to the current (e.g., most recently determined) acoustic region of the talker.

[0081] Output 201A may indicate the geometric position of the talker relative to the talker's current acoustic region, and the geometric position and distance of each of the speakers 208 relative to the talker, for example, so that the renderer 207 may perform rendering in an optimal manner for the relevant activity being performed by the system. The optimal manner possible may depend on the activity and region, and optionally also on the talker's previously determined (e.g., recorded) preferences. For example, if the activity is a movie, and the talker is in a living room, then output 201A may instruct the renderer 207 to use as many speakers as possible to play back the audio of the movie to obtain a movie theater-like experience. If the activity is music or a podcast, and the talker is in a kitchen or bedroom, then output 201A may instruct the renderer 207 to render the music using only the closest speakers to obtain a more intimate experience.

[0082] Output 201B may indicate a ranked list of some or all of microphones 205 for use by subsystem 204 (i.e., microphone(s) whose outputs should not be ignored but should be used (i.e., processed) by subsystem 204), and the geometric location of each such microphone relative to the user (talker). In some embodiments, subsystem 204 may process the outputs of some or all of microphones 205 in a manner determined by one or more of the following: the distance of each microphone from the talker (as indicated by output 201B); the wake-up word score of each microphone (i.e., the likelihood that the microphone heard the wake-up word uttered by the user) (if available); the signal-to-noise ratio of each microphone (i.e., how loud the speech uttered by the talker is relative to the ambient noise and / or the audio playback captured from the microphone); or a combination of two or more of the foregoing. The wake-up word score and signal-to-noise ratio may be calculated by preprocessing subsystem 204. In some applications, such as a phone call, subsystem 204 may use only the output of the best of microphones 205 (as indicated by the list), or may implement beamforming using signals from multiple microphones of the list. To implement some applications, such as, for example, a distributed speech recognizer or a distributed wake-up word detector, subsystem 204 may use the output of multiple microphones 205 (e.g., determined from a ranked list indicated by output 201B, where the ranking may be, for example, in order of proximity to the user).

[0083] In some example applications, subsystem 204 (together with modules 201 and 202) implements a microphone selection or adaptive beamforming scheme that attempts to use (i.e., at least partially respond to) output 201B to more effectively pick up sound from the user's region (e.g., to better recognize a command following a wake word). In such scenarios, module 202 may use output 204A of subsystem 204 as feedback on the quality of the user region prediction to improve the user region determination in any of a variety of ways, including (but not limited to) the following:

[0084] Penalize predictions that result in incorrect recognition of the voice command following the wake-up word. For example, user region predictions that result in the user interrupting the voice assistant's response to a command (e.g., by issuing a counter-command, such as, for example, "Amanda, stop!") may be penalized;

[0085] penalizing predictions that result in a low confidence that the utterance recognizer (implemented by subsystem 204) has successfully recognized the command;

[0086] Penalizing predictions that cause the second-pass wake word detector (implemented by subsystem 204) to fail to retrospectively detect the wake word with high confidence; and / or

[0087] Strengthening predictions that result in high confidence recognition of the wake word and / or correct recognition of the user's voice command.

[0088] Figure 3 yes Figure 2 A block diagram of components of an exemplary embodiment of module 201 is shown. Figure 3 The marked components are:

[0089] Figure 2 System components (in Figure 2 and 3 are marked identically in );

[0090] 304: A module coupled and configured to recognize at least one specific type of voice command 401 and assert an indication to module 303 (in response to recognizing that the voice command 401 has a specific recognized type);

[0091] 303: A module coupled and configured to generate output signals 201A and 201B (or only one of signal 201A or signal 201B in some implementations); and

[0092] 401: Voice command(s) from the talker.

[0093] exist Figure 3 In an embodiment, the "follow" module 201 is configured to operate as follows. In response to a voice command 401 from a talker (e.g., "Amanda, move the call here" issued while the subsystem 206 is conducting a phone call), a set of changing speakers (indicated by output 201A) and / or microphones (indicated by output 201B) are determined for use by the renderer 207 and / or subsystem 204, respectively.

[0094] In such Figure 3 In the case of implementing module 201 as in Figure 2 ) may be or include a simple command and control module that directly recognizes local speech from the talker (i.e., the microphone signal(s) 204A provided from subsystem 204 to module 202 indicates this local speech, or command 401 is provided to module 202 as well as module 201). For example, Figure 2 The pre-processing subsystem 204 may contain a simple command and control module coupled and configured to recognize voice commands (indicated by the output(s) of the one or more microphones 205 ) and provide output 401 (indicative of such commands) to the module 202 and the module 201 .

[0095] In module 201 Figure 3 In an example of an embodiment, module 201 is configured to respond to a voice command 401 (e.g., "move the call here") from a talker, including by:

[0096] The position of the talker is known as a result of the zone mapping (indicated by input 202A) to instruct the renderer 207 according to the current talker acoustic zone information (indicated by output 201A), so that the renderer can change its rendering configuration to use the best speaker(s) for the current acoustic zone of the talker; and / or

[0097] The location of the talker is known as a result of the region mapping (indicated by input 202A) to instruct the pre-processing module 204 to use the output of only the best microphone(s) according to the current talker acoustic region information (indicated by output 201B).

[0098] In module 201 Figure 3 In an example of an implementation, module 201 is configured to operate as follows:

[0099] 1. Waiting for voice command (401);

[0100] 2. Upon receiving the voice command 401, determining (in module 304) whether the received command 401 is of a predetermined specific type (e.g., one of the following: "Move [activity] here" or "Follow", where

[0101] "[Activity]" herein means any of the activities currently being performed by the system (e.g., subsystem 206);

[0102] 3. If the voice command is not of the specific type, then the voice command is ignored (so that output signal 201A and / or output signal 201B are generated by module 303 as if the ignored voice command had not been received); and

[0103] 4. If the voice command is of a specific type, then output signal 201A and / or output signal 201B is generated (in module 303) to instruct other elements of the system to change their processing according to the current acoustic region (as detected by user positioning module 202 and indicated by input 202A).

[0104] Figure 4 yes Figure 2 Another exemplary embodiment of the module 201 (in Figure 4 300) and a block diagram of its operation.

[0105] exist Figure 4 The marked components are:

[0106] 300: "Follow" module;

[0107] Figure 2 System components (in Figure 2 and 4 are marked identically in );

[0108] Components 303 and 304 (labeled as Figure 3 corresponding elements of module 201);

[0109] 301: A database of data indicating preferences learned from past experiences of a talker (eg, a user). The database 301 may be implemented as a memory storing data in a non-transitory manner;

[0110] 301A: Information (data) from database 301 about preferences learned from the talker's past experience;

[0111] 302: a learning module coupled and configured to update database 301 in response to one or more of inputs 401 and / or 206A, and / or one or both of outputs 201A and 201B (generated by module 303);

[0112] 302A: updated information (data) about the talker's preferences (generated by module 302 and provided to database 301 for storage therein);

[0113] 306: A module coupled and configured to evaluate the confidence of the determined talker location;

[0114] 307: A module coupled and configured to evaluate whether the determined talker location is a new location; and

[0115] 308: A module coupled and configured to request user confirmation (eg, confirmation of user location).

[0116] Figure 4 The following module 300 implements Figure 3 An extension of the exemplary embodiment of the following module 201, wherein module 300 is configured to make automatic decisions regarding the best speaker(s) and the best microphone(s) to be used based on the talker's past experience.

[0117] exist Figure 2 The module 201 is implemented as Figure 4 In the case of module 300, Figure 2 The pre-processing subsystem 204 of the embodiment may include a simple command and control module coupled and configured to recognize voice commands (indicated by the output(s) of one or more of the microphones 205) and provide output 401 (indicative of the recognized commands) to both the module 202 and the module 300. More generally, the user positioning module 202 or the subsystem 204 (both in Figure 2) may be or implement a command and control module configured to recognize commands from direct local speech of a talker (e.g., microphone signal(s) 204A provided from subsystem 204 to module 202 indicating such local speech, or recognized speech commands 401 provided from subsystem 204 to module 202 and to module 300), and module 202 is configured to use the recognized commands to automatically detect the location of the talker.

[0118] exist Figure 4 In an embodiment, module 202 together with area map 203 may implement an acoustic area mapper (module 202 may be coupled and configured to operate with area map 203, or may be integrated with area map 203). In some implementations, the area mapper may use the output of a Bluetooth device or other radio frequency beacon to determine the location of the talker within the area. In some implementations, the area mapper may save historical information in its own system and generate an output 202A indicating a probability confidence in the location of the talker (for providing to the Figure 4 Module 300, or Figure 2 Module 306 (of module 300) may use the probability that the talker's location has been correctly determined to influence the acuity of the speaker renderer (e.g., to cause output 201A to in turn cause renderer 207 to render the relevant audio in a more focused manner, provided that module 306 has sufficient confidence in the talker's location, e.g., because module 300 has seen other examples of the talker speaking from that location, indicated by data 301A). Conversely, if module 300 does not recognize that the talker has been previously located in a particular location, and module 306 has insufficient confidence in the talker's location (e.g., confidence below a predetermined threshold), module 306 may cause output 201A to be generated so as to cause renderer 207 to render the relevant audio to be perceived in a more general vicinity.

[0119] exist Figure 4 In one embodiment, command 401 from the talker may cause module 300 to generate output 201A and / or output 201B to indicate a new set of current speakers and / or microphones and thus overrule the current speakers and / or microphones in use, such as in Figure 3 In an example embodiment of . Depending on the location of the talker within the acoustic region (e.g., as indicated by input 202A), the confidence that the talker is actually within the determined region (as determined by module 306), the activity currently being performed (i.e., as indicated by input 202A), the level of confidence that the talker is actually within the determined region (as determined by module 306), the level of confidence that the talker is actually within the determined region, ... Figure 2Based on which activities are being performed by the subsystem 206 of the speaker (e.g., as indicated by input 206A) and past learned experience (e.g., as indicated by data 301A), module 300 is configured to make an automatic decision to change the currently used speaker and / or microphone for the activity determined to be ongoing. In some embodiments, if the system does not have sufficient confidence in this automatic decision (e.g., if module 306 has a confidence level that does not exceed a predetermined threshold for the determined talker location), it may issue a request (e.g., module 306 may cause module 308 to produce output 201A to cause the request to be issued) to confirm the location from the talker. This request may be in the form of a voice prompt from the speaker closest to the talker (e.g., the prompt "We noticed you moved to the kitchen, do you want to play music here?").

[0120] Figure 4 Module 300 is configured to make automatic decisions about the configuration of renderer 207 and which microphone(s) should be used by subsystem 204 based on the movement of the talker within the acoustic area and optionally based on past experience (indicated by data in database 301). To this end, module 300 may consider input (e.g., command(s) 401) from the above-mentioned command and control module (implemented by pre-processing subsystem 204 or module 202) indicating commands indicated by the talker's direct local voice, as well as information indicating the talker's location (e.g., input 202A generated by module 202).

[0121] After a decision is made by module 300 (i.e., to produce output 201A and / or output 201B to cause a change in a previously determined set of speakers and / or microphones), learning module 302 may store data 302A in database 301, where data 302A may indicate whether the decision was satisfactory (e.g., the talker did not manually overturn the decision) or unsatisfactory (e.g., the talker manually overturned the decision by issuing a voice command) to ensure that better results are automatically determined in the future.

[0122] More generally, the generation (e.g., updating) of output 201A and / or output 201B may be performed during an ongoing audio activity in response to data (e.g., from database 301) indicating learned experience (e.g., a user's learned preferences) determined by learning module 302 (and / or another learning module of an embodiment of the system of the present invention) from at least one previous activity (which occurred prior to the generation of output 201A and / or 201B, e.g., prior to the ongoing audio activity). For example, learned experience may be determined from previous user commands asserted under conditions that are the same or similar to those present during the currently ongoing audio activity, and output 201A and / or output 201B may be updated according to a probabilistic confidence based on data (e.g., from database 301) indicating such learned experience (e.g., to affect the acuity of speaker renderer 207 in the sense that updated output 201A causes renderer 207 to render the relevant audio in a more focused manner, provided that module 300 has sufficient confidence in the user's preferences based on the learned experience).

[0123] The learning module 302 may implement a simple database of the most recent correct decisions made in response to (and / or with) each identical set of inputs (provided to the module 300) and / or features. The inputs to this database may be or include the current system activity (e.g., indicated by input 206A), the current talker acoustic region (indicated by input 202A), the previous talker acoustic region (also indicated by input 202A), and an indication of whether the previous decision in the same situation was correct (e.g., indicated by voice command 401). Alternatively, the module 302 may implement a state diagram with the probability that the talker wants to automatically change the state of the system, with each past decision (correct and incorrect) added to this probability diagram. Alternatively, the module 302 may be implemented as a neural network that learns based on all or some of the inputs in the module 300, with its output used to generate outputs 201A and 201B (e.g., to indicate to the renderer 207 and pre-processing module 204 whether a region change is needed).

[0124] Depend on Figure 2 System (wherein module 201 is implemented as Figure 4 The example flow of the processing performed by the module 300 is as follows:

[0125] 1. The talker is in acoustic area 1 (e.g. Figure 1A 107) and initiates a telephone call to Anthony;

[0126] 2. The user location module 202 and the following module 300 know that the talker is in region 1, and the module 300 generates outputs 201A and 201B to cause the pre-processing module 204 to use the best microphone (or microphones) for that region, and the renderer 207 to use the best speaker configuration for that region;

[0127] 3. The talker moves to acoustic area 2 (e.g. Figure 1B Component 112);

[0128] 4. User location module 202 detects a change in the acoustic region of the talker and asserts input 202A to module 300 to indicate the change;

[0129] 5. Module 300 recalls from past experience (i.e., data indications in database 301) that when the talker moved in an environment like the current environment, the talker requested that the phone call be moved to a new acoustic zone. After a short period of time, the confidence that the call should be moved exceeds a set threshold (as determined by module 306), and module 300 instructs pre-processing subsystem 204 to change the microphone configuration to the new acoustic zone, and also instructs renderer 207 to adjust its speaker configuration to provide the best experience for the new acoustic zone; and

[0130] 6. The speaker does not overrule the automatic decision by issuing a voice command 401 (so that module 304 does not indicate this overruling to learning module 302 and module 303), and learning module 302 causes data 302A to be stored in database 301 to indicate that module 300 made the correct decision in this case, thereby reinforcing this decision for similar future situations.

[0131] Figure 5 is a block diagram of another exemplary embodiment of the system of the present invention. Figure 5 The marked components are:

[0132] Figure 2 System components (in Figure 2 and 5 are marked identically in );

[0133] 211: Activity Manager, which is coupled to subsystem 206 and module 201 and is aware of the talker's activities inside and outside the environment (e.g., home) in which the system is implemented;

[0134] 212: a smartphone (of a user of the system (who is sometimes referred to herein as a talker)) coupled to the activity manager 211, and a Bluetooth headset connected to the smartphone; and

[0135] 206B: Information (data) generated by activity manager 211 and / or subsystem 206 and provided as input to module 201 regarding currently ongoing activities implemented by subsystem 206 (and / or activities outside the context in which the talker implements the system).

[0136] exist Figure 5 In the system, the outputs 201A and 201B of the "follow" module 201 are instructions to the activity manager 211, the renderer 207, and the pre-processing subsystem 204, which may cause the activity manager 211, the renderer 207, and each of the pre-processing subsystem 204 to adapt processing according to the current acoustic region of the talker (for example, the new acoustic region in which the talker is determined to be located).

[0137] exist Figure 5 In the system, module 201 is configured to generate output 201A and / or output 201B in response to input 206B (and other inputs provided to module 201). Output 201A of module 201 instructs renderer 207 (and / or activity manager 211) to adapt processing according to the current (e.g., newly determined) acoustic region of the talker. Output 201B of module 201 instructs pre-processing subsystem 204 (and / or activity manager 211) to adapt processing according to the current (e.g., newly determined) acoustic region of the talker.

[0138] Depend on Figure 5 The example flow of the processing implemented by the system assumes that the system is implemented in a house, except that the element 212 can be operated inside or outside the house. Figure 4 The example process is as follows:

[0139] 1. The talker goes out for a walk and receives a phone call from Anthony on the smart phone element 212;

[0140] 2. During the call, the talker walks into the house and enters acoustic zone 1 (e.g. Figure 1A Component 107), and turn off the Bluetooth headset of component 212;

[0141] 3. User location module 202 and module 201 detect that the talker enters acoustic zone 1, and module 201 knows (from input 206B) that the talker is on a phone call (implemented by subsystem 206) and the Bluetooth headset of element 212 is turned off;

[0142] 4. Module 201 recalls from past experience that the talker requested that the call be moved to a new acoustic area in an environment similar to the current environment. After a short period of time, the confidence that the call should be moved rises above a threshold, and module 201 instructs activity manager 211 (by asserting appropriate output(s) 201A and / or 201B) that the call should be moved from smartphone component 212 to the home-based Figure 5 device of the system; module 201 instructs the pre-processing subsystem 204 (by asserting the appropriate output 201B) to change the microphone configuration to the new acoustic region; and module 201 also instructs the renderer 207 (by asserting the appropriate output 201A) to adjust its speaker configuration to provide the best experience for the new acoustic region; and

[0143] 5. The speaker does not override the automated decision (made by module 201) by issuing a voice command, and the learning module (302) of module 201 stores data indicating that module 201 made the correct decision in this case, for use in reinforcing this decision for similar future situations.

[0144] Other embodiments of the method of the present invention are:

[0145] A method of controlling a system including a plurality of smart audio devices in an environment, wherein the system includes a set of one or more microphones (e.g., each of the microphones is included in or configured to communicate with at least one of the smart audio devices in the environment) and a set of one or more speakers, and wherein the environment includes a plurality of user zones, the method comprising the steps of: determining an estimate of a position of a user in the environment at least in part from output signals of the microphones, wherein the estimate indicates in which of the user zones the user is located;

[0146] A method of managing audio sessions across multiple smart audio devices, comprising the steps of changing a set of currently used microphones and speakers for an ongoing audio activity in response to a user's request or other sound uttered by the user; and

[0147] A method of managing audio sessions across multiple smart audio devices includes the steps of changing a set of currently used microphones and speakers for an ongoing audio activity based on at least one prior experience (e.g., based on at least one learned preference from a user's past experience).

[0148] Examples of embodiments of the present invention include (but are not limited to) the following:

[0149] X1. A method comprising the following steps:

[0150] performing at least one audio activity using a speaker set of a system implemented in an environment, wherein the system includes at least two microphones and at least two speakers, and the speaker set includes at least one of the speakers;

[0151] determining an estimated position of a user in the environment in response to a sound emitted by a user, wherein the sound emitted by the user is detected by at least one of the microphones of the system; and

[0152] Controlling the audio activity in response to determining the estimated location of the user includes by at least one of:

[0153] controlling at least one setting or state of the speaker group; or

[0154] The audio activity is caused to be performed using a modified speaker group, wherein the modified speaker group includes at least one speaker of the system, but wherein the modified speaker group is different from the speaker group.

[0155] X2. The method according to X1, wherein the sound emitted by the user is a voice command.

[0156] X3. A method according to X1 or X2, wherein the audio activity is making a phone call or playing audio content while using at least one microphone of the system to detect sound.

[0157] X4. A method according to X1, X2 or X3, wherein at least some of the microphones and at least some of the speakers of the system are implemented in or coupled to a smart audio device.

[0158] X5. A method according to X1, X2, X3 or X4, wherein the step of controlling the audio activity is performed in response to determining the estimated position of the user and in response to at least one learned experience.

[0159] X6. The method according to X5, wherein the system comprises at least one learning module and further comprises the following steps:

[0160] Prior to the controlling step, data indicative of the learned experience is generated and stored using the at least one learning module.

[0161] X7. A method according to X6, wherein the step of generating data indicative of the learned experience includes recognizing at least one voice command issued by the user.

[0162] X8. A method comprising the following steps:

[0163] performing at least one audio activity using a transducer set of a system implemented in an environment, wherein the transducer set includes at least one microphone and at least one speaker, the environment having an area, and the area being indicated by an area map; and

[0164] Determining an estimated position of a user in response to a sound emitted by the user includes detecting the sound emitted by the user using at least one microphone of the system and estimating in which of the areas the user is located.

[0165] X9. A method according to X8, wherein the sound emitted by the user is a voice command.

[0166] X10. A method according to X8 or X9, wherein the audio activity is making a phone call or playing audio content while using at least one microphone of the system to detect sound.

[0167] X11. A method according to X8, X9 or X10, wherein the transducer group includes a microphone and a speaker, and at least some of the microphones and at least some of the speakers are implemented in or coupled to a smart audio device.

[0168] X12. The method according to X8, X9, X10 or X11, further comprising:

[0169] Controlling the audio activity in response to determining the estimated location of the user includes by at least one of:

[0170] controlling at least one setting or state of the transducer set; or

[0171] The audio activity is caused to be performed using a modified transducer set, wherein the modified transducer set includes at least one microphone and at least one speaker of the system, but wherein the modified transducer set is different from the transducer set.

[0172] X13. A method according to X12, wherein the step of controlling the audio activity is performed in response to determining the estimated location of the user and in response to at least one learned experience.

[0173] X14. The method according to X12 or X13, wherein the system comprises at least one learning module and further comprises the following steps:

[0174] Prior to the controlling step, data indicative of the learned experience is generated and stored using the at least one learning module.

[0175] X15. A method according to X14, wherein the step of generating data indicative of the learned experience includes recognizing at least one voice command issued by the user.

[0176] X16. A computer-readable medium that stores in a non-transitory manner code for executing a method or steps of any of the methods described according to X1, X2, X3, X4, X5, X6, X7, X8, X9, X10, X11, X12, X13, X14 or X15.

[0177] X17. A system for controlling at least one audio activity in an environment, wherein the audio activity uses at least two microphones and at least two speakers in the environment, the system comprising:

[0178] a user location and activity control subsystem coupled and configured to determine an estimated position of a user in the environment in response to a sound emitted by the user and detected by at least one of the microphones, and to control the audio activity in response to determining the estimated position of the user, wherein the control is or includes at least one of:

[0179] controlling at least one setting or state of a speaker group, wherein the speaker group includes at least one of the speakers; or

[0180] The audio activity is caused to be performed using a modified speaker group, wherein the modified speaker group includes at least one speaker of the system, but wherein the modified speaker group is different from the speaker group.

[0181] X18. A system according to X17, wherein the sound emitted by the user is a voice command.

[0182] X19. A system according to X17 or X18, wherein the audio activity is or includes playing audio content while detecting sound using at least one of the at least two microphones, or making a telephone call.

[0183] X20. A system according to X17, X18 or X19, wherein at least some of the microphones and at least some of the speakers are implemented in or coupled to a smart audio device.

[0184] X21. A system according to X17, X18, X19 or X20, wherein the user positioning and activity control subsystem is configured to control the audio activity in response to determining the estimated position of the user and in response to at least one learned experience.

[0185] X22. A system according to X21, wherein the system is configured to generate and store data indicative of the learned experience, including by recognizing at least one voice command issued by the user.

[0186] X23. A system for determining a user's position during at least one audio activity performed in an environment using a transducer set, wherein the environment has an area indicated by an area map, the environment includes at least two microphones and at least two speakers, and the transducer set includes at least one of the microphones and at least one of the speakers, the system comprising:

[0187] A user positioning subsystem is coupled and configured to determine an estimated position of a user in the environment in response to a sound emanating from the user and detected using at least one of the microphones, including by estimating in which of the areas the user is located.

[0188] X24. A system according to X23, wherein the sound emitted by the user is a voice command.

[0189] X25. A system according to X23 or X24, wherein the audio activity is making a phone call or playing audio content while using at least one of the microphones to detect sound.

[0190] X26. A system according to X23, X24 or X25, wherein at least some of the microphones and at least some of the speakers are implemented in or coupled to a smart audio device.

[0191] X27. A system according to X23, X24, X25 or X26, wherein the user positioning subsystem is a user positioning and activity control subsystem coupled and configured to control the audio activity in response to determining the estimated position of the user, wherein the control is or includes at least one of the following:

[0192] controlling at least one setting or state of the transducer set; or

[0193] The audio activity is caused to be performed using a modified transducer set, wherein the modified transducer set includes at least one of the microphones and at least one of the speakers, but wherein the modified transducer set is distinct from the transducer set.

[0194] X28. A system according to X23, X24, X25, X26 or X27, wherein the user positioning and activity control subsystem is coupled and configured to control the audio activity in response to determining the estimated position of the user and in response to at least one learned experience.

[0195] X29. A system according to X28, wherein the system is configured to generate and store data indicative of the learned experience, including by recognizing at least one voice command issued by the user.

[0196] Aspects of the present invention include a system or device configured (e.g., programmed) to perform any embodiment of the present invention method, and a tangible computer-readable medium (e.g., disk) storing code for implementing any embodiment of the present invention method or its steps. For example, the present invention system may be or include a programmable general-purpose processor, digital signal processor, or microprocessor that is programmed using software or firmware and / or otherwise configured to perform any of a variety of operations on data, including embodiments of the present invention method or its steps. Such a general-purpose processor may be or include a computer system that includes an input device, a memory, and a processing subsystem that is programmed (and / or otherwise configured) to perform embodiments of the present invention method (or its steps) in response to data asserted thereto.

[0197] Some embodiments of the inventive system are implemented as a configurable (e.g., programmable) digital signal processor (DSP) configured (e.g., programmed and otherwise configured) to perform the required processing on (several) audio signals, including an embodiment of the inventive method. Alternatively, embodiments of the inventive system (or its elements) are implemented as a general-purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and memory) that is programmed and / or otherwise configured using software or firmware to perform any of a variety of operations including embodiments of the inventive method. Alternatively, elements of some embodiments of the inventive system are implemented as a general-purpose processor or DSP configured (e.g., programmed) to perform embodiments of the inventive method, and the system also includes other elements (e.g., one or more speakers and / or one or more microphones). A general-purpose processor configured to perform an embodiment of the inventive method will typically be coupled to an input device (e.g., a mouse and / or keyboard), a memory, and a display device.

[0198] Another aspect of the present invention is a tangible non-transitory computer-readable medium (e.g., a disk or other tangible storage medium) storing code (e.g., code executable for execution) for performing any embodiment of the method or steps of the present invention in a non-transitory manner. For example, ( Figure 2 or Figure 5The elements 201, 202 and 203 of the system may be implemented by a DSP (for example, implemented in a smart audio device or other audio device) or a general-purpose processor, wherein the DSP or general-purpose processor is programmed to perform an embodiment of the method of the present invention or its steps, and the general-purpose processor or DSP (or another element of the system) may include a computer-readable medium storing code for performing the embodiments of the method of the present invention or its steps in a non-transitory manner.

[0199] Although specific embodiments of the invention and applications of the invention have been described herein, it will be apparent to those of ordinary skill in the art that many variations of the embodiments and applications described herein are possible without departing from the scope of the invention described and claimed herein. It should be understood that although certain forms of the invention have been shown and described, the invention is not limited to the specific embodiments described and shown or the specific methods described.

Claims

1. A method comprising the following steps: performing at least one audio activity using a speaker group of a system implemented in an environment, wherein the system comprises at least two microphones and at least two speakers, the speaker group includes at least one of the speakers, the environment has areas, the areas are indicated by an area map, and the system stores data indicating the area map, determining an estimated location of the user in the environment in response to a sound emitted by the user, comprising detecting the sound emitted by the user using at least one of the microphones of the system and estimating in which of the areas the user is located using the data indicative of the area map; and In response to determining the estimated location of the user, controlling the audio activity comprises at least one of: controlling at least one setting or state of the speaker group; and Causing the audio activity to be performed using a modified speaker group, wherein the modified speaker group includes at least one speaker of the system, and wherein the modified speaker group is different from the speaker group, wherein the data indicative of the zone map includes a list of devices of the system located in each zone of the zone, a geometric location of each device of the system, and a size of each zone in the zone.

2. The method of claim 1, wherein the sound uttered by the user is a voice command. 3 . The method of claim 1 , wherein the audio activity is making a phone call or playing audio content while using at least one microphone of the system to detect sound. 4 . The method of claim 1 , wherein at least some of the microphones and at least some of the speakers of the system are implemented in or coupled to a smart audio device. 5 . The method of claim 1 , wherein the step of controlling the audio activity is performed in response to determining the estimated location of the user and in response to at least one learned experience.

6. The method according to claim 5, wherein the system comprises at least one learning module and further comprises the following steps: Prior to the controlling step, data indicative of the learned experience is generated and stored using the at least one learning module.

7. The method of claim 6, wherein the step of generating data indicative of the learned experience comprises recognizing at least one voice command uttered by the user. The method of claim 6 , wherein the learning module implements a state diagram.

9. The method of claim 6, wherein the learning module implements a neural network.

10. The method of claim 6, wherein the learning module implements a database storing a plurality of inputs and a plurality of decisions, wherein each of the plurality of decisions corresponds to a most recent correct decision made in response to a given one of the plurality of inputs.

11. A computer-readable medium storing, in a non-transitory manner, codes for executing the method or steps of the method according to claim 1.

12. A system for controlling at least one audio activity in an environment, wherein the audio activity uses at least two microphones and at least two speakers in the environment, the environment having zones, the zones being indicated by a zone map, and the system storing data indicating the zone map, the system comprising: a user location and activity control subsystem coupled and configured to determine an estimated position of the user in the environment in response to a sound emitted by the user, comprising detecting the sound emitted by the user using at least one of the microphones of the system and estimating in which area of ​​the area the user is located using the data indicative of the area map, and controlling the audio activity in response to determining the estimated position of the user, wherein the controlling is or includes at least one of the following: controlling at least one setting or state of a speaker group, wherein the speaker group includes at least one of the speakers; and Causing the audio activity to be performed using a modified speaker group, wherein the modified speaker group includes at least one speaker of the system, and wherein the modified speaker group is different from the speaker group, wherein the data indicative of the zone map includes a list of devices of the system located in each zone of the zone, a geometric location of each device of the system, and a size of each zone in the zone.

13. The system of claim 12, wherein the sound uttered by the user is a voice command.

14. The system of claim 12, wherein the audio activity is or includes playing audio content while detecting sound using at least one of the at least two microphones, or making a phone call.

15. The system of claim 12, wherein at least some of the microphones and at least some of the speakers are implemented in or coupled to a smart audio device.

16. The system of claim 12, wherein the user location and activity control subsystem is configured to control the audio activity in response to determining the estimated location of the user and in response to at least one learned experience.

17. The system of claim 16, wherein the system is configured to generate and store data indicative of the learned experience, including by recognizing at least one voice command uttered by the user.

18. The system of claim 16, further comprising: At least one learning module is configured to store data indicative of the learned experience, wherein the at least one learning module implements a state diagram.

19. The system of claim 16, further comprising: At least one learning module is configured to store data indicative of the learned experience, wherein the at least one learning module implements a neural network.

20. The system of claim 16, further comprising: At least one learning module configured to store data indicative of the learned experience, wherein the at least one learning module implements a database storing a plurality of inputs and a plurality of decisions, wherein each of the plurality of decisions corresponds to a most recent correct decision made in response to a given input of the plurality of inputs.

Images