Context-based model selection

Abstract

An apparatus includes one or more processors configured to receive sensor data from one or more sensor devices. The one or more processors are also configured to determine a context of the apparatus based on the sensor data. The one or more processors are further configured to select a model based on the context. The one or more processors are also configured to process an input signal using the model to generate a context-specific output.

Description

[0001] I. Priority Claims

[0002] This application claims priority to jointly owned U.S. nonprovisional patent application No. 17 / 102,748, filed November 24, 2020, the contents of which are expressly incorporated herein by reference in their entirety.

[0003] II. Domain

[0004] This disclosure generally relates to context-based model selection.

[0005] III. Related Technical Descriptions

[0006] Technological advancements have led to smaller and more powerful computing devices. For example, a wide variety of portable personal computing devices exist today, including small, lightweight, and easily portable cordless phones (such as mobile and smartphones, tablets, and laptops). These devices can transmit voice and data packets over wireless networks. Furthermore, many of these devices incorporate additional functionality, such as still digital cameras, digital video cameras, digital recorders, and audio file players. Moreover, such devices can process executable instructions, including software applications that can be used to access the Internet (such as web browser applications). Thus, these devices can include considerable computing power, including, for example, sound event classification (SEC) systems that attempt to identify sound events in audio signals (e.g., door closing, car horn, etc.), noise reduction systems, automatic speech recognition (ASR) systems, natural language processing (NLP) systems, and so on.

[0007] Systems performing operations (such as SEC, noise reduction, ASR, NLP, etc.) can use models trained to provide broad applicability but which are difficult or costly to update. For example, SEC systems are typically trained using supervised machine learning techniques to identify a specific set of sounds identified in labeled training data. As a result, each SEC system tends to be domain-specific (e.g., capable of classifying a predetermined set of sounds). After the SEC system has been trained, updating it to identify new sound categories not identified in the labeled training data is difficult. Additionally, some sound categories that the SEC system is trained to detect may represent sound events with more variations than those represented in the labeled training data. While user experience can be improved by updating these models to increase accuracy in the environments in which user devices are typically exposed, the training involved in updating these models can be very time-consuming and require large amounts of data, and the number of different categories that these models are updated to identify (e.g., different new sounds for the SEC system) can grow rapidly, consuming significant amounts of memory on the device.

[0008] IV. Overview

[0009] In one aspect, a device includes one or more processors configured to receive sensor data from one or more sensor devices. The one or more processors are further configured to determine the context of the device based on the sensor data. The one or more processors are further configured to select a model based on the context. The one or more processors are also configured to use the model to process an input signal to generate a context-dependent output. In one aspect, a method includes receiving sensor data from one or more sensor devices at one or more processors of the device. The method includes determining the context of the device based on the sensor data at one or more processors. The method includes selecting a model based on the context at one or more processors. The method also includes using the model at one or more processors to process an input signal to generate a context-dependent output. In one aspect, a device includes means for receiving sensor data. The device includes means for determining a context based on the sensor data. The device includes means for selecting a model based on the context. The device also includes means for processing an input signal using the model to generate a context-dependent output.

[0010] In one particular aspect, a non-transient computer-readable storage medium includes instructions that, when executed by a processor, cause the processor to receive sensor data from one or more sensor devices. These instructions, when executed by the processor, cause the processor to determine a context based on the sensor data. These instructions, when executed by the processor, cause the processor to select a model based on the context. These instructions, when executed by the processor, also cause the processor to use the model to process input signals to generate an output that varies depending on the context.

[0011] Other aspects, advantages, and features of this disclosure will become apparent upon reading the entire application, which comprises the following sections: brief description of the drawings, detailed description, and claims.

[0012] V. Brief Description of the Attached Diagram

[0013] Figure 1 This is a block diagram of an example system comprising a device configured to perform context-based model selection, based on some examples of this disclosure.

[0014] Figure 2 This is an explanation based on some examples from this disclosure. Figure 1 The diagram illustrates various aspects of the operation of the components.

[0015] Figure 3 The explanation based on some examples of this disclosure can be derived from Figure 1 The diagram illustrates various aspects of the device's updated model.

[0016] Figure 4 This is a diagram illustrating the updating of the sound event classification model to take into account various aspects of drift, based on some examples of this disclosure.

[0017] Figure 5 This is a diagram illustrating the updating of the sound event classification model to take into account various aspects of the new sound categories, based on some examples of this disclosure.

[0018] Figure 6 The explanation based on some examples of this disclosure can be derived from Figure 1 A diagram illustrating a specific example of the update model performed by the device.

[0019] Figure 7 The explanation based on some examples of this disclosure can be derived from Figure 1 This is a diagram illustrating another specific example of the update model performed by the device.

[0020] Figure 8 This is an explanation based on some examples from this disclosure. Figure 1 A block diagram of a specific example of the device.

[0021] Figure 9 It is based on the inclusion of some examples of this disclosure. Figure 1 Explanatory examples of various aspects of the equipment and vehicles.

[0022] Figure 10 The inclusion of some examples according to this disclosure is explained. Figure 1 Virtual reality, mixed reality, or augmented reality headsets are devices that cover various aspects of the virtual reality, mixed reality, or augmented reality experience.

[0023] Figure 11 The inclusion of some examples according to this disclosure is explained. Figure 1 Wearable electronic devices in all aspects of the device.

[0024] Figure 12 The inclusion of some examples according to this disclosure is explained. Figure 1 The various aspects of the equipment's voice-controlled speaker system.

[0025] Figure 13 The inclusion of some examples according to this disclosure is explained. Figure 1 The equipment includes cameras in various aspects.

[0026] Figure 14 The inclusion of some examples according to this disclosure is explained. Figure 1 The devices are mobile devices in all aspects.

[0027] Figure 15 The inclusion of some examples according to this disclosure is explained. Figure 1 Hearing aid devices in all aspects of the equipment.

[0028] Figure 16 The inclusion of some examples according to this disclosure is explained. Figure 1 The various aspects of the aerial equipment.

[0029] Figure 17 The inclusion of some examples according to this disclosure is explained. Figure 1 The head-mounted devices for all aspects of the equipment.

[0030] Figure 18 The inclusion of some examples according to this disclosure is explained. Figure 1 The electrical components of the equipment.

[0031] Figure 19 This is an explanation based on some examples from this disclosure. Figure 1 A flowchart illustrating an example of how the device operates.

[0032] Figure 20 This is an explanation based on some examples of this disclosure. Figure 1 A flowchart illustrating another operating method of the device.

[0033] Figure 21 This is an explanation based on some examples of this disclosure. Figure 1 A flowchart illustrating another operating method of the device.

[0034] Figure 22 This is an explanation based on some examples of this disclosure. Figure 1 A flowchart illustrating another operating method of the device.

[0035] Figure 23 This is an explanation based on some examples of this disclosure. Figure 1 A flowchart illustrating another operating method of the device.

[0036] Figure 24 This is an explanation based on some examples of this disclosure. Figure 1 A flowchart illustrating another operating method of the device.

[0037] VI. Detailed Description

[0038] Systems performing operations (such as SEC, noise reduction, ASR, NLP, etc.) can use models trained to provide broad applicability but which are difficult or costly to update. While user experience can be improved by updating models to enhance accuracy in the environments in which users or their devices are typically exposed, retraining such models can be very time-consuming and require large amounts of data. Furthermore, as users encounter more and more environments and situations in their daily lives, updating models to adapt to new environments and situations leads to a continuously increasing amount of memory consumed by the models.

[0039] According to some aspects, the disclosed systems and methods use context-aware model selection to choose from multiple models based on the context in which the selected model will be used (e.g., the acoustic environment). For example, a suitable acoustic model can be selected for a specific acoustic environment to deliver a better user experience in most voice user interface applications. As an explanation, to provide a pleasant listening experience for users of hearing aids, ambient noise should be accurately estimated. As an explanatory, non-limiting example, an appropriate model can be selected based on the user's specific location (such as a particular street corner, building, or restaurant) or based on the type of environment the user is in (such as in a vehicle, a park, or a subway station). A particular context can be identified based on one or more of a wide variety of techniques, such as location data (e.g., from a Global Positioning System (GPS) system), activity detection data, camera recognition, audio classification, user input, one or more other techniques, or any combination thereof.

[0040] In some respects, the acoustic characteristics of noisy areas (such as shopping malls, restaurants, stadiums, etc.) can be known, making their models publicly available to users. When a user enters any of these locations, access to the model for that location can be granted. After the user leaves, the model can be removed from the user's device or "pruned." On the other hand, as a user moves from one place to another (e.g., by car, on foot, or via public transport), the model on the user's device can be changed to allow the most appropriate model to be used for each location or setting the user encounters. In some respects, a library of available models is provided to enable the searching and downloading of appropriate models. Some models can be uploaded by other users (such as updated models already trained based on other users' devices exposed to various environments) and can be publicly available as part of a crowdsourced context-aware model library (or with specific access permissions).

[0041] Depending on a specific aspect, models can be combined or "generalized" by grouping categories of related phyla into a single model and creating a collection of various source models. For example, in SEC applications, related sound categories can be grouped based on location, sound type, or one or more other characteristics. To illustrate, one model might include a group of sound categories typically representing crowded areas (such as public squares, shopping malls, subways, etc.), while another model might include a group of sound categories related to family activities. These generalized models achieve generally improved performance based on the broad phyla of the generalized model, while using less memory compared to the amount of memory required to adapt multiple specific models to each specific environment or activity. Furthermore, if a specific model is unavailable due to privacy concerns or other accessibility limitations, a more general model can be used instead. For example, if a user arrives at a busy restaurant and there is no specific public model available for that restaurant, a general model for crowded areas or a general model for crowded restaurants can be downloaded to the user's device and used instead.

[0042] By modifying the model based on the device's context, systems using such models can operate with higher accuracy compared to using a single model for all contexts. Furthermore, modifying the model allows such systems to operate with improved accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with retraining an existing model from scratch on the device for specific contexts. Using a context-based generalized model achieves improved system performance compared to using a default model, and also reduces bandwidth, memory, and processing resource usage compared to downloading multiple highly accurate, context-dependent models and switching between them. Moreover, the operation of such systems using context-based models enables improved operation of the device itself, such as faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a higher-accuracy model specific to a particular context.

[0043] Specific aspects of this disclosure are described below with reference to the accompanying drawings. In this specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms “a,” “some,” and “the” are intended to also include the plural forms, unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some implementations and plural in others. For the purpose of explanation, Figure 1 It describes a system that includes one or more processors ( Figure 1The term "(various) processors" in the context of device 101 indicates that in some implementations, device 101 includes a single processor 110, while in other implementations, device 101 includes multiple processors 110. For ease of reference herein, such features are generally described as "one or more" features and are subsequently referred to in the singular or optional plural (usually indicated by a term ending with "(various)"), unless an aspect relating to multiple features of these features is being described.

[0044] The terms “comprising,” “having,” and “containing” are used interchangeably with “including,” “with,” or “owning” herein. Additionally, the term “wherein” is used interchangeably with “in this case.” As used herein, “exemplary” indicates an example, an implementation, and / or aspect, and should not be construed as limiting or indicating a preference or preferred implementation. As used herein, ordinal numbers used to modify elements (such as structures, components, operations, etc.) (e.g., “first,” “second,” “third,” etc.) do not themselves indicate any priority or order of that element relative to another element, but merely distinguish that element from another element with the same name (but using an ordinal number). As used herein, the term “set” refers to one or more specific elements, while the term “multiple” refers to multiple specific elements (e.g., two or more specific elements).

[0045] As used herein, “coupling” can include “communicationally coupled,” “electrically coupled,” or “physical coupled,” and may additionally (or alternatively) include any combination thereof. Two devices (or components) may be coupled directly or indirectly (e.g., communicationally coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof). Electrically coupled two devices (or components) may be included in the same or different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative and non-limiting examples. In some implementations, communicationally coupled two devices (or components) (such as in electrical communications) may directly or indirectly transmit and receive electrical signals (e.g., digital or analog signals) via one or more wires, buses, networks, etc. As used herein, “direct coupling” means that two devices are coupled without an intermediary component (e.g., communicationally coupled, electrically coupled, or physically coupled).

[0046] In this disclosure, terms such as “determine,” “calculate,” “estimate,” “shift,” and “adjust” may be used to describe how one or more operations are performed. It should be noted that such terms should not be construed as restrictive, and similar operations may be performed using other techniques. Furthermore, as referenced herein, “generate,” “calculate,” “estimate,” “use,” “select,” “access,” and “determine” may be used interchangeably. For example, “generate,” “calculate,” “estimate,” or “determine” a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or it may refer to using, selecting, or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0047] Figure 1 This is a block diagram of an example system including device 100 configured to perform context-based model selection. Device 100 includes one or more processors 110 coupled to memory 108. Memory 108 includes L available models 114 (L is an integer greater than 1) that can be selected by one or more processors 110, which are interpreted as a first model 116 and one or more additional models including the Lth model 118.

[0048] One or more processors 110 are configured to receive sensor data 138 from one or more sensor devices 134 and determine the context 142 of device 100 based on the sensor data 138. Although sensor devices 134 are interpreted as being coupled to device 100, in other implementations, one or more (or all) sensor devices 134 are integrated with or included in device 100.

[0049] One or more sensor devices 134 include one or more microphones 104 coupled to one or more processors 110, and sensor data 138 includes audio data 105 from the one or more microphones 104. In one example, the audio data 105 corresponds to an audio scene, and the context 142 is at least partially based on the audio scene. To illustrate, based on the amount and type of noise detected in the audio data, and acoustic characteristics such as echo and absorption, the audio scene can indicate that the device 100 is in a confined noisy space, a large enclosed space, a large outdoor space, a moving vehicle, etc.

[0050] One or more sensor devices 134 include a position sensor 152 coupled to one or more processors 110, and sensor data 138 includes position data 153 from the position sensor 152 (such as a global positioning sensor that provides global position data for device 100). In one example, the position data 153 indicates the location of device 100, and the context 142 is based at least in part on that location.

[0051] One or more sensor devices 134 include a camera 150 coupled to one or more processors 110, and sensor data 138 includes image data 151 (e.g., still image data, video data, or both) from the camera 150. In one example, image data 151 corresponds to a visual scene, and context 142 is at least partially based on that visual scene.

[0052] One or more sensor devices 134 include an activity detector 154 coupled to one or more processors 110, and sensor data 138 includes motion data, such as activity data 155 from the activity detector 154. In one example, the motion data corresponds to the motion of device 100, and context 142 is based at least in part on the motion of device 100.

[0053] One or more sensor devices 134 may also include one or more other sensors 156 that provide additional sensor data 157 to one or more processors 110 for use in determining context 142. These other sensors 156 may include, for example, orientation sensors, magnetometers, light sensors, contact sensors, temperature sensors, or any other sensors coupled to or included within device 100 and that may be used to generate sensor data 157 useful for determining the context 142 associated with device 100 at a particular time. As another example, these other sensors 156 may include a wireless network detector that may be used to determine context 142, such as by detecting when device 100 is near an identified wireless network location (e.g., by detecting a home or business Wi-Fi or Bluetooth network associated with a friend or family member of the user of device 100).

[0054] One or more processors 110 include a context detector 140, a model selector 190, and a model-based application 192. In a particular implementation, the context detector 140 is a neural network trained to determine context 142 based on sensor data 138. In other implementations, the context detector 140 is a classifier trained using different machine learning techniques. For example, the context detector 140 may include or correspond to a decision tree, random forest, support vector machine, or another classifier trained to generate an output indicating context 142 based on sensor data 138. In yet another implementation, the context detector 140 uses a heuristic approach to determine context 142 based on sensor data 138. In yet another implementation, the context detector 140 uses a combination of artificial intelligence and heuristics to determine context 142 based on sensor data 138. For example, sensor data 138 may include image data, video data, or both, and the context detector 140 may include an image recognition model trained using machine learning techniques to detect specific objects, motion, background, or other image or video information. In this example, the output of the image recognition model may be evaluated via one or more heuristics to determine context 142.

[0055] Model selector 190 is configured to select model 112 based on context 142. In some implementations, model 112 is selected from a plurality of available models 114 stored in memory 108. In some implementations, model 112 is selected from a model library 162 accessible via network 160 (such as a cloud-based model library that can be searched and downloaded). An example of model library 162 will be provided in [reference]. Figure 2 To describe in further detail.

[0056] As used herein, the "download" and "upload" models involve transmitting data (e.g., compressed data) corresponding to the model over wired links, wireless links, or combinations thereof. For example, a wireless local area network ("WLAN") can be used as a replacement for or supplement to a wired network. Wireless technologies (such as...) Bluetooth and Wi-Fi, or variations of Wi-Fi (e.g., Wi-Fi Direct), enable high-speed communication between mobile electronic devices (e.g., cellular phones, watches, headsets, remote controls, etc.) over relatively short distances (e.g., 100 to 200 meters or less, depending on the specific wireless technology). Wi-Fi is often used to connect and exchange information between devices with an access point (e.g., routers) and Wi-Fi-enabled devices. Examples of such devices include smart TVs, laptops, thermostats, personal assistants, home automation devices, wireless speakers, and other similar devices. Similarly, Bluetooth is used to couple devices together. Examples of such devices are mobile phones, computers, digital cameras, wireless headsets, keyboards, mice, or other input peripherals, and similar devices.

[0057] Devices (such as those previously mentioned) may have Bluetooth and Wi-Fi capabilities, or other wireless means of communication with each other. Networked devices may have wireless means of communication with each other and may also connect based on different cellular communication systems, such as Long Term Evolution (LTE), Code Division Multiple Access (CDMA), Global System for Mobile Communications (GSM), Wireless Local Area Network (WLAN), or other wireless systems. CDMA systems may implement Wideband CDMA (WCDMA), CDMA1X, Evolved Data Optimized (EVDO), Time Division Synchronous CDMA (TD-SCDMA), or some other version of CDMA. As used herein, “wireless” means one or more of the technologies listed above, one or more other technologies capable of transmitting information via means other than wired connections, or a combination thereof.

[0058] Model-based application 192 is configured to use the selected model 112 to process input signal 106 to generate output 122 that varies depending on the context. Although a single model-based application 192 has been described, in other implementations, one or more processors 110 may use various models 112 selected based on context 142 to execute multiple model-based applications 192 to perform various operations.

[0059] For example, in some implementations, model 112 includes a sound event detection model, and input signal 106 includes audio signals such as audio data 105 from microphone 104, audio data retrieved from an audio file at memory 108, audio signals received via wireless transmission (such as a telephone call or streaming audio session), or any combination thereof. Model-based application 192 is configured to use the sound event detection model to process input signal 106 to generate context-dependent output 122 including a classification of sound events in the audio signal.

[0060] In some implementations, model 112 includes a noise reduction model, and input signal 106 includes audio signals such as audio data 105 from microphone 104, audio data retrieved from an audio file at memory 108, audio signals received via wireless transmission (such as telephone calls or streaming audio sessions), or any combination thereof. Model-based application 192 is configured to use the noise reduction model to process input signal 106 to generate context-dependent output 122 including a noise-reduced audio signal based on the audio signal.

[0061] In some implementations, model 112 includes an Automatic Speech Recognition (ASR) model, and input signal 106 includes audio signals such as audio data 105 from microphone 104, audio data retrieved from an audio file at memory 108, audio signals received via wireless transmission (such as telephone calls or streaming audio sessions), or any combination thereof. Model-based application 192 is configured to use the ASR model to process input signal 106 to generate context-dependent output 122 including text data representing speech in the audio signals.

[0062] In some implementations, model 112 includes a natural language processing (NLP) model, and input signal 106 includes text data, such as text data generated by the NLP model, user keyboard input, text messages, etc. Model-based application 192 is configured to use the NLP model to process input signal 106 to generate context-dependent output 122, which includes text-based NLP output data.

[0063] In some implementations, model 112 is associated with automatic adjustment of device operating modes. For example, model 112 may map or otherwise associate user input (e.g., voice commands, gestures, touchscreen selections, etc.) with operating mode adjustment based on context 142. To illustrate, when model 112 is selected based on context 142 corresponding to a public area, model 112 may cause model-based application 192 to map the user command "play music" to playback operation at the user's headphones, and context-dependent output 122 may include signals for adjusting device operating modes to initiate audio playback operation and routing output audio signals to the user's headphones. However, when model 112 is selected based on context 142 corresponding to the user's home, model 112 may cause model-based application 192 to map the "play music" command to playback operation at the user's home entertainment system, and context-dependent output 122 may include signals for adjusting device operating modes to initiate audio playback and routing output audio signals to the speakers of the home entertainment system.

[0064] In some implementations, model selector 190 is configured to prune model 112 in response to detecting a change in context 142 (such as when a change in context 142 causes model 112 to no longer be applicable to the changed context 142 or to be less applicable to the changed context 142 compared to another available model). As used herein, “pruning” a model includes stopping the use of the model, such as by replacing the model with another model at model-based application 192, permanently deleting the model from memory 108, removing the model from model-based application 192, marking the model as unused, or otherwise making the model inaccessible (e.g., by removing access permissions to the model, as referenced). Figure 2 Further described), or combinations thereof. In some implementations, pruning the model can reduce memory usage, processor cycles, one or more other processing or system resources, or combinations thereof, and thus can improve the operation of one or more processors 110 or the device 100 as a whole (e.g., increased speed, reduced power consumption). In implementations where the model has been updated at device 100, pruning the model may include retaining updates to the model. For example, as an illustrative, non-limiting example, if a sound model has been created or modified during use at device 100 to identify new sound categories (such as references) Figure 3-5 If described, these new sound categories can be preserved (e.g., in memory 108 or uploaded to model library 162).

[0065] During operation, the context detector 140 monitors sensor data 138 and can update the context 142 based on detected changes in the sensor data 138. In response to a change in the context 142, the model selector 190 accesses available models 114 in memory 108, model library 162, or both to select one or more models that may be more suitable for the updated context 142 than the current model 112. For example, the model selector 190 may send a query to the model library 162 identifying one or more aspects of the context 142 (such as geographic location, place name, acoustic or visual scene description, one or more other aspects, or a combination thereof). The model selector 190 may receive the query results, select a specific model based on the query results, and cause the device 100 to download the specific model from the model library 162 and store the downloaded model in memory 108. In some implementations, in response to the detection that the context 142 has changed, the currently selected model 112 is pruned (e.g., the model selector 190 causes the model-based application 192 to stop using model 112) and replaced by a model downloaded from memory 108.

[0066] By modifying the model based on context 142, device 100 can execute model-based applications 192 with higher accuracy compared to using a single default model. As a result, the user experience of device 100 can be improved.

[0067] In some implementations, device 100 is also configured to modify or customize the selected model 112 to further improve accuracy, such as in response to the detection of a new sound event or a variant of an existing classification that might not be accurately identified by model 112. An example of modifying model 112 based on data acquired at device 100 (e.g., sensor data 138) will be referenced. Figure 3-5 Further description. After modifying model 112, device 100 can upload the modified model 112 to model library 162 for use by other devices. In some implementations, model 112 includes a trained model uploaded to library 162 from another user device. Thus, device 100 can utilize and contribute to a crowdsourced model library in a distributed context-aware system. Device 100 may include, correspond to, or be included in the following: voice-activated devices, audio devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, hearing aid devices, smart speakers, mobile computing devices, mobile communication devices, smartphones, cellular phones, laptops, computers, tablet devices, personal digital assistants, display devices, televisions, game consoles, electrical appliances, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, or any combination thereof. In one particular aspect, one or more processors 110, memory 108, or combinations thereof are included in an integrated circuit. Various implementations including aspects of device 100 will be referenced. Figure 7-16 Further description.

[0068] Although device 100 is described as storing available models 114 in memory 108 and accessing model library 162, in other implementations, device 100 may not store models in memory 108 and may instead retrieve models from model library 162 in response to detecting a change in context 142. In other implementations, device 100 operates without accessing model library 162 and instead selects from the available models 114 at memory 108. In some implementations, the available models 114 at memory 108 represent a portion of the local storage of the distributed context-aware system and may be accessed by other devices as part of model library 162, such as in a peer-to-peer model-sharing configuration.

[0069] Although sensor 134 is described as including microphones 104, cameras 150, position sensors 152, activity detectors 154, and other sensors 156, in other implementations, one or more of microphones 104, cameras 150, position sensors 152, activity detectors 154, or other sensors 156 are omitted. In the illustrative example, context detector 140 operates using only audio data 105, image data 151, position data 153, activity data 155, or the other sensor data 157. In another illustrative example, the context detector 140 operates using two of the following: audio data 105, image data 151, location data 153, activity data 155, and the other sensor data 157; three of the following: audio data 105, image data 151, location data 153, activity data 155, and the other sensor data 157; or four of the following: audio data 105, image data 151, location data 153, activity data 155, and the other sensor data 157.

[0070] Although the context detector 140, model selector 190, and model-based application 192 are described as separate components, in other implementations, the context detector 140 and model selector 190 are combined into a single component, the model selector 190 and model-based application 192 are combined into a single component, or the context detector 140, model selector 190, and model-based application 192 are combined into a single component. In some implementations, each of the context detector 140, model selector 190, and model-based application 192 can be implemented via processor-executed instructions, dedicated hardware or circuitry, or a combination of both.

[0071] Figure 2 Specific examples illustrating various aspects of the operation of device 100 and model library 162 are given. Device 100 is described as being located within building 202 (such as an office building), which includes room 204, room 206, and elevator 208. Device 100 is in wireless communication with model library 162.

[0072] Model library 162 includes various types of models, such as representative sound event detection model 220, representative noise reduction model 222, representative ASR model 224, representative NLP model 226, representative operating mode adjustment model 228, and various acoustic models 250. Although for clarity, each of the sound event detection model 220, noise reduction model 222, ASR model 224, NLP model 226, and operating mode adjustment model 228 is explained individually, it should be understood that model library 162 may include multiple versions of each of these different types of models, such as models that have been trained for different contexts, different levels of personalization, and different levels of generality in a manner similar to that described below regarding acoustic model 250.

[0073] Acoustic model 250 is illustrated by an arrangement in which models for general categories (e.g., generalized models) are depicted as the roots of a tree structure, and models for more specific contexts are depicted as branches or leaves of the tree structure. For example, the “congested area” model 252 is a general category model whose branches include (e.g.) “highway” models, (e.g.) “metro center” models, (e.g.) “suburb” models, (e.g.) “theme park” models, (e.g.) “shopping mall” models, and (e.g.) “public square” models. Although not explicitly described, each of these branch models can be used as a general category model for various more specific models. For example, model library 162 may include acoustic models for multiple specific theme parks. If a user’s device (e.g., device 100) is located at a particular theme park, the device can request an acoustic model for that particular theme park. If no acoustic model is available for that particular theme park, the user’s device can request a “theme park” model that is broadly applicable to theme parks but not specifically for any particular theme park. If the "Theme Park" model is unavailable, the user's device can request the "Crowded Area" model, which is broadly applicable to crowded areas but not specifically for theme parks. Therefore, the user's device can search and select the most specific model from the model library 162 that is available for the device's specific context.

[0074] Acoustic model 250 also includes a "closed space" model 254, which is a general category model with branches corresponding to the "office building" model 262, the "house" model 264, and the "vehicle" model 266. The "office building" model 262 is a general category model for various locations within an office building. More specific models for various locations within an office building include the "lobby" model 270, the "elevator" model 272, and the "office" model 274. The "house" model 264 is a general category model for various locations within a house. More specific models for locations within a house include the "kitchen" model, the "room" model, and the "garage" model. The "vehicle" model 266 is a general category model for various locations within a vehicle. More specific models for locations within a vehicle include the "driver's seat" model, the "passenger seat" model, and the "rear seat" model.

[0075] It should be understood that the models described are depicted for the purpose of explanation and clarity. In other implementations, model library 162 may have any number (e.g., hundreds, thousands, millions, etc.) of models arranged at any number of generality levels, for any number of different contexts, and for any number of different applications. It should also be understood that although acoustic model 250 is organized according to a tree structure, in other implementations, model library 162 utilizes one or more other data structures or classification techniques as a substitute for or supplement to the tree structure.

[0076] As explained, device 100 determines that its context 142 is within room 204 based on sensor 134. Device 100 may determine whether the available acoustic models 250 of model library 162 include acoustic models specific to the particular acoustic environment associated with context 142 and available to device 100 (e.g., model selector 190). For example, device 100 transmits data indicating the acoustic environment 210 of device 100, such as location data, the name of building 202, a general description of building 202 (e.g., “office building”), a general description of room 204 (e.g., “office”), or any combination thereof. In a particular implementation, in response to model library 162 not having acoustic models specific to a particular acoustic environment 210 and available to one or more processors 110, device 100 (e.g., model selector 190) determines whether acoustic models of a general category for the particular acoustic environment 210 are available.

[0077] As an illustrative example, device 100 can send data indicating the acoustic environment 210 by transmitting the location coordinates of device 100. If model library 162 has an acoustic model specific to (e.g., matching) the location of device 100 (e.g., an acoustic model corresponding to a geofence containing the location of device 100), device 100 downloads the acoustic model 212 corresponding to that location. Otherwise, in response to model library 216 not having a model specific to location coordinates (e.g., not specific to building 202), device 100 can transmit additional data indicating the acoustic environment 210, such as an “office” descriptor for room 204. In response to determining that model library 162 includes an “office” model 274 available to device 100, device 100 downloads the “office” model 274 as acoustic model 212 for use within room 204. If the “office” model 274 is unavailable, device 100 can request a more general “office building” model 262, or even a more general “enclosed space” model 254. In some implementations, the model library 162 is configured to automatically locate and send the most specific model corresponding to the acoustic environment 210 to the device 100, rather than the device 100 sending a series of requests for increasingly generalized models until the appropriate model is located.

[0078] In some implementations, device 100 may also receive one or more access permissions 214 authorizing device 100 to access acoustic model 212. Access permissions 214 may enable the pre-reservation of models predicted for use by device 100. For example, a pre-reserved model may be downloaded from model library 162 to device 100's memory 108 before predicted use. Pre-downloading such models may be scheduled based on available bandwidth (e.g., during periods of reduced network traffic) or to reduce the latency for accessing the model when a change in device 100's context 142 is detected. Each of the pre-reserved models remains inaccessible (e.g., encrypted) at memory 108 until a corresponding permission 214 (e.g., encryption key) for the model is received from model library 162 or another permission management system. For example, device 100 may receive access permission 214 for model 112 at least in part based on a match between device 100's location and a specific location (e.g., a specific location) associated with model 112.

[0079] In a specific example, device 100 transmits data indicating the acoustic environment 210 of room 204, receives an "office" model 274 (and any associated access permission 214) from model library 162, stores a copy of the "office" model 274 at memory 108, and uses the "office" model 274 at model-based application 197 (e.g., to perform noise reduction). When device 100 is moved from room 204 to elevator 208 within building 202, device 100 detects a new context 142 and requests an acoustic model for the acoustic environment 210 corresponding to elevator 208. In response, "elevator" model 272 is transmitted to device 100 as acoustic model 212, and access permission 214 for "elevator" model 272 may also be transmitted. Device 100 replaces "office" model 274 with "elevator" model 272 at model-based application 192. In some implementations, device 100 removes “office” model 274 from memory 108, such as when the available storage capacity at memory 108 is constrained.

[0080] Upon exiting elevator 208 and entering room 206, device 100 can search for available models 114 in memory 108 to locate the “office” model 274. If the “office” model is not available in memory 108, device 100 transmits data indicating the acoustic environment 210 of room 206, receives the “office” model 274 (and any associated access permissions 214) from model library 162, stores a copy of the “office” model 274 in memory 108, and uses the “office” model 274 in model-based application 197. Thus, as device 100 moves from one location to the next, the model is switched to a more appropriate model for the changing context 142 of device 100. In some implementations, when device 100 exits building 202, any building 202-specific stored models in memory 108 can be deleted or archived to save storage space in memory 108, or they can be made inaccessible in response to geographical or other constraints imposed on the use of the model by access permissions 214.

[0081] Combined with reference Figure 1 and Figure 2 In all aspects described, device 100 includes one or more processors 110 configured to select an acoustic model corresponding to a specific room in a building where device 100 is located (such as acoustic model 212 corresponding to room 204 in building 202), and to use acoustic model 212 to process input audio signals. For example, input signal 106 may include audio data 105 generated by microphones 104 and processed by model-based application 192 to perform noise reduction.

[0082] In some implementations, one or more processors 110 are configured to download an acoustic model 212 from an acoustic model library 162 in response to determining that device 100 has entered the particular room 204. In some implementations, one or more processors 110 are further configured to remove the acoustic model 212 in response to device 100 leaving the particular room 204. As a non-limiting example, removing the acoustic model 212 may include: replacing the acoustic model 212 with another model at a model-based application 192, deleting the acoustic model 212 from memory 108, deleting the acoustic model 212 from the model-based application 192, marking the acoustic model 212 as unused, or making the acoustic model 212 inaccessible (e.g., removing access permission 214 for the acoustic model 212).

[0083] One or more sensor devices 134 (also referred to as “sensors 134”) coupled to one or more processors 110 are configured to generate sensor data 138 indicating the location of device 100, and one or more processors 110 are configured to select an acoustic model 212 based on the sensor data 138. For example, device 100 may include a modem (as referenced). Figure 6 As described, the modem is coupled to one or more processors 110 and configured to receive location data 153 indicating the location of device 100, and the one or more processors 110 are configured to select an acoustic model 212 based on the location data 153.

[0084] In some implementations, model selection can be performed predictively based on context 142. For example, based on sensor data 138 (e.g., activity detection, GPS analysis, camera recognition, audio classification, or a combination thereof), one or more processors 110 can determine that a user of device 100 is moving to a new location (e.g., New York City), where assisted Internet of Things (IoT) devices associated with the user can exhibit improved performance through updated settings for the new location. As a result, an appropriate source model (e.g., an acoustic model for traffic) can be retrieved and used from memory 108 or from one or more model libraries (e.g., model library 162). Upon leaving the new location, the source model for the new location is removed and the previous source model can be restored.

[0085] Combined with reference Figure 1 and Figure 2 The various aspects described herein, in response to device 100 entering a vehicle (such as further references) Figure 7As described, one or more processors 110 are configured to select a personalized acoustic model for a user of device 100 from a plurality of personalized acoustic models corresponding to a vehicle, and to use that personalized acoustic model to process the input audio signal. For example, device 100 may train or otherwise generate a user-specific model specific to device 100 (as described in reference). Figures 3-5 (As described in further detail), and the personalized acoustic model can be accessed from model library 162, memory 108, or both. For illustration, one or more processors 110 may be configured to download a personalized acoustic model (e.g., acoustic model 250 in model library 162) from the acoustic model library in response to determining that device 100 has entered a vehicle. In some implementations, one or more processors 110 are further configured to remove the personalized acoustic model in response to device 100 leaving the vehicle.

[0086] For example, one or more processors 110 may be configured to determine that device 100 has entered a vehicle based on sensor data 138. Alternatively, one or more processors 110 may determine that device 100 has entered (or exited) a vehicle based on location data 153.

[0087] Combined with reference Figure 1 and Figure 2 In all the aspects described, one or more processors 110 of device 100 are configured to download an acoustic model corresponding to a specific location in which device 100 is located, use the acoustic model to process input audio signals, and remove the acoustic model in response to device 100 leaving the location. In the illustrative example, the location corresponds to a specific restaurant, and the acoustic model is downloaded from an acoustic model library (e.g., acoustic model 250 in model library 162) in response to determining that device 100 has entered the specific restaurant.

[0088] Combined with reference Figure 1 and Figure 2 In all aspects described, one or more processors 110 of device 100 are configured to select an acoustic model corresponding to a specific location, receive access permission for the acoustic model at least in part based on the matching of the location of device 100 with the specific location, and use the acoustic model to process input audio signals.

[0089] In some implementations, as illustrative, non-restrictive examples, Figure 1 and Figure 2 The device 100 is further configured to update one or more models, such as to personalize a model for a particular user or to provide accuracy for a model for environments frequently encountered by the device 100. Figure 3-5 An illustrative example is depicted in which device 100 is configured to update the model. Although Figure 3-5 An updated sound event classification model is described as a specific example, but the described techniques are generally applicable to any type of model that the updated device 100 can use.

[0090] Figure 3 This is a block diagram of an example of the components of device 100, which is configured to generate sound identification data in response to audio data sample 310 and is configured to update the sound event classification model. Figure 3 The device 100 includes one or more microphones 304 (e.g., microphones 104) configured to generate an audio signal 306 (e.g., audio data 105) based on sound 302 detected within an acoustic environment. The microphones 304 are coupled to a feature extractor 308 that generates audio data samples 310 based on the audio signal 306. For example, the audio data sample 310 may include an array or matrix of data elements, where each data element corresponds to a feature detected in the audio signal 306. As a specific example, the audio data sample 310 may correspond to Mel-spectral features extracted from a one-second audio signal 306. In this example, the audio data sample 310 may include a 128x128 matrix of eigenvalue elements. In other examples, other audio data sample configurations or sizes may be used.

[0091] Audio data sample 310 is provided to sound event classification (SEC) engine 320 (e.g., model-based application 192). SEC engine 320 is configured to perform inference operations based on one or more SEC models (such as SEC model 312). An "inference operation" refers to assigning audio data sample 310 to a sound category identified by SEC model 312. For example, SEC engine 320 may include or correspond to software implementing a machine learning runtime environment, such as the Qualcomm Neural Processing SDK, which is available from Qualcomm Technologies, Inc., San Diego, California, USA. In one particular aspect, SEC model 312 is one of several SEC models (e.g., available SEC model 314) that can be used with SEC engine 320.

[0092] In a specific example, each available SEC model 314 (e.g., stored at memory 108 or model library 162) includes or corresponds to a neural network trained as a sound event classifier. For illustration, SEC model 312 (and each of the other available SEC models 314) may include an input layer, one or more hidden layers, and an output layer. In this example, the input layer is configured to correspond to an array or matrix of values ​​for the audio data sample 310 generated by feature extractor 308. For illustration, if the audio data sample 310 includes 15 data elements, the input layer may include 15 nodes (e.g., one node per data element). The output layer is configured to correspond to the sound categories that the SEC model 312 is trained to identify. The specific arrangement of the output layer can vary depending on the information to be provided as output. As an example, the SEC model 312 may be trained to output an array comprising one bit per sound category, wherein the output layer performs "one-hot encoding" such that all bits of the output array except one bit have a value of 0, and the bit corresponding to the detected sound category has a value of 1. Other output schemes can be used to indicate, for example, a confidence metric for each sound category, where the confidence metric indicates the probability estimate of the audio data sample 310 corresponding to the corresponding sound category. To illustrate, if the SEC model 312 is trained to recognize four sound categories, the SEC model 312 can generate output data including four values ​​(one for each sound category), and each value can indicate the probability estimate of the audio data sample 310 corresponding to the corresponding sound category.

[0093] Each hidden layer comprises multiple nodes, and each node is interconnected (via links) with other nodes in the same or different layers. Each input link of a node is associated with a link weight. During operation, a node receives input values ​​from other nodes to which it is linked, weights the input values ​​based on the corresponding link weights to determine a combined value, and substitutes the combined value into an activation function to generate the node's output value. The output value is provided to one or more other nodes via the node's output links. Nodes may also include bias values ​​used to generate the combined value. Nodes can be linked in various arrangements and can include various other features (e.g., prior memory) to facilitate the processing of specific data. In the case of audio data samples, a convolutional neural network (CNN) can be used. For illustration, one or more of the SEC model 312 may include three linked CNNs, and each CNN may include two-dimensional (2D) convolutional layers, max-pooling layers, and batch normalization layers. In other implementations, the hidden layers comprise different numbers of CNNs or other layers. Training the neural network involves modifying the link weights to reduce the output error of the neural network.

[0094] During operation, the SEC engine 320 can provide audio data sample 310 as input to a single SEC model (e.g., SEC model 312), to multiple selected SEC models (e.g., SEC model 312 and the Kth SEC model 318 among available SEC models 314), or to each SEC model (e.g., SEC model 312, the first SEC model 316 among available SEC models 314, the Kth SEC model 318, and any other SEC model). For example, the SEC engine 320 (or another component of device 100) can select SEC model 312 from the available SEC models 314 based on factors such as user input, device settings associated with device 100, sensor data, the time when the audio data sample 310 is received, or other factors. In this example, the SEC engine 320 can choose to use only SEC model 312 or can choose to use two or more SEC models among the available SEC models 314. To illustrate, device settings can indicate that SEC model 312 and the first SEC model 316 should be used during a specific time frame. In another example, the SEC engine 320 can (e.g., sequentially or in parallel) provide audio data samples 310 to each of the available SEC models 314 to generate output from each SEC model. In a particular aspect, the SEC models are trained to recognize different sound categories, to recognize the same sound categories in different acoustic environments, or both. For example, SEC model 312 can be configured to recognize a first set of sound categories and a first SEC model 316 can be configured to recognize a second set of sound categories, wherein the first set of sound categories is different from the second set of sound categories.

[0095] In one particular aspect, the SEC engine 320 determines whether the SEC model 312 has identified the sound category of the audio data sample 310 based on the output of the SEC model 312. If the SEC engine 320 provides the audio data sample 310 to multiple SEC models, the SEC engine 320 can determine whether any of these SEC models has identified the sound category of the audio data sample 310 based on the output of each of these SEC models. If the SEC model 312 (or another available SEC model 314) has identified the sound category of the audio data sample 310, the SEC engine 320 generates an output 324 indicating the sound category 322 of the audio data sample 310. For example, the output 324 can be sent to a display to notify the user that a sound category 322 associated with sound 302 has been detected, or it can be sent to another device or another component of device 100 and used to trigger an action (e.g., sending a command to activate a light in response to the recognition of a door closing sound).

[0096] If the SEC engine 320 determines that the SEC model 312 (and other SEC models in the available SEC models 114 for which audio data sample 310 has been provided) has not identified the sound category of the audio data sample 310, the SEC engine 320 provides a trigger signal 326 to the drift detector 328. For example, the SEC engine 320 may set a trigger flag in the memory of the device 100. In some implementations, the SEC engine 320 may also provide other data to the drift detector 328. To illustrate, if the SEC model 312 generates confidence metrics for each sound category that the SEC model 312 is trained to identify, one or more of these confidence metrics may be provided to the drift detector 328. For example, if the SEC model 312 is trained to identify three sound categories, the SEC engine 320 may provide the drift detector 328 with the highest confidence value among the three confidence values ​​output by the SEC model 312 (one for each of the three sound categories).

[0097] In one specific aspect, the SEC engine 320 determines whether the SEC model 312 has identified the sound category of the audio data sample 310 based on a confidence metric. In this specific aspect, the confidence metric for a particular sound category indicates the probability that the audio data sample 310 is associated with that particular sound category. To illustrate, if the SEC model 312 is trained to identify four sound categories, the SEC model 312 may generate an array as output comprising four confidence metrics (one value for each sound category). In some implementations, the SEC engine 320 determines that the SEC model 312 has identified sound category 322 if the confidence metric for sound category 322 of the audio data sample 310 is greater than a detection threshold. For example, the SEC engine 320 determines that the SEC model 312 has identified sound category 322 if the confidence metric for sound category 322 of the audio data sample 310 is greater than 0.90 (e.g., 90% confidence), 0.95 (e.g., 95% confidence), or some other value of the detection threshold. In some implementations, if the confidence metric for each sound category that the SEC model 312 is trained to identify is less than a detection threshold, the SEC engine 320 determines that the SEC model 312 has not identified the sound category of the audio data sample 310. For example, if each confidence metric is less than 0.90 (e.g., 90% confidence), 0.95 (e.g., 95% confidence), or some other value of the detection threshold, the SEC engine 320 determines that the SEC model 312 has not identified the sound category 322 of the audio data sample 310.

[0098] Drift detector 328 is configured to determine whether SEC model 312, which cannot identify the sound category of audio data sample 310, corresponds to audio scene 342 associated with audio data sample 310. Figure 1In the example described herein, scene detector 340 (e.g., context detector 140) is configured to receive scene data 338 (e.g., including a portion of sensor data 138) and use the scene data 338 to determine an audio scene 342 (e.g., context 142) associated with audio data sample 310. In one aspect, scene data 338 is generated based on setup data 330 indicating one or more device settings associated with device 100, the output of clock 332, sensor data from one or more sensors 334 (e.g., sensor 134), input received via input device 336, or a combination thereof. In some aspects, scene detector 340 uses information different from that used by SEC engine 320 to select SEC model 312 to determine audio scene 342. For example, if SEC engine 320 selects SEC model 312 based on the time of day, scene detector 340 could use position sensor data from the position sensors among the sensors 334 to determine audio scene 342. In some respects, the scene detector 340 uses at least some of the same information as the information used by the SEC engine 320 to select the SEC model 312, and uses additional information. To explain, if the SEC engine 320 selects the SEC model 312 based on the time of day and setting data 330, the scene detector 340 can use position sensor data and setting data 330 to determine the audio scene 342. Therefore, the scene detector 340 uses an audio scene detection mode different from the audio scene detection mode used by the SEC engine 320 to select the SEC model 312.

[0099] In one implementation, scene detector 340 is a neural network trained to determine audio scene 342 based on scene data 338. In other implementations, scene detector 340 is a classifier trained using different machine learning techniques. For example, scene detector 340 may include or correspond to decision trees, random forests, support vector machines, or another classifier trained to generate an output indicating audio scene 342 based on scene data 338. In yet another implementation, scene detector 340 uses a trial-and-error approach to determine audio scene 342 based on scene data 338. In yet another implementation, scene detector 340 uses a combination of artificial intelligence and trial-and-error to determine audio scene 342 based on scene data 338. For example, scene data 338 may include image data, video data, or both, and scene detector 340 may include an image recognition model trained using machine learning techniques to detect specific objects, motion, background, or other image or video information. In this example, the output of the image recognition model may be evaluated via one or more trial-and-error methods to determine audio scene 342.

[0100] Drift detector 328 compares the audio scene 342 indicated by scene detector 340 with information describing SEC model 312 to determine whether SEC model 312 is associated with audio scene 342 of audio data sample 310. If drift detector 328 determines that SEC model 312 is associated with audio scene 342 of audio data sample 310, drift detector 328 causes drift data 344 to be stored as model update data 348. In a particular implementation, drift data 344 includes audio data sample 310 and a label, wherein the label identifies SEC model 312, indicates the sound category associated with audio data sample 310, or both. If drift data 344 indicates the sound category associated with audio data sample 310, the sound category can be selected based on the highest confidence value generated by SEC model 312. As an illustrative example, if the SEC engine 320 uses a detection threshold of 0.90, and for a specific sound category, the highest confidence metric output by the SEC model 312 is 0.85, then the SEC engine 320 determines that the sound category of the audio data sample 310 is not identified and sends a trigger signal 326 to the drift detector 328. In this example, if the drift detector 328 determines that the SEC model 312 corresponds to the audio scene 342 of the audio data sample 310, then the drift detector 328 stores the audio data sample 310 as drift data 344 associated with the specific sound category. In one particular aspect, the metadata associated with the SEC model 314 includes information specifying one or more audio scenes associated with each SEC model 314. For example, the SEC model 312 may be configured to detect sound events in a user's home, in which case the metadata associated with the SEC model 312 may indicate that the SEC model 312 is associated with a "home" audio scene. In this example, if audio scene 342 indicates that device 100 is in a home location (e.g., based on location information, user input, detection of a home wireless network signal, image or video data representing the home location, etc.), then drift detector 328 determines that SEC model 312 corresponds to audio scene 342.

[0101] In some implementations, the drift detector 328 also stores some audio data samples 310 as model update data 348 and designates them as unknown data 346. As a first example, if the drift detector 328 determines that the SEC model 312 does not correspond to the audio scene 342 of the audio data sample 310, the drift detector 328 can store unknown data 346. As a second example, if the confidence metric output by the SEC model 312 fails to meet a drift threshold, the drift detector 328 can store unknown data 346. In this example, the drift threshold is less than the detection threshold used by the SEC engine 320. For example, if the SEC engine 320 uses a detection threshold of 0.95, the drift threshold can have some other value, such as 0.80, 0.75, or less than 0.95. In this example, if the highest confidence metric of the audio data sample 310 is less than the drift threshold, the drift detector 328 determines that the audio data sample 310 belongs to a sound category that the SEC model 312 was not trained to recognize and designates the audio data sample 310 as unknown data 346. In one specific aspect, if the drift detector 328 determines that the SEC model 312 corresponds to the audio scene 342 of the audio data sample 310, then the drift detector 328 only stores the unknown data 346. In another specific aspect, the drift detector 328 stores the unknown data 346 regardless of whether the drift detector 328 determines that the SEC model 312 corresponds to the audio scene 342 of the audio data sample 310.

[0102] After model update data 348 is stored, model updater 352 can access model update data 348 and use it to update one of the available SEC models 314 (e.g., SEC model 312). For example, each entry in model update data 348 indicates the SEC model associated with that entry, and model updater 352 uses that entry as training data to update the corresponding SEC model. In one particular aspect, model updater 352 updates the SEC model when update criteria are met or when a user or another party (e.g., the vendor of device 100, SEC engine 320, SEC model 314, etc.) initiates a model update. Update criteria can be met when a specific number of entries are available in model update data 348, when a specific number of entries are available in model update data 348 for a particular SEC model, when a specific number of entries are available in model update data 348 for a particular sound category, when a specific amount of time has elapsed since a previous update, when other updates occur (e.g., when a software update associated with device 100 occurs), or based on the occurrence of another event.

[0103] Model updater 352 uses backpropagation or a similar machine learning optimization process to update the training of SEC model 312 using drift data 344 as labeled training data. For example, model updater 352 provides audio data samples from drift data 344 of model update data 348 as input to SEC model 312, determines the value of the error function (also known as the loss function) based on the output of SEC model 312 and the labels associated with the audio data samples (as indicated in the drift data 344 stored by drift detector 328), and determines the updated link weights of SEC model 312 using gradient descent (or some variant thereof) or another machine learning optimization process.

[0104] Model updater 352 (in addition to the audio data samples of drift data 344) can also provide additional audio data samples to SEC model 312 during update training. For example, model update data 348 may include one or more known audio data samples (such as a subset of the audio data samples initially used to train SEC model 312), which reduces the chance that update training will cause SEC model 312 to forget previous training (where "forgetting" here refers to losing the reliability of detecting the sound categories that SEC model 312 was previously trained to recognize). Since the sound categories associated with the audio data samples of drift data 344 are indicated by drift detector 328, update training to take drift into account can be done automatically (e.g., without user input). As a result, the functionality of device 100 (e.g., accuracy in recognizing sound categories) can improve over time without user intervention and with fewer computational resources compared to those used to generate a new SEC model from scratch. Reference Figure 4 A specific example is described of a transfer learning process in which model updater 352 can be used to update SEC model 312 based on drift data 344.

[0105] In some respects, model updater 352 can also use unknown data 346 from model update data 348 to update the training of SEC model 312. For example, periodically or occasionally, such as when update criteria are met, model updater 352 can prompt a user to label the sound category of an entry in unknown data 346 from model update data 348. If the user selects to label the sound category of an entry in unknown data 346, device 100 (or another device) can play the sound corresponding to the audio data sample of unknown data 346. The user can (e.g., via input device 336) provide one or more labels 350 identifying the sound category of the audio data sample. If the sound category indicated by the user is the sound category that SEC model 312 is trained to recognize, the unknown data 346 is reclassified as drift data 344 associated with the user-specified sound category and SEC model 312. Depending on the configuration of the model updater 352, if the sound category indicated by the user is a sound category that the SEC model 312 was not trained to recognize (e.g., a new sound category), the model updater 352 may discard the unknown data 346, send the unknown data 346 and the user-specified sound category to another device for use in generating a new or updated SEC model, or it may use the unknown data 346 and the user-specified sound category to update the SEC model 312. (See reference) Figure 5 A specific example is described of a transfer learning process in which model updater 352 can be used to update SEC model 312 based on unknown data 346 and user-specified sound categories.

[0106] An updated SEC model 354 generated by model updater 352 is added to the available SEC models 314 so that the updated SEC model 354 can be used to evaluate audio data samples 310 received after the updated SEC model 354 is generated. Therefore, the set of available SEC models 314 that can be used to evaluate sound is dynamic. For example, one or more of the available SEC models 314 can be automatically updated to account for drift data 344. Additionally, one or more of the available SEC models 314 can be updated using transfer learning operations to account for unknown sound categories, which use fewer computational resources (e.g., memory, processing time, and power) compared to training a new SEC model from scratch.

[0107] Figure 4 It is a diagram illustrating how the SEC model 408 is updated based on a specific example to take into account various aspects of drift. Figure 4 SEC Model 408 includes or corresponds to Figure 3The specific SEC model associated with drift data 344 in the available SEC models 314. For example, if SEC engine 320 generates trigger signal 326 in response to the output of SEC model 312, then drift data 344 is associated with SEC model 312, and SEC model 408 corresponds to or includes SEC model 312. As another example, if SEC engine 320 generates trigger signal 326 in response to the output of the Kth SEC model 318, then drift data 344 is associated with the Kth SEC model 318, and SEC model 408 corresponds to or includes the Kth SEC model 318.

[0108] exist Figure 4 In the example explained, training data 402 is used to update the SEC model 408. Training data 402 includes drift data 344 and one or more labels 404. Each entry of drift data 344 includes an audio data sample (e.g., audio data sample 406) and is associated with a corresponding label in the labels 404. The audio data sample of each entry of drift data 344 includes a set of values ​​representing features extracted from or determined based on a sound never recognized by the SEC model 408. The label 404 corresponding to each entry of drift data 344 identifies the sound category to which the expected sound belongs. As an example, the label 404 corresponding to each entry of drift data 344 could be derived from... Figure 3 The drift detector 328 is assigned in response to determining that the SEC model 408 corresponds to the audio scene from which the audio data samples were generated. In this example, the drift detector 328 may assign the audio data samples to the sound category associated with the highest confidence metric in the output of the SEC model 408.

[0109] exist Figure 4 In this process, audio data sample 406 corresponding to a sound is provided to SEC model 408, and SEC model 408 generates output 410 indicating the sound category, one or more confidence metrics, or both, assigned to the audio data sample 406. Model updater 352 uses output 410 and label 404 corresponding to audio data sample 406 to determine updated link weights 412 of SEC model 408. SEC model 408 is updated based on updated link weights 412, and the training process is iteratively repeated until the training termination condition is met. During training, each entry of drift data 344 may be provided to SEC model 408 (e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train SEC model 408) may also be provided to SEC model 408 to reduce the chance of SEC model 408 forgetting previous training data.

[0110] The training termination condition can be met when all drift data 344 has been provided to the SEC model 408 at least once, after a certain number of training iterations have been performed, when the convergence metric meets the convergence threshold, or when some other condition indicating the end of training is met. When the training termination condition is met, the model updater 352 stores the updated SEC model 414, which corresponds to the SEC model 408 with link weights based on the updated link weights 412 applied during training.

[0111] Figure 5 This is a diagram illustrating how the SEC model 510 is updated based on training data 502 for a specific example to take into account various aspects of the unknown data. Figure 5 SEC Model 510 includes or corresponds to Figure 3 The specific SEC model associated with the unknown data 346 in the available SEC models 314. For example, if the SEC engine 320 generates a trigger signal 326 in response to the output of SEC model 312, then the unknown data 346 is associated with SEC model 312, and SEC model 510 corresponds to or includes SEC model 312. As another example, if the SEC engine 320 generates a trigger signal 326 in response to the output of the Kth SEC model 318, then the unknown data 346 is associated with the Kth SEC model 318, and SEC model 510 corresponds to or includes the Kth SEC model 318.

[0112] exist Figure 5In the example, model updater 352 generates an updated model 506. The updated model 506 includes the SEC model 510 to be updated, an incremental model 508, and one or more adapter networks 512. The incremental model 508 is a copy of the SEC model 510 with a different output layer than the SEC model 510. Specifically, the output layer of the incremental model 508 includes more output nodes than the output layer of the SEC model 510. For example, the output layer of the SEC model 510 includes a first count of nodes (e.g., N nodes, where N is a positive integer corresponding to the number of sound categories that the SEC model 510 was trained to recognize), and the output layer of the incremental model 508 includes a second count of nodes (e.g., N+M nodes, where M is a positive integer corresponding to the number of new sound categories that the updated SEC model 524 was trained to recognize but the SEC model 510 was not trained to recognize). The first node count corresponds to the sound category count of the first set of sound categories that the SEC model 510 is trained to recognize (e.g., the first set of sound categories includes N different sound categories that the SEC model 510 can recognize), and the second node count corresponds to the sound category count of the second set of sound categories that the updated SEC model 524 is to be trained to recognize (e.g., the second set of sound categories includes N+M different sound categories that the updated SEC model 524 is to be trained to recognize). The second set of sound categories includes the first set of sound categories (e.g., N categories) plus one or more additional sound categories (e.g., M categories). The model parameters (e.g., link weights) of the incremental model 508 are initialized to be equal to the model parameters of the SEC model 510.

[0113] (A)Adapter network 512 includes a neural adapter and a merging adapter. The neural adapter includes one or more adapter layers configured to receive input from SEC model 510 and generate an output that can be merged with the output of incremental model 508. For example, SEC model 510 generates a first output corresponding to a first category count of a first set of sound categories. In one particular aspect, the first output includes one data element (e.g., N data elements) for each node of the output layer of SEC model 510. In contrast, incremental model 508 generates a second output corresponding to a second category count of a second set of sound categories. For example, the second output includes one data element (e.g., N+M data elements) for each node of the output layer of incremental model 508. In this example, the adapter layer of (A)Adapter network 512 receives the output of SEC model 510 as input and generates an output with a second count (e.g., N+M) data elements. In a particular example, the adapter layer of (A)Adapter network 512 includes two fully connected layers (e.g., an input layer including N nodes and an output layer including N+M nodes, wherein each node of the input layer is connected to each node of the output layer).

[0114] The merging adapter of the (variable) adapter network 512 is configured to generate the output 514 of the updated model 506 by merging the outputs of the (variable) adapter layers and the output of the incremental model 508. For example, the merging adapter combines the outputs of the (variable) adapter layers and the output of the incremental model 508 element-wise to generate a combined output, and applies an activation function (such as a sigmoid function) to this combined output to generate output 514. Output 514 indicates the sound category to which the audio data sample 504 is assigned by the updated model 506, one or more confidence metrics determined by the updated model 506, or both.

[0115] Model updater 352 uses output 504 and labels 350 corresponding to audio data sample 514 to determine updated link weights 508 of incremental model 508, adapter network(s), or both. The link weights of SEC model 510 remain unchanged during training. The training process is repeated iteratively until the training termination condition is met. During training, each entry of unknown data 346 may be provided to update model 506 (e.g., one entry per iteration). Additionally, in some implementations, other audio data samples (e.g., audio data samples previously used to train SEC model 510) may also be provided to update model 506 to reduce the chance of incremental model 508 forgetting previous training data of SEC model 510.

[0116] The training termination condition can be satisfied when all unknown data 346 has been provided to the updated model 506 at least once, after a certain number of training iterations have been performed, when the convergence metric meets the convergence threshold, or when some other condition indicating the end of training is met. When the training termination condition is met, the model checker 520 selects the updated SEC model 524 (e.g., a combination of SEC model 510, incremental model 508, and adapter networks 512) from the incremental model 508 and the updated model 506.

[0117] In one particular aspect, model checker 520 selects the updated SEC model 524 based on the accuracy of the sound category 522 assigned by incremental model 508 and the accuracy of the sound category 522 assigned by SEC model 510. For example, model checker 520 can determine the F1 score of incremental model 508 (based on the sound category 522 assigned by incremental model 508) and the F1 score of SEC model 510 (based on the sound category 522 assigned by SEC model 510). In this example, if the F1 score of incremental model 508 is greater than or equal to the F1 score of SEC model 510, then model checker 520 selects incremental model 508 as the updated SEC model 524. In some implementations, if the F1 score of incremental model 508 is greater than or equal to the F1 score of SEC model 510 (or less than a threshold amount of the F1 score of SEC model 510), then model checker 520 selects incremental model 508 as the updated SEC model 524. If the F1 score of incremental model 508 is less than the F1 score of SEC model 510 (or less than the F1 score of SEC model 510 by more than a threshold amount), then model checker 520 selects updated model 506 as the updated SEC model 524. If incremental model 508 is selected as the updated SEC model 524, then SEC model 510, adapter network 512, or both may be discarded.

[0118] In some implementations, the model checker 520 is omitted or integrated with the model updater 352. For example, after training the updated model 506, the updated model 506 may be stored as the updated SEC model 524 (e.g., there is no choice between the updated model 506 and the incremental model 508). As an example, when training the updated model 506, the model updater 352 may determine the accuracy metric of the incremental model 508. In this example, the training termination condition may be based on the accuracy metric of the incremental model 508, such that after training, the incremental model 508 is stored as the updated SEC model 524 (e.g., there is no choice between the updated model 506 and the incremental model 508).

[0119] Using references Figure 5The described transfer learning technique, Model Checker 520 enables... Figure 3 The device 100 is able to update the SEC model to identify previously unknown sound categories. Additionally, the described transfer learning technique uses far fewer computer resources (e.g., memory, processing time, and power) than would be used to train the SEC model from scratch to identify previously unknown sound categories.

[0120] In some implementations, refer to Figure 4 The described operation (e.g., generating an updated SEC model 414 based on drift data 344) in Figure 3 The device 100 executes (e.g., at one or more processors 110), and references Figure 5 The described operations (e.g., generating an updated SEC model 524 based on unknown data 346) are performed on different devices (such as...) Figure 8 The process is performed at a remote computing device 818. For illustration, unknown data 346 and (various) tags 350 can be captured at device 100 and transmitted to a second device with more available computing resources. In this example, the second device generates an updated SEC model 524, and device 100 downloads or receives transmissions or data representing the updated SEC model 524 from the second device. Generating an updated SEC model 524 based on unknown data 346 is a more resource-intensive process (e.g., using more memory, power, and processor time) compared to generating an updated SEC model 414 based on drift data 344. Therefore, the reference is partitioned between different devices. Figure 4 The described operations and references Figure 5 The described operation can save 100% of the device's resources.

[0121] Figure 6 It is an explanation Figure 1 A diagram illustrating a specific instance of the operation of device 100, wherein the determination of whether an active SEC model (e.g., SEC model 312) corresponds to an audio scene captured by audio data sample 310 is based on comparing the current audio scene with a previous audio scene.

[0122] exist Figure 6 In this process, audio data captured by microphones 104 is used to generate audio data samples 310. Audio data samples 310 are then used to perform audio classification 602. For example, one or more of the available SEC models 314 are... Figure 3 The SEC engine 320 is used as the active SEC model. In one particular aspect, the active SEC model is selected from the available SEC models 314 based on the audio scene indicated by the scene detector 340 (also referred to as the previous audio scene 608) during the previous sampling period.

[0123] Audio classification 602 uses an active SEC model to generate a result 604 based on the analysis of audio data sample 310. Result 604 may indicate the sound category associated with audio data sample 310, the probability that audio data sample 310 corresponds to a specific sound category, or that the sound category of audio data sample 310 is unknown. If result 604 indicates that audio data sample 310 corresponds to a known sound category, a decision is made in box 606 to generate an output 324 indicating the sound category 322 associated with audio data sample 310. For example, Figure 1 The SEC engine 320 can generate an output of 324.

[0124] If result 604 indicates that audio data sample 310 does not correspond to a known sound category, a decision is made in box 606 to generate trigger 326. Trigger 326 activates the drift detection scheme, which in Figure 6 This includes enabling scene detector 340 to identify the current audio scene 607 based on data from sensors 134.

[0125] In box 610, the current audio scene 607 is compared with the previous audio scene 608 to determine whether an audio scene change has occurred since the active SEC model was selected. In box 612, a determination is made as to whether the sound category of audio data sample 310 has been unrecognized due to drift. For example, if the current audio scene 607 does not correspond to the previous audio scene 608, then the determination in box 612 is: drift is not the reason why the sound category of audio data sample 310 is not recognized. In this case, audio data sample 310 can be discarded or stored as unknown data in box 614.

[0126] If the current audio scene 607 corresponds to the previous audio scene 608, then the determination in box 612 is that the sound category of audio data sample 310 is not identified due to drift, because the active SEC model corresponds to the current audio scene 607. In this case, the drifted sound category is identified in box 616, and audio data sample 310 and the identifier of that sound category are stored as drift data in box 618.

[0127] When sufficient drift data has been stored, the SEC model in box 620 is updated to generate an updated SEC model 354. The updated SEC model 354 is added to the available SEC models 314. In some implementations, the updated SEC model 354 replaces the active SEC model that generated result 604.

[0128] Figure 6 It is an explanation Figure 1The diagram illustrates another specific instance of the operation of device 100, wherein the determination of whether an active SEC model (e.g., SEC model 312) corresponds to the audio scene captured by audio data sample 310 is based on a comparison of information describing the active SEC model of the current audio scene.

[0129] exist Figure 6 In this process, audio data captured by microphones 104 is used to generate audio data samples 310. Audio data samples 310 are then used to perform audio classification 602. For example, one or more of the available SEC models 314 are... Figure 3 The SEC engine 320 is used as the active SEC model. In one particular aspect, the active SEC model is selected from the available SEC models 314. In some implementations, a set of available SEC models 314 is used, rather than selecting one or more of the available SEC models 314 as the active SEC model.

[0130] Audio classification 602 uses one or more of the available SEC models to generate a result 604 based on the analysis of audio data sample 310. Result 604 may indicate the sound category associated with audio data sample 310, the probability that audio data sample 310 corresponds to a specific sound category, or that the sound category of audio data sample 310 is unknown. If result 604 indicates that audio data sample 310 corresponds to a known sound category, a decision is made in box 606 to generate an output 324 indicating the sound category 322 associated with audio data sample 310. For example, Figure 3 The SEC engine 320 can generate an output of 324.

[0131] If result 604 indicates that audio data sample 310 does not correspond to a known sound category, a decision is made in box 606 to generate trigger 326. Trigger 326 activates the drift detection scheme, which in Figure 7 This includes enabling scene detector 340 to identify the current audio scene based on data from (a) sensors 134 and to determine whether the current audio scene corresponds to the SEC model that generates result 604 that leads to trigger 326.

[0132] In box 612, a determination is made regarding whether the sound category of audio data sample 310 was not identified due to drift. For example, if the current audio scene does not correspond to the SEC model that generated result 604, then the determination in box 612 is: drift is not the reason why the sound category of audio data sample 310 was not identified. In this case, audio data sample 310 can be discarded or stored as unknown data in box 614.

[0133] If the current audio scene corresponds to the SEC model that generated result 604, then the determination at box 612 is: the sound category of audio data sample 310 was not identified due to drift. In this case, the drifted sound category is identified in box 616, and audio data sample 310 and the identifier of that sound category are stored as drift data in box 618.

[0134] When sufficient drift data has been stored, the SEC model in box 620 is updated to generate an updated SEC model 354. The updated SEC model 354 is added to the available SEC models 314. In some implementations, the updated SEC model 354 replaces the active SEC model that generated result 604.

[0135] For reference Figure 3-7 The operations described for updating the model can be combined with, as in the reference... Figure 1-2 The described context-based model selection is used. For example, a first and second person living together in a first residence (e.g., a rural house) might have devices using a shared model associated with the first residence, and such models can be updated by training to accommodate new sound events and drift associated with the first residence. After the first person moves to a second residence (e.g., a university dormitory), the first person's devices can update one or more models to improve accuracy at the second residence, and therefore the models used by the first and second person's devices may differ significantly. Upon the first person's return to the first residence, the first person's devices can select and download models used by the second person's devices to achieve higher accuracy during use at the first residence. For example, the second person can provide access permission to share one or more models with the first person's devices, such as via peer-to-peer transfer between devices or via a local wireless home network. Upon leaving the first residence, the first person's devices can remove the shared model associated with the first residence and revert to the model associated with the second residence.

[0136] Figure 8 It is an explanation Figure 1 A block diagram of a specific instance of device 100. In various implementations, device 100 may have more than Figure 8 More or fewer components as explained in the text.

[0137] In a particular implementation, device 100 includes a processor 804 (e.g., a central processing unit (CPU)). Device 100 may include one or more additional processors 806 (e.g., one or more digital signal processors (DSPs)). Processor 804, processor(s), or both may correspond to one or more processors 110. For example, in Figure 8 In this context, the processor 806 includes a context detector 140, a model selector 190, and a model-based application 192.

[0138] exist Figure 8 In addition, device 100 also includes memory 108 and CODEC (codec) 824. Memory 108 stores instructions 860 that can be executed by processor 804 or processor(s) 806 to implement reference... Figure 1-7 One or more operations are described. In one example, memory 108 corresponds to a non-transient computer-readable medium storing instructions 860 executable by one or more processors 110, and instructions 860 include or correspond to context detector 140 (e.g., operations executable by a processor to perform that contribute to context detector 140), model selector 190, model-based application 192, or a combination thereof. Memory 108 may also store available models 114.

[0139] exist Figure 8 In this configuration, loudspeakers 822 and microphones 104 can be coupled to CODEC 824. Figure 8 In the example described, CODEC 824 includes a digital-to-analog converter (DAC 826) and an analog-to-digital converter (ADC 828). In a particular implementation, CODEC 824 receives analog signals from microphones 104, converts the analog signals into digital signals using the ADC 828, and provides digital signals to processors 806. In a particular implementation, processors 806 provide digital signals to CODEC 824, and CODEC 824 uses the DAC 826 to convert the digital signals into analog signals and provides analog signals to speakers 822.

[0140] exist Figure 8 In addition, device 100 also includes input device 336. Device 100 may also include display 820 coupled to display controller 810. In one particular aspect, input device 336 includes sensors, keyboards, pointing devices, etc. In some implementations, input device 336 and display 820 are combined in a touchscreen or similar touch or motion-sensitive display.

[0141] In some implementations, device 100 also includes a modem 812 coupled to transceiver 814. For example... Figure 8As shown, transceiver 814 is coupled to antenna 816 to enable wireless communication with other devices, such as remote computing device 818 (e.g., a server or network memory storing at least a portion of model library 162). For example, modem 812 may be configured to receive model 112, access permission to model 112, or both, from remote computing device 818 via wireless transmission, at least in part, based on a location of device 100 matching a specific location. In other examples, transceiver 814 is additionally or alternatively coupled to a communication port (e.g., an Ethernet port) to enable wired communication with other devices, such as remote computing device 818.

[0142] exist Figure 8 In this device 100, clock 332 and sensor 134 are included. As a specific example, sensor 134 includes one or more cameras 150, one or more position sensors 152, microphones 104, activity detectors 154, other sensors 156, or combinations thereof.

[0143] In one particular aspect, clock 332 generates a clock signal that can be used to assign timestamps to specific sensor data samples to indicate when a particular sensor data sample was received. In this respect, model selector 190 can use the timestamps to select the model to be used to process the input data. Additionally or alternatively, the timestamps can be used by context detector 140 to determine the context 142 associated with a particular sensor data sample.

[0144] In one particular aspect, cameras 150 generate image data, video data, or both. A model selector 190 can use the image data, video data, or both to select a specific model for analyzing input data. Additionally or alternatively, the image data, video data, or both can be used by a context detector 140 to determine the context 142 associated with a particular sensor data sample. For example, a particular model 112 may be designated for outdoor use, and the image data, video data, or both may be used to confirm that device 100 is located in an outdoor environment.

[0145] In one particular aspect, the location sensors 152 generate location data, such as Global Positioning data indicating the location of device 100. A model selector 190 can use the location data to select a model to be used for analyzing the input data. Additionally or alternatively, the location data can be used by a context detector 140 to determine the context 142 associated with a particular sample of sensor data. For example, a particular model 112 may be specified for home use, and the location data may be used to confirm that device 100 is located at a home location. The location sensors 852 may include receivers for satellite-based positioning systems, receivers for local positioning systems, inertial navigation systems, landmark-based positioning systems, or combinations thereof.

[0146] (Various) Other sensors 156 may include, for example, orientation sensors, magnetometers, light sensors, contact sensors, temperature sensors, or any other sensors coupled to or included in device 100 and that may be used to generate sensor data useful for determining the context 142 associated with device 100 at a particular time.

[0147] In a particular implementation, device 100 is included in a system-in-package or system-on-a-chip device 802. In a particular implementation, memory 108, processor 804, processor(s) 806, display controller 810, CODEC 824, modem 812, and transceiver 814 are included in a system-in-package or system-on-a-chip device 802. In a particular implementation, input device 336 and power supply 830 are coupled to system-on-a-chip device 802. Furthermore, in a particular implementation, such as Figure 8 As explained, the display 820, input device 336, speakers(s) 822, sensor 134, clock 332, antenna 816, and power supply 830 are external to the system-on-a-chip device 802. In a particular implementation, each of the display 820, input device 336, speakers(s) 822, sensor 134, clock 332, antenna 816, and power supply 830 may be coupled to components of the system-on-a-chip device 802, such as an interface or controller.

[0148] Device 100 may include, correspond to, or be included in the following: voice-activated devices, audio devices, wireless speakers and voice-activated devices, portable electronic devices, automobiles, vehicles, computing devices, communication devices, Internet of Things (IoT) devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, smart speakers, mobile computing devices, mobile communication devices, smartphones, cellular phones, laptops, computers, tablet devices, personal digital assistants, display devices, televisions, game consoles, electrical appliances, music players, radios, digital video players, digital video disc (DVD) players, tuners, cameras, navigation devices, or any combination thereof. In one particular aspect, processor 804, processors 806, or combinations thereof are included in an integrated circuit.

[0149] Figure 9 Is it included Figure 1 The illustrative example of various aspects of the vehicle 900 in the device 100. According to one implementation, the vehicle 900 is an autonomous vehicle. According to other implementations, the vehicle 900 is a car, truck, motorcycle, aircraft, watercraft, etc. Figure 9 In this embodiment, vehicle 900 includes a display 820, one or more sensors 134, a device 100 including a context detector 140, a model selector 190, a model-based application 192, or a combination thereof. Sensors 134, context detector 140, model selector 190, and model-based application 192 are shown using dashed lines to indicate that these components may not be visible to passengers of vehicle 900. Device 100 may be integrated into or coupled to vehicle 900.

[0150] In one particular aspect, device 100 is coupled to display 820 and provides output to display 820 in response to model-based application 192 (such as in response to the detection or identification of various events described herein, e.g., sound events). For example, device 100 provides output to display 820. Figure 3 The output 324 indicates the sound category of sound 302 (such as a car horn) in the audio data 105 received from microphones 104. In some implementations, device 100 may perform an action in response to a recognized sound event, such as alerting a vehicle operator or activating one of sensors 134. In a particular example, device 100 provides an output indicating whether an action is being performed in response to a recognized sound event. In one particular aspect, a user can select an option displayed on display 820 to enable or disable the performance of actions in response to a recognized sound event.

[0151] In a particular implementation, sensor 134 includes Figure 1The sensor 134 may include microphones 104, vehicle occupancy sensors, eye-tracking sensors, position sensors 152, or external environment sensors (e.g., lidar sensors or cameras). In one aspect, the sensor input of sensor 134 indicates the user's location. For example, sensor 134 may be associated with various locations within vehicle 900.

[0152] Therefore, refer to Figure 1-8 The described technology enables users of vehicle 900 to select the model to use based on the specific context in which device 100 operates.

[0153] Figure 10 Examples of devices 100 coupled to or integrated within a headset 1002 are depicted, such as a virtual reality headset, an augmented reality headset, a mixed reality headset, an extended reality headset, a head-mounted display, or a combination thereof. A visual interface device (such as a display 820) is placed in front of a user to display augmented reality, mixed reality, or virtual reality images or scenes to the user when the headset 1002 is worn. In a particular example, the display 820 is configured to display the output of the device 100. The headset 1002 includes sensors 134, such as microphones 104, cameras 150, position sensors 152, other sensors 156, or combinations thereof. Although described in a single location, in other implementations, sensors 134 may be positioned at other locations on the headset 1002 (such as an array of one or more microphones and one or more cameras distributed around the headset 1002) to detect multimodal input.

[0154] Sensor 134 enables the detection of sensor data, which device 100 uses to detect the context of head-mounted device 1002 and to update the model based on the detected context. For example, model-based application 192 (e.g., SEC engine 320) can use one or more models to generate sound event classification data, which can be provided to display 820 to indicate that an identified sound event (such as a car horn) has been detected in audio data samples received from sensor 134. In some implementations, device 100 may perform actions in response to the recognition of a sound event, such as activating a camera or another sensor 134, or providing haptic feedback to the user.

[0155] Figure 11An example of a device 100 integrated into a wearable electronic device 1102 (described as a "smartwatch") is depicted, including a display 820 and sensors 134. Sensors 134 perform context detection, for example, based on modalities such as location, video, voice, and gestures, which the device 100 can use to update one or more models used by a model-based application 192. Sensors 134 also enable the detection of sound and other events in the environment surrounding the wearable electronic device 1102, which the device 100 can use the model-based application 192 to detect or interpret. For example, the device 100 provides... Figure 3 The output 324 indicates that a recognized sound event has been detected in the audio data sample received from sensor 134. In some implementations, device 100 may perform an action in response to the recognition of a sound event, such as activating a camera or another sensor 134, or providing haptic feedback to the user.

[0156] Figure 12 This is an illustrative example of a voice-controlled speaker system 1200. The voice-controlled speaker system 1200 may have wireless network connectivity and be configured to perform auxiliary operations. Figure 12 In this system, device 100 is included in a voice-controlled speaker system 1200. The voice-controlled speaker system 1200 also includes a speaker 1202 and a sensor 134. Sensor 134 includes... Figure 1 The microphones 104 are used to receive voice input or other audio input.

[0157] During operation, in response to a received verbal command or a recognized sound event, the voice-controlled speaker system 1200 can perform auxiliary operations. These operations may include adjusting the temperature, playing music, turning on lights, etc. Sensor 134 enables the detection of data samples, which device 100 can use to update the context of the voice-controlled speaker system 1200 and, based on that context, update one or more models. Additionally, the voice-controlled speaker system 1200 can perform operations based on events recognized by device 100. For example, if device 100 recognizes the sound of a door closing, the voice-controlled speaker system 1200 can turn on one or more lights.

[0158] Figure 13 The explanation included Figure 1 The equipment includes 100 cameras of various types, totaling 1300. Figure 13 In this configuration, device 100 is incorporated into or coupled to camera 1300. Camera 1300 includes image sensor 1302 and one or more other sensors (e.g., sensor 134), such as... Figure 1Microphone 104. Additionally, camera 1300 includes device 100 configured to determine the context of camera 1300 and update one or more models based on the context. In one particular aspect, camera 1300 is configured to perform one or more actions in response to a recognized sound event. For example, camera 1300 may cause image sensor 1302 to capture an image in response to device 100 detecting a specific sound event in an audio data sample from sensor 134.

[0159] Figure 14 The explanation included Figure 1 The device comprises 100 mobile devices of various aspects, totaling 1400. Figure 14 In the middle, the mobile device 1400 includes or is coupled to Figure 1 Device 100. Mobile device 1400 includes a telephone or tablet device, as an illustrative and non-limiting example. Mobile device 1400 includes a display 820 and sensors 134, such as microphones 104, cameras 150, position sensors 152, or other sensors 156. During operation, mobile device 1400 may perform specific actions in response to device 100 recognizing specific sound events. For example, these actions may include sending commands to other devices, such as thermostats, home automation systems, another mobile device, etc.

[0160] Figure 15 The explanation included Figure 1 The equipment includes 100 types of hearing aids and 1500 types of hearing aids. Figure 15 In the middle, hearing aid device 1500 includes or is coupled to Figure 1 The hearing aid device 1500 includes sensors 134, such as microphones 104, cameras 150, position sensors 152, or other sensors 156. During operation, the hearing aid device 1500 may update one or more models in response to the device 100 recognizing the context of the hearing aid device 1500 (such as the acoustic environment of the hearing aid device 1500) for use by model-based applications 192 to process audio data (such as for location-dependent noise reduction).

[0161] Figure 16 The explanation included Figure 1 The equipment includes 100 types of aerial equipment in various aspects, totaling 1600. Figure 16 In the middle, aerial equipment 1600 includes or is coupled to Figure 1Device 100. Aerial device 1600 is a manned, unmanned, or remotely controlled aerial device (e.g., a package delivery drone). Aerial device 1600 includes a control system 1602 and sensors 134, such as microphones 104, cameras 150, position sensors 152, or other sensors 156. The control system 1602 controls various operations of aerial device 1600, such as cargo release, sensor activation, takeoff, navigation, landing, or combinations thereof. For example, the control system 1602 can control the flight of aerial device 1600 between designated points and the deployment of cargo at a specific location. During operation, aerial device 1600 can update one or more models in response to device 100 recognizing the context of aerial device 1600 (such as the location or acoustic environment of aerial device 1600) for use by model-based application 192 to detect events. For illustration, the control system 1602 can initiate a safe landing protocol in response to device 100 detecting an aircraft engine.

[0162] Figure 17 The explanation included Figure 1 The device comprises 100 head-mounted devices of various aspects, totaling 1700. Figure 17 In the middle, the head-mounted device 1700 includes or is coupled to Figure 1 Device 100. Head-mounted device 1700 includes... Figure 1 The headset 1700 may include microphones 104 positioned primarily to capture the user's voice. The headset 1700 may also include one or more additional microphones positioned primarily to capture ambient sound (e.g., for noise cancellation operations); and one or more sensors 134 (such as cameras 150, position sensors 152, or other sensors 156). In one particular aspect, the headset 1700 may update one or more models in response to the device 100 recognizing changes in the context of the headset 1700 (such as the location of the headset 1700 or the acoustic environment) for use by the model-based application 192 to perform operations (such as noise cancellation features).

[0163] Figure 18 The explanation included Figure 1 The equipment includes 100 electrical components of various types, totaling 1800. Figure 18 In this implementation, appliance 1800 is a table lamp; however, in other implementations, appliance 1800 includes another IoT appliance, such as a refrigerator, coffee maker, oven, or another household appliance. Appliance 1800 includes or is coupled to... Figure 1Device 100. Appliance 1800 includes sensors 134, such as microphones 104, cameras 150, position sensors 152, activity detectors 154, or other sensors 156. In one aspect, appliance 1800 may update one or more models in response to a change in the context recognized by device 100, for use by model-based application 192 to perform operations (such as activating a light in response to device 100 detecting a door is closed).

[0164] Figure 19 It is an explanation Figure 1 A flowchart illustrating an example of an operation method 1900 for device 100 is provided. Method 1900 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as...). Figure 8 Instruction 860) is used to perform context-based model selection.

[0165] Method 1900 includes: at block 1902, receiving sensor data from one or more sensor devices at one or more processors of the device. For example, a context detector 140 in one or more processors 110 receives sensor data 138 from one or more sensor devices 134. In some implementations, the sensor data includes location data of the device's position, such as location data 153, and the context is at least partially based on the location. In some implementations, the sensor data includes image data corresponding to a visual scene, such as image data 151, and the context is at least partially based on the visual scene. In some implementations, the sensor data includes audio corresponding to an audio scene, such as audio data 105, and the context is at least partially based on the audio scene. In some implementations, the sensor data includes motion data corresponding to the device's motion, such as activity data 155, and the context is at least partially based on the device's motion. At block 1904, method 1900 includes determining the context of the device at the one or more processors based on the sensor data. For example, a context detector 140 in one or more processors 110 receives sensor data 138 from one or more sensor devices 134 and determines context 142 based on the sensor data 138. In some implementations,

[0166] In box 1906, method 1900 includes selecting a model based on the context at one or more processors. For example, model selector 190 selects model 112 based on context 142. In a particular implementation, the model is selected from a plurality of models stored in the device's memory, such as available model 114. According to some implementations, the model is downloaded from a library (such as model library 162) corresponding to a library of shared models. The model may include a trained model uploaded to the library from another user device. In one example, the library corresponds to a crowdsourced model library. The library may be included in a distributed context-aware system.

[0167] In block 1908, method 1900 includes using the model at the one or more processors to process the input signal to generate a context-dependent output. For example, model-based application 192 uses a selected model 112 to process the input signal 106 to generate a context-dependent output 122.

[0168] In some implementations, method 1900 includes pruning the model in response to determining that the context has changed. For example, model selector 190 may permanently delete the model in response to detecting that context 142 has changed (e.g., device 100 has been moved to a different location) and detecting that the current model is no longer suitable for the new context or that another model is more suitable for the new context.

[0169] In some implementations, the model includes a sound event detection model, the input signal being an audio signal, and the context-dependent output including a classification of sound events in the audio signal. In some implementations, the model includes an automatic speech recognition model, the input signal being an audio signal, and the context-dependent output including text data representing speech in the audio signal. In some implementations, the model includes a natural language processing (NLP) model, the input signal being text data, and the context-dependent output including NLP output data based on the text data. In some implementations, the model includes a noise reduction model, the input signal being an audio signal, and the context-dependent output including a noise-reduced audio signal based on the audio signal. In some implementations, the model is associated with automatic adjustment of device operating modes, and the context-dependent output includes signals used to adjust the device operating mode.

[0170] In some implementations, method 1900 includes receiving the model from a second device via wireless transmission. For example, the context may correspond to the location of the device, and the model may include an acoustic model corresponding to a specific location. In some implementations, method 1900 includes receiving access permission to the model based at least in part on the device's location matching the specific location.

[0171] In some implementations, the context includes a specific acoustic environment, and method 1900 includes determining whether the available acoustic model library includes acoustic models specific to that specific acoustic environment and available for the device, and determining whether acoustic models of a general category for that specific acoustic environment are available for the device in response to the absence of acoustic models specific to that specific acoustic environment available for the device. For example, in response to the absence of an “office” model 274 available for device 100 in model library 162 when device 100 is located in room 204, device 100 may request an “office building” model 262 of a general model for the acoustic environment of room 204.

[0172] By selecting a model based on the device's context, Method 1900 enables the device to operate with higher accuracy compared to using a single model for all contexts. Furthermore, changing the model allows the device to operate with improved accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with retraining an existing model from scratch on the device for specific contexts. Moreover, operating the device using a context-based model enables improved operation of the device itself, such as achieving faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a higher-accuracy model specific to a particular context.

[0173] Figure 18 It is an explanation Figure 1 A flowchart illustrating an example of an operation method 1800 of device 100. Method 1800 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as... Figure 6 Instruction 660) is used to perform context-based model selection.

[0174] Method 1800 includes: in box 1802, selecting an acoustic model at one or more processors of the device that corresponds to a specific room in a building where the device is located. For example, model selector 190 selects... Figure 2 Acoustic model 212, which corresponds to room 204 in which device 100 is located.

[0175] Method 2000 includes: at block 2004, using the acoustic model at one or more processors to process the input audio signal. As an illustrative example, model-based application 192 uses acoustic model 212 to perform noise reduction on input signal 106 (e.g., audio data 105 from microphones 104) to generate a noise-reduced audio signal as a context-dependent output 122.

[0176] In some implementations, method 2000 includes downloading the acoustic model from an acoustic model library in response to determining that the device has entered the particular room. In some implementations, method 2000 includes trimming (e.g., removing) the acoustic model in response to the device leaving the particular room. In some implementations, method 2000 includes selecting the acoustic model based on sensor data indicating the device's location (such as detecting the location of device 100 by analyzing image data 151). In some implementations, method 2000 includes selecting the acoustic model based on location data indicating the device's location (such as location data 153).

[0177] Method 2000 enables devices to operate with higher accuracy by selecting acoustic models based on device context, compared to using a single acoustic model for all contexts. Furthermore, changing the acoustic model allows the device to operate with improved accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with retraining an existing acoustic model from scratch on the device for specific contexts. Moreover, operating the device using a context-based acoustic model enables improved operation of the device itself, such as achieving faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a context-specific, higher-accuracy acoustic model.

[0178] Figure 21 It is an explanation Figure 1 Device 100 is located within a vehicle (such as integrated in...) Figure 9 A flowchart illustrating an example of an operation method 2100 in a vehicle 900. Method 2100 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as...) Figure 8 Instruction 860) is used to perform context-based model selection.

[0179] Method 2100 includes: at block 2102, at one or more processors of the device and in response to detecting that a user has entered a vehicle, selecting a personalized acoustic model for the user from a plurality of personalized acoustic models corresponding to the vehicle. For example, Figure 9 Device 100 in vehicle 900 can store multiple acoustic models personalized for each user of vehicle 900, and can select a personalized model or set of models for a specific user entering vehicle 900. In some implementations, method 2100 includes determining that the user has entered the vehicle based on sensor data indicating the user's location. Figure 9The sensors 134 can determine that a user is inside the vehicle 900 via facial recognition, voice recognition, gestures, voice or interaction with an input device (e.g., a touchscreen in the vehicle 900) to input identity or one or more other technologies used to identify the user.

[0180] Method 2100 includes, at block 2104, using the personalized acoustic model at the one or more processors to process the input audio signal. For example, in some implementations, the personalized acoustic model corresponds to an ASR model trained for a specific user, and the personalized acoustic model is used by a model-based application 192 in vehicle 900 to perform speech recognition on the user's speech captured via one or more microphones in vehicle 900. To illustrate, the ASR model is used to enhance the accuracy of the voice interface used to control one or more operations of vehicle 900 (e.g., navigation systems, entertainment systems, climate control, driver assistance, or autopilot settings, etc.). As another example, the personalized acoustic model may include a SEC model personalized for a specific user. To illustrate, if a particular user frequently takes their dog on car trips, the user's personalized SEC model for use in vehicle 900 can be trained to recognize the additional sound category "dog barking within the vehicle."

[0181] In some implementations, method 2100 includes downloading the personalized acoustic model from an acoustic model library in response to determining that the user has entered the vehicle. In some implementations, method 2100 includes trimming (e.g., removing) the personalized acoustic model in response to the device leaving the vehicle.

[0182] By selecting a personalized acoustic model in response to detecting a user entering a vehicle, method 2100 enables the device to operate with higher accuracy compared to using a single acoustic model for all contexts. Furthermore, changing the acoustic model allows the device to operate with higher accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with retraining an existing acoustic model from scratch at the device for specific contexts. Moreover, operating the device using a context-based acoustic model enables improved operation of the device itself, such as achieving faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a context-specific, higher-accuracy acoustic model.

[0183] Figure 22 It is an explanation Figure 1 A flowchart illustrating an example of an operation method 2200 for device 100. Method 2200 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as...) Figure 8 Instruction 860) is used to perform context-based model selection.

[0184] Method 2200 includes: at block 2202, downloading an acoustic model corresponding to a specific location of the device at one or more processors of the device. In some implementations, method 2200 includes determining that the device has entered the specific location based on sensor data indicating the location of the device. In one example, in response to device 100 entering room 204 or (e.g., based on location data 153) determining that the device is located within room 204, device 100 downloads acoustic model 212 (e.g., an "office" model 274 corresponding to room 204).

[0185] Method 2200 includes: at block 2204, using the acoustic model to process the input audio signal at the one or more processors. In one example, device 100 corresponds to Figure 15 The hearing aid device 1500 uses an acoustic model 212 (e.g., an "office" model 274 corresponding to room 204) at model-based application 192 to perform noise reduction.

[0186] Method 2200 includes, at block 2206, removing the acoustic model at the one or more processors in response to the device leaving the location. In some implementations, method 2200 includes determining that the device has entered the particular location based on location data indicating the location of the device. In one example, in response to device 100 leaving room 204 or (e.g., based on location data 153) determining that the device's location is no longer within room 204, device 100 trims acoustic model 212 (e.g., "office" model 274 corresponding to room 204).

[0187] Although the examples provided above illustrate that... Figure 2 Method 2200 is executed in building 202; however, it should be understood that method 2200 is not limited to any particular location or location type. For example, in some implementations, the location corresponds to a specific restaurant, and the acoustic model is downloaded from an acoustic model library in response to determining that a device has entered that specific restaurant. In other implementations, the location may correspond to a park, subway station, car, train, airplane, a specific room in a user's home, a specific room in a museum, an auditorium or concert hall, etc.

[0188] By selecting an acoustic model corresponding to a specific location of the device, method 2200 enables the device to operate with higher accuracy compared to using a single acoustic model for all locations. Furthermore, changing the acoustic model allows the device to operate with higher accuracy without incurring the power consumption, memory requirements, and processing resource usage associated with retraining an existing acoustic model from scratch at the device for a specific context. Removing the acoustic model in response to the device leaving that location improves device operation by freeing up memory and resources associated with continuing to store acoustic models that are no longer used. This enables a reduction in power consumption associated with model storage and latency associated with subsequent searches of acoustic models stored at the device. Moreover, operating the device using a location-based acoustic model enables improved operation of the device itself, such as faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a location-specific, higher-accuracy acoustic model.

[0189] Figure 23 It is an explanation Figure 1 A flowchart illustrating an example of an operation method 2300 for device 100. Method 2300 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as... Figure 8 Instruction 860) is used to perform context-based model selection.

[0190] Method 2300 includes: in block 2302, selecting an acoustic model corresponding to a specific location at one or more processors of the device.

[0191] Method 2300 includes, at block 2304, receiving access permission for the acoustic model at the one or more processors based at least in part on the device's location matching the specific location. In some implementations, method 2300 includes receiving the access permission in response to detecting the device within the specific location. For example, device 100 may receive access permission in response to detecting device 100 in room 204. Figure 2 Access permission 214 for the "office" model 274. Device 100 can transmit data indicating the acoustic environment 210 (such as location data), and in response, the permission management system can send access permission, such as reference, to device 100. Figure 2 As described.

[0192] Method 2300 includes: at block 2306, using the acoustic model to process the input audio signal at one or more processors.

[0193] By selecting an acoustic model corresponding to a specific location of the device, method 2300 enables the device to operate with higher accuracy compared to using a single acoustic model for all locations. Furthermore, by preventing the use of the acoustic model except when the device is in a specific location, location-matched access permission for the acoustic model is maintained to ensure its security. This security allows the acoustic model to be downloaded to the device as pre-stored data, such as reducing peak network bandwidth usage and reducing latency associated with using the acoustic model, since the model may already be stored at the device and does not need to be downloaded upon entering a specific location. Moreover, operating the device using a location-based acoustic model enables improved operation of the device itself, such as faster convergence when performing iterative or dynamic processes (e.g., in noise cancellation techniques) due to the use of a location-specific, higher-accuracy acoustic model.

[0194] Figure 24 It is an explanation Figure 1 A flowchart illustrating an example of an operation method 2400 of device 100. Method 2400 can be initiated, controlled, or executed by device 100. For example, processor 110 can execute instructions from memory 108 (such as...) Figure 8 Instruction 860) is used to perform context-based model selection.

[0195] Method 2400 includes: at block 2402, detecting the context of the device at one or more processors of the device. For illustration, in some implementations, one or more processors 110 are configured to detect the context, such as context 142 detected by a context detector 142 based on sensor data 138. In an illustrative, non-limiting instance, this context corresponds to location or activity, such as driving in a car.

[0196] Method 2400 includes, at block 2404, sending a request to a remote device indicating the context. For illustration, in some implementations, one or more processors 110 are configured to initiate a request to a remote device (such as a remote computing device 818 (e.g., a server, which may be part of a service)) or a network memory of at least a portion of a storage model library 162 indicating the context (e.g., acoustic environment 210).

[0197] Method 2400 includes, at block 2406, receiving a model corresponding to the context. For illustration, in some implementations, one or more processors 110 are configured to receive model 112 from another server or network storage device 818 or a storage model library 162 in response to a request to send an indication of the context. In some implementations, model 112 is received as a compressed source model and decompressed by one or more processors 110 for use at device 100. In some implementations, the model is received based on private access. In an illustrative example, acoustic model 212 is received along with access permission 214 authorizing device 100 to access acoustic model 212 (e.g., along with a license to use model 212). In an illustrative example, the model is received based on access granted by a family member or friend of the user of device 100.

[0198] Method 2400 includes: at block 2408, using the model at the one or more processors while the context remains detected. For illustration, in some implementations, one or more processors 110 are configured to receive model 112 in response to a request to send an indication of context 142; and to continue using model 112 at the model-based application 192 while context 142 remains unchanged (e.g., temporarily using the received model 112 as long as context 142 is detected).

[0199] Method 2400 includes: at block 2410, pruning the model at the one or more processors in response to detecting a change in the context. For illustration, in some implementations, one or more processors 110 are configured to prune model 112 in response to detecting a change in context 142 (e.g., pruning model 112 once the context changes). In some implementations, pruning the model includes permanently deleting the model.

[0200] In some implementations, method 2400 includes generating at least one new sound category while the context remains detected, and pruning the model includes retaining the at least one new sound category. For illustration, in some implementations, one or more processors 110 are configured to generate new sound categories (such as by generating an updated model or a new model, as referenced). Figure 3-5 As described (e.g., update model 506), these new sound categories are preserved by storing the updated or new model (such as at memory 108 or via uploading to model library 162).

[0201] In conjunction with the described implementation, an apparatus includes means for receiving sensor data. For example, the means for receiving sensor data includes device 100, instruction 860, processor 804, processor(s) 806, context detector 140, microphone(s) 104, camera(s) 150, position sensor(s) 152, activity detector 154, other sensors(s) 156, CODEC 824, one or more other circuits or components configured to receive sensor data, or any combination thereof.

[0202] The equipment also includes means for determining context based on sensor data. For example, means for determining context based on sensor data includes device 100, instruction 860, processor 804, processor(s) 806, context detector 140, one or more circuits or components configured to determine context based on sensor data, or any combination thereof.

[0203] The equipment also includes means for selecting a model based on context. For example, means for selecting a model based on context includes device 100, instruction 860, processor 804, processor(s) 806, model selector 190, one or more circuits or components configured to select a model based on context, or any combination thereof.

[0204] The equipment also includes means for using a model to process input signals to generate context-dependent outputs. For example, means for using a model to process input signals to generate context-dependent outputs include device 100, instruction 860, processor 804, processor(s) 806, model-based application 192, one or more circuits or components, or any combination thereof, configured to use a model to process input signals to generate context-dependent outputs.

[0205] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of both. The various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in their functional form. Whether such functionality is implemented as hardware or processor-executable instructions depends on the specific application and the design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in different ways for each specific application, and such implementation decisions shall not be construed as departing from the scope of this disclosure.

[0206] The steps of the methods or algorithms described herein can be implemented directly in hardware, in a software module executed by a processor, or in a combination of both. The software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, removable disks, compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read and write information from / to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or user terminal. In an alternative, the processor and storage medium may reside as discrete components in a computing device or user terminal.

[0207] Specific aspects of this disclosure are described in the following first set of interrelated clauses:

[0208] According to Clause 1, a device includes one or more processors configured to: receive sensor data from one or more sensor devices; determine the context of the device based on the sensor data; select a model based on the context; and use the model to process an input signal to generate an output that varies depending on the context.

[0209] Clause 2 includes the device of Clause 1, and further includes: a position sensor coupled to the one or more processors, wherein the sensor data includes position data from the position sensor indicating the position of the device, and wherein the context is at least partially based on the position.

[0210] Clause 3 includes the device of Clause 1 or Clause 2, and further includes: a camera coupled to the one or more processors, wherein the sensor data includes image data from the camera corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

[0211] Clause 4 includes the device of any one of Clauses 1 to 3, and further includes: a microphone coupled to the one or more processors, wherein the sensor data includes audio data from the microphone corresponding to an audio scene, and wherein the context is at least partially based on the audio scene.

[0212] Clause 5 includes the device of any one of Clauses 1 to 4, and further includes: an activity detector coupled to the one or more processors, wherein the sensor data includes motion data from the activity detector corresponding to the motion of the device, and wherein the context is based at least in part on the motion of the device.

[0213] Clause 6 includes the device of any one of Clauses 1 to 5, and further includes: a memory coupled to the one or more processors, wherein the model is selected from a plurality of models stored in the memory.

[0214] Clause 7 includes any of the devices in Clauses 1 to 6, wherein: the model includes a sound event detection model, the input signal includes an audio signal, and the context-dependent output includes a classification of sound events in the audio signal.

[0215] Clause 8 includes the device of any one of Clauses 1 to 7, wherein: the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-dependent output includes text data representing speech in the audio signal.

[0216] Clause 9 includes any of the devices in Clauses 1 to 8, wherein: the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-dependent output includes NLP output data based on the text data.

[0217] Clause 10 includes the device of any one of Clauses 1 to 9, wherein: the model includes a noise reduction model, the input signal includes an audio signal, and the context-dependent output includes a noise-reduced audio signal based on the audio signal.

[0218] Clause 11 includes any of the devices in Clauses 1 to 10, wherein: the model is associated with automatic adjustment of the device operating mode, and wherein the context-dependent output includes signals for adjusting the device operating mode.

[0219] Clause 12 includes the device of any one of Clauses 1 to 11, and further includes: a modem coupled to the one or more processors and configured to receive the modem from the second device via wireless transmission.

[0220] Clause 13 includes the device of Clause 12, wherein: the context corresponds to the location of the device, and wherein the model includes an acoustic model corresponding to the specific location.

[0221] Clause 14 includes the device of Clause 13, wherein: the one or more processors are further configured to: receive access permission to the model via the modem based at least in part on the location of the device matching the particular location.

[0222] Clause 15 includes a device of any one of Clauses 1 to 14, wherein: the one or more processors are further configured to prune the model in response to determining that the context has changed.

[0223] Clause 16 includes any of the devices in Clauses 1 to 15, wherein: the model is downloaded from a library of shared models.

[0224] Clause 17 includes the device of Clause 16, wherein: the model includes a trained model uploaded to the library from another user device.

[0225] Clause 18 includes the equipment of Clause 16 or Clause 17, wherein: the library corresponds to the crowdsourcing model library.

[0226] Clause 19 includes any of the devices in Clauses 16 to 18, wherein: the library is included in a distributed context-aware system.

[0227] Clause 20 includes a device of any one of Clauses 1 to 19, wherein: the context includes a specific acoustic environment, and wherein the one or more processors are configured to: determine whether an available acoustic model library includes acoustic models specific to the specific acoustic environment and available to the one or more processors; and, in response to the absence of acoustic models specific to the specific acoustic environment available to the one or more processors, determine whether acoustic models of a general class for the specific acoustic environment are available to the one or more processors.

[0228] Clause 21 includes any of the devices in Clauses 1 to 20, wherein: the one or more processors are integrated into an integrated circuit.

[0229] Clause 22 includes any of the devices in Clauses 1 to 20, wherein: the one or more processors are integrated into a vehicle.

[0230] Clause 23 includes any of the devices described in Clauses 1 to 20, wherein: the one or more processors are integrated into at least one of: a mobile phone, a tablet computer device, a virtual reality headset, an augmented reality headset, a mixed reality headset, a wireless speaker device, a wearable device, a camera device, or a hearing aid device.

[0231] Specific aspects of this disclosure are described in the following second set of interrelated clauses:

[0232] According to Clause 24, a method includes: receiving sensor data from one or more sensor devices at one or more processors of the device; determining the context of the device at the one or more processors based on the sensor data; selecting a model at the one or more processors based on the context; and using the model at the one or more processors to process an input signal to generate an output that varies depending on the context.

[0233] Clause 25 includes the method of Clause 24, wherein: the sensor data includes location data of the device’s location, and wherein the context is based at least in part on the location.

[0234] Clause 26 includes the method of Clause 24 or Clause 25, wherein: the sensor data includes image data corresponding to the visual scene, and wherein the context is at least partially based on the visual scene.

[0235] Clause 27 includes the method of any one of Clauses 24 to 26, wherein: the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on the audio scene.

[0236] Clause 28 includes the method of any one of Clauses 24 to 27, wherein: the sensor data includes motion data corresponding to the motion of the device, and wherein the context is based at least in part on the motion of the device.

[0237] Clause 29 includes the method of any one of Clauses 24 to 28, wherein: the model is selected from a plurality of models stored in the memory of the device.

[0238] Clause 30 includes the method of any one of Clauses 24 to 29, wherein: the model includes a sound event detection model, the input signal includes an audio signal, and the context-dependent output includes a classification of sound events in the audio signal.

[0239] Clause 31 includes the method of any one of Clauses 24 to 30, wherein: the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-dependent output includes text data representing speech in the audio signal.

[0240] Clause 32 includes the method of any one of Clauses 24 to 31, wherein: the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-dependent output includes NLP output data based on the text data.

[0241] Clause 33 includes the method of any one of Clauses 24 to 32, wherein: the model includes a noise reduction model, the input signal includes an audio signal, and the context-dependent output includes a noise-reduced audio signal based on the audio signal.

[0242] Clause 34 includes the method of any one of Clauses 24 to 33, wherein: the model is associated with automatic adjustment of the device operating mode, and wherein the context-dependent output includes signals for adjusting the device operating mode.

[0243] Clause 35 includes the method of any one of Clauses 24 to 34, and further includes: receiving the model from a second device via wireless transmission.

[0244] Clause 36 includes the method of any one of Clauses 24 to 35, wherein: the context corresponds to the location of the device, and wherein the model includes an acoustic model corresponding to the specific location.

[0245] Clause 37 includes the method of Clause 36, and further includes: receiving access permission to the model based at least in part on the location of the device matching the specific location.

[0246] Clause 38 includes the methods of any one of Clauses 24 to 37, and further includes: pruning the model in response to determining that the context has changed.

[0247] Clause 39 includes the method of any one of Clauses 24 to 38, wherein: the model is downloaded from a library of shared models.

[0248] Clause 40 includes the method of Clause 39, wherein: the model includes a trained model uploaded to the library from another user device.

[0249] Clause 41 includes the methods of Clause 39 or Clause 40, wherein: the library corresponds to the crowdsourcing model library.

[0250] Clause 42 includes the methods of any one of Clauses 39 to 41, wherein: the library is included in a distributed context-aware system.

[0251] Clause 43 includes the method of any one of Clauses 24 to 42, wherein: the context includes a specific acoustic environment, and the method further includes: determining whether the available acoustic model library includes acoustic models specific to the specific acoustic environment and applicable to the device; and determining whether acoustic models of a general category for the specific acoustic environment are applicable to the device in response to the absence of acoustic models specific to the specific acoustic environment available to the device.

[0252] Specific aspects of this disclosure are described in the following third set of interrelated clauses:

[0253] According to Clause 44, an apparatus includes: means for receiving sensor data; means for determining a context based on the sensor data; means for selecting a model based on the context; and means for processing an input signal using the model to generate an output that varies depending on the context.

[0254] Clause 45 includes the device of Clause 44, wherein: the sensor data includes location data of the device’s location, and wherein the context is based at least in part on the location.

[0255] Clause 46 includes the device of Clause 44 or Clause 45, wherein: the sensor data includes image data corresponding to a visual scene, and wherein the context is based at least in part on the visual scene.

[0256] Clause 47 includes the device of any one of Clauses 44 to 46, wherein: the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on the audio scene.

[0257] Clause 48 includes a device of any one of Clauses 44 to 47, wherein: the sensor data includes motion data corresponding to the motion of the device, and wherein the context is based at least in part on the motion of the device.

[0258] Clause 49 includes the device of any one of Clauses 44 to 48, and further includes: means for storing a model, wherein the model is selected from a plurality of models stored at the means for storing the model.

[0259] Clause 50 includes the device of any one of Clauses 44 to 49, wherein: the model includes a sound event detection model, the input signal includes an audio signal, and the context-dependent output includes a classification of sound events in the audio signal.

[0260] Clause 51 includes the device of any one of Clauses 44 to 50, wherein: the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-dependent output includes text data representing speech in the audio signal.

[0261] Clause 52 includes the device of any one of Clauses 44 to 51, wherein: the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-dependent output includes NLP output data based on the text data.

[0262] Clause 53 includes the device of any one of Clauses 44 to 52, wherein: the model includes a noise reduction model, the input signal includes an audio signal, and the context-dependent output includes a noise-reduced audio signal based on the audio signal.

[0263] Clause 54 includes the device of any one of Clauses 44 to 53, wherein: the model is associated with automatic adjustment of the device operating mode, and wherein the context-dependent output includes signals for adjusting the device operating mode.

[0264] Clause 55 includes any of the devices in Clauses 44 to 54, wherein: the model is received from the second device via wireless transmission.

[0265] Clause 56 includes the device of any one of Clauses 44 to 55, wherein: the context corresponds to the location of the device, and wherein the model includes an acoustic model corresponding to the specific location.

[0266] Clause 57 includes the device of Clause 56, wherein: access to the model is granted based at least in part on the device’s location matching the particular location.

[0267] Clause 58 includes the device of Clause 56, and further includes: means for removing the model in response to the device leaving the particular location.

[0268] Clause 59 includes any of the devices in Clauses 44 to 58, wherein: the model is downloaded from a library of shared models.

[0269] Clause 60 includes the devices of Clause 59, wherein: the model includes a trained model uploaded to the library from another user device.

[0270] Clause 61 includes the equipment of Clause 59 or Clause 60, wherein: the library corresponds to the crowdsourcing model library.

[0271] Clause 62 includes any of the devices in Clauses 59 to 61, wherein: the library is included in a distributed context-aware system.

[0272] Specific aspects of this disclosure are described in the following fourth set of interrelated clauses:

[0273] According to Clause 63, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: receive sensor data from one or more sensor devices; determine a context relating to the sensor data; select a model based on the context; and use the model to process an input signal to generate an output that varies depending on the context.

[0274] Clause 64 includes the non-transient computer-readable storage medium of Clause 63, wherein: the sensor data includes location data of the device’s location, and wherein the context is based at least in part on the location.

[0275] Clause 65 includes the non-transient computer-readable storage medium of Clause 63 or Clause 64, wherein: the sensor data includes image data corresponding to a visual scene, and wherein the context is at least partially based on the visual scene.

[0276] Clause 66 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 65, wherein: the sensor data includes audio corresponding to an audio scene, and wherein the context is at least partially based on the audio scene.

[0277] Clause 67 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 66, wherein: the sensor data includes motion data corresponding to the motion of the device, and wherein the context is based at least in part on the motion of the device.

[0278] Clause 68 includes any of the nontransient computer-readable storage media of Clauses 63 to 67, wherein: the model is selected from a plurality of models stored in the memory of the device.

[0279] Clause 69 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 68, wherein: the model includes a sound event detection model, the input signal includes an audio signal, and the context-dependent output includes a classification of sound events in the audio signal.

[0280] Clause 70 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 69, wherein: the model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-dependent output includes text data representing speech in the audio signal.

[0281] Clause 71 includes any of the nontransient computer-readable storage media of Clauses 63 to 70, wherein: the model includes a natural language processing (NLP) model, the input signal includes text data, and the context-dependent output includes NLP output data based on the text data.

[0282] Clause 72 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 71, wherein: the model includes a noise reduction model, the input signal includes an audio signal, and the context-dependent output includes a noise-reduced audio signal based on the audio signal.

[0283] Clause 73 includes a non-transient computer-readable storage medium of any of Clauses 63 to 72, wherein: the model is associated with automatic adjustment of a device operating mode, and wherein the context-dependent output includes signals for adjusting the device operating mode.

[0284] Clause 74 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 73, wherein: these instructions further cause the processor to receive the model from a second device via wireless transmission.

[0285] Clause 75 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 74, wherein: the context corresponds to the location of the device, and wherein the model includes an acoustic model corresponding to the specific location.

[0286] Clause 76 includes the non-transient computer-readable storage medium of Clause 75, wherein: these instructions further enable the processor to receive access permission to the model based at least in part on the location of the device matching the particular location.

[0287] Clause 77 includes the non-transient computer-readable storage medium of Clause 75, wherein: these instructions further cause the processor to remove the model in response to the device leaving the particular location.

[0288] Clause 78 includes any of the non-transient computer-readable storage media of Clauses 63 to 77, wherein: the model is downloaded from a library of shared models.

[0289] Clause 79 includes the non-transient computer-readable storage medium of Clause 78, wherein: the model includes a trained model uploaded to the library from another user device.

[0290] Clause 80 includes the non-transient computer-readable storage medium of Clause 78 or Clause 79, wherein: the library corresponds to the crowdsourcing model library.

[0291] Clause 81 includes any of the non-transient computer-readable storage media of Clauses 78 to 80, wherein: the library is included in a distributed context-aware system.

[0292] Clause 82 includes a non-transient computer-readable storage medium of any one of Clauses 63 to 81, wherein: the context includes a particular acoustic environment, and wherein the instructions further cause the processor to: determine whether an available acoustic model library includes acoustic models specific to the particular acoustic environment and available for use with the device; and, in response to the absence of acoustic models specific to the particular acoustic environment available for use with the device, determine whether acoustic models of a general class for the particular acoustic environment are available for use with the device.

[0293] Specific aspects of this disclosure are described in the following fifth set of interrelated clauses:

[0294] According to Clause 83, a device includes one or more processors configured to: select an acoustic model corresponding to a specific room in a building where the device is located; and use the acoustic model to process an input audio signal.

[0295] Clause 84 includes the device of Clause 83, wherein the one or more processors are configured to download the acoustic model from the acoustic model library in response to determining that the device has entered the particular room.

[0296] Clause 85 includes the device of Clause 83 or 84, wherein the one or more processors are further configured to remove the acoustic model in response to the device leaving the particular room.

[0297] Clause 86 includes the device of any one of Clauses 83 to 85, and further includes: one or more microphones configured to generate the input audio signal.

[0298] Clause 87 includes the device of any one of Clauses 83 to 86, and further includes: one or more sensor devices coupled to the one or more processors, and the one or more sensor devices are configured to generate sensor data indicating the location of the device, and wherein the one or more processors are configured to select the acoustic model based on the sensor data.

[0299] Clause 88 includes the device of any one of Clauses 83 to 87, and further includes: a modem coupled to the one or more processors, the modem being configured to receive location data indicating the location of the device, and wherein the one or more processors are configured to select the acoustic model based on the location data.

[0300] Specific aspects of this disclosure are described in the following sixth set of interrelated clauses:

[0301] According to Clause 89, a method includes: selecting an acoustic model at one or more processors of the device that corresponds to a specific room in a building where the device is located; and using the acoustic model at the one or more processors to process an input audio signal.

[0302] Clause 90 includes the method of Clause 89, and further includes: downloading the acoustic model from the acoustic model library in response to determining that the device has entered the particular room.

[0303] Clause 91 includes the methods of Clause 89 or Clause 90, and further includes: removing the acoustic model in response to the device leaving the particular room.

[0304] Clause 92 includes the method of any one of Clauses 89 to 91, and further includes: selecting the acoustic model based on sensor data indicating the location of the device.

[0305] Clause 93 includes the method of any one of Clauses 89 to 91, and further includes: selecting the acoustic model based on location data indicating the location of the device.

[0306] According to Clause 94, an apparatus includes: means for selecting an acoustic model corresponding to a specific room in a building where the apparatus is located; and means for using the acoustic model to process an input audio signal.

[0307] According to Clause 95, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: select an acoustic model corresponding to a specific room in a building in which the device is located; and use the acoustic model to process an input audio signal.

[0308] Specific aspects of this disclosure are described in the following seventh set of interrelated clauses:

[0309] According to Clause 96, an apparatus includes: one or more processors configured to: select a personalized acoustic model for a user of the apparatus from a plurality of personalized acoustic models corresponding to the vehicle in response to the apparatus entering a vehicle; and use the personalized acoustic model to process an input audio signal.

[0310] Clause 97 includes the device of Clause 96, wherein the one or more processors are configured to download the personalized acoustic model from an acoustic model library in response to determining that the device has entered the vehicle.

[0311] Clause 98 includes the device of Clause 96 or Clause 97, wherein the one or more processors are further configured to remove the personalized acoustic model in response to the device leaving the vehicle.

[0312] Clause 99 includes the device of any one of Clauses 96 to 98, and further includes: one or more microphones configured to generate the input audio signal.

[0313] Clause 100 includes the device of any one of Clauses 96 to 99, and further includes: one or more sensor devices coupled to the one or more processors, and the one or more sensor devices are configured to generate sensor data indicating the location of the device, and wherein the one or more processors are configured to determine, based on the sensor data, that the device has entered the vehicle.

[0314] Clause 101 includes the device of any one of Clauses 96 to 100, and further includes: a modem coupled to the one or more processors and configured to receive location data indicating the location of the device, and wherein the one or more processors are configured to determine, based on the location data, that the device has entered the vehicle.

[0315] Specific aspects of this disclosure are described in the following group of eight interconnected clauses:

[0316] According to Clause 102, a method includes: selecting a personalized acoustic model for the user from a plurality of personalized acoustic models corresponding to the vehicle at one or more processors of the device and in response to detecting that the user has entered the vehicle; and using the personalized acoustic model at the one or more processors to process an input audio signal.

[0317] Clause 103 includes the method of Clause 102, and further includes: downloading the personalized acoustic model from the acoustic model library in response to determining that the user has entered the vehicle.

[0318] Clause 104 includes the methods of Clause 102 or Clause 103, and further includes: removing the personalized acoustic model in response to the user leaving the vehicle.

[0319] Clause 105 includes the method of any one of Clauses 102 to 104, and further includes: determining that the user has entered the vehicle based on sensor data indicating the user's location.

[0320] According to Clause 106, an apparatus includes: means for selecting a personalized acoustic model for a user from a plurality of personalized acoustic models corresponding to the vehicle in response to detecting that a user has entered the vehicle; and means for processing an input audio signal using the personalized acoustic model.

[0321] According to Clause 107, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: select a personalized acoustic model for the user from a plurality of personalized acoustic models corresponding to the vehicle in response to detecting that the user has entered the vehicle; and use the personalized acoustic model to process an input audio signal.

[0322] Specific aspects of this disclosure are described in the following ninth set of interconnected clauses:

[0323] According to Clause 108, a device includes: one or more processors configured to: download an acoustic model corresponding to a specific location in which the device is located; use the acoustic model to process an input audio signal; and remove the acoustic model in response to the device leaving the location.

[0324] Clause 109 includes the device of Clause 108, wherein: the location corresponds to a specific restaurant, and wherein the acoustic model is downloaded from an acoustic model library in response to determining that the device has entered the specific restaurant.

[0325] Clause 110 includes the device of Clause 108 or 109, and further includes: one or more microphones configured to generate the input audio signal.

[0326] Clause 111 includes the device of any one of Clauses 108 to 110, and further includes: one or more sensor devices coupled to the one or more processors, and the one or more sensor devices are configured to generate sensor data indicating the location of the device, and wherein the one or more processors are configured to determine, based on the sensor data, that the device has entered the particular location.

[0327] Clause 112 includes the device of any one of Clauses 108 to 110, and further includes: a modem coupled to the one or more processors and configured to receive location data indicating the location of the device, and wherein the one or more processors are configured to determine, based on the location data, that the device has entered the particular location.

[0328] Specific aspects of this disclosure are described in the following tenth set of interrelated clauses:

[0329] According to Clause 113, a method includes: downloading an acoustic model corresponding to a specific location of the device at one or more processors; using the acoustic model at the one or more processors to process an input audio signal; and removing the acoustic model at the one or more processors in response to the device leaving the location.

[0330] Clause 114 includes the method of Clause 113, wherein: the location corresponds to a specific restaurant, and wherein the acoustic model is downloaded from an acoustic model library in response to determining that the device has entered the specific restaurant.

[0331] Clause 115 includes the methods of Clause 113 or Clause 114, and further includes: determining that the device has entered the particular location based on sensor data indicating the location of the device.

[0332] Clause 116 includes the methods of Clause 113 or 114, and further includes: determining that the device has entered the particular location based on location data indicating the location of the device.

[0333] According to Clause 117, an apparatus includes: means for downloading an acoustic model corresponding to a specific location of the apparatus therein; means for using the acoustic model to process an input audio signal; and means for removing the acoustic model in response to the apparatus leaving the location.

[0334] According to Clause 118, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: download an acoustic model corresponding to a specific location in which the device is located; use the acoustic model to process an input audio signal; and remove the acoustic model in response to the device leaving the location.

[0335] Specific aspects of this disclosure are described in the following eleventh set of interconnected clauses:

[0336] According to Clause 119, an apparatus includes: one or more processors configured to: select an acoustic model corresponding to a particular location; receive access permission to the acoustic model based at least in part on the fact that the location of the apparatus matches the particular location; and use the acoustic model to process an input audio signal.

[0337] Clause 120 includes the device of Clause 119, and further includes a modem, wherein the one or more processors are further configured to receive access permission via the modem in response to the detection of the device in the particular location.

[0338] Specific aspects of this disclosure are described in the following set of twelfth related clauses:

[0339] According to Clause 121, a method includes: selecting an acoustic model corresponding to a specific location at one or more processors of a device; receiving access permission to the acoustic model at the one or more processors based at least in part on the matching of the device's location with the specific location; and using the acoustic model at the one or more processors to process an input audio signal.

[0340] Clause 122 includes the method of Clause 121, and further includes: receiving access permission in response to detecting the device within the specific location.

[0341] According to Clause 123, an apparatus includes: means for selecting an acoustic model corresponding to a particular location; means for receiving access permission to the acoustic model based at least in part on the location of the apparatus matching the particular location; and means for using the acoustic model to process an input audio signal.

[0342] According to Clause 124, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: select an acoustic model corresponding to a particular location; receive access permission to the acoustic model based at least in part on the fact that the location of the device matches the particular location; and use the acoustic model to process an input audio signal.

[0343] Specific aspects of this disclosure are described in the following set of thirteen interconnected clauses:

[0344] According to Clause 125, an apparatus includes one or more processors configured to: detect a context of the apparatus; send a request to a remote device indicating the context; receive a model corresponding to the context; use the model while the context remains detected; and prune the model in response to detecting a change in the context.

[0345] Clause 126 includes the device of Clause 125, wherein: the one or more processors are further configured to generate at least one new sound category when the context is still detected, and wherein pruning the model includes retaining the at least one new sound category.

[0346] Clause 127 includes the equipment of Clause 125 or Clause 126, wherein: trimming the model includes permanently deleting the model.

[0347] Clause 128 includes any of the devices in Clauses 125 to 127, wherein: the one or more processors are further configured to receive the model based on private access.

[0348] According to Clause 129, a method includes: detecting the context of the device at one or more processors of the device; sending a request to a remote device indicating the context; receiving a model corresponding to the context; using the model at the one or more processors while the context remains detected; and pruning the model at the one or more processors in response to detecting a change in the context.

[0349] Clause 130 includes the method of Clause 129, further comprising: generating at least one new sound category while the detection is maintained in the context, and wherein pruning the model includes retaining the at least one new sound category.

[0350] Clause 131 includes the methods of Clause 129 or Clause 130, wherein: pruning the model includes permanently deleting the model.

[0351] Clause 132 includes the method of any one of Clauses 129 to 131, wherein: the model is received based on private access.

[0352] According to Clause 133, an apparatus includes: means for sending a request to a remote device indicating the context; means for receiving a model corresponding to the context; means for using the model while the context is still detected; and means for pruning the model in response to detecting a change in the context.

[0353] According to Clause 134, a non-transient computer-readable storage medium comprising instructions that, when executed by a processor of a device, cause the processor to: detect a context of the device; send a request to a remote device indicating the context; receive a model corresponding to the context; use the model while the context remains detected; and prune the model in response to detecting a change in the context.

[0354] The prior description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein can be applied to other aspects without departing from the scope of this disclosure. Therefore, this disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the appended claims.

Claims

1. An apparatus comprising: The memory is configured to store one or more models; as well as One or more processors coupled to the memory, the one or more processors being configured to: Receive sensor data from one or more sensor devices, wherein the sensor data includes image data from a camera, the image data corresponding to a visual scene; The context of the device is determined based on the sensor data, the context being at least partially based on the visual scene, wherein the context is determined based on a trained classifier that is trained to determine the context based on the trained sensor data. The device receives the results of a query based on the context and selects a specific model based on the query results, the specific model including a sound event classification SEC model. The selection of the specific model causes the device to download the specific model from a cloud-based model library and store the downloaded specific model in the memory. as well as The specific model is used to process the input signal to generate an output that varies depending on the context.

2. The device as claimed in claim 1, further comprising: A position sensor coupled to the one or more processors, wherein the sensor data includes position data from the position sensor indicating the position of the device, and wherein the context is at least partially based on the position.

3. The device as claimed in claim 1, further comprising: The camera coupled to the one or more processors.

4. The device of claim 1, further comprising: A microphone coupled to one or more processors, wherein the sensor data includes audio data from the microphone that is different from the input signal, the audio data corresponding to an audio scene, and wherein the context is at least partially based on the audio scene.

5. The device of claim 1, further comprising: An activity detector coupled to one or more processors, wherein the sensor data includes motion data from the activity detector corresponding to the motion of the device, and wherein the context is at least partially based on the motion of the device.

6. The device of claim 1, wherein the input signal comprises an audio signal, and the context-dependent output comprises a classification of sound events in the audio signal.

7. The device of claim 1, wherein the one or more processors are further configured to select a second specific model based on the query, wherein the second specific model includes an automatic speech recognition model, the input signal includes an audio signal, and the context-dependent output includes text data representing speech in the audio signal.

8. The device of claim 1, wherein the one or more processors are further configured to select a second specific model based on the query, wherein the second specific model includes a natural language processing (NLP) model, the input signal includes text data, and the context-dependent output includes NLP output data based on the text data.

9. The device of claim 1, wherein the one or more processors are further configured to select a second specific model based on the query, wherein the second specific model includes a noise reduction model, the input signal includes an audio signal, and the context-dependent output includes a noise-reduced audio signal based on the audio signal.

10. The device of claim 1, wherein the one or more processors are further configured to select a second specific model based on the query, wherein the second specific model is associated with automatic adjustment of the device operating mode, and wherein the context-dependent output includes signals for adjusting the device operating mode.

11. The device of claim 1, further comprising: A modem, which is coupled to one or more processors and configured to receive the specific model from the library via wireless transmission.

12. The device of claim 11, wherein the context corresponds to the location of the device, and wherein the particular model includes an acoustic model corresponding to the particular location.

13. The device of claim 12, wherein the one or more processors are further configured to receive access permission to the particular model via the modem, at least in part based on the location of the device matching the particular location.

14. The device of claim 1, wherein the one or more processors are further configured to prune the particular model in response to determining that the context has changed.

15. The device of claim 1, wherein the one or more processors are integrated into an integrated circuit.

16. The device of claim 1, wherein the one or more processors are integrated into a vehicle.

17. The device of claim 1, wherein the one or more processors are integrated into at least one of: a mobile phone, a tablet computer device, a virtual reality headset, an augmented reality headset, a mixed reality headset, a wireless speaker device, a wearable device, a camera device, or a hearing aid device.

18. The device of claim 1, wherein the trained classifier includes an image recognition model trained to determine the context based on objects detected in the training sensor data.

19. A context-based model selection method, comprising: The device receives sensor data from one or more sensor devices at one or more processors, wherein the sensor data includes image data from a camera that corresponds to a visual scene; The context of the device is determined at one or more processors based on the sensor data, the context being at least partially based on the visual scene, wherein the context is determined based on a trained classifier trained to determine the context based on training sensor data. The device receives the results of a query based on the context, and selects a specific model at one or more processors based on the results of the query, the specific model including a sound event classification SEC model, the selection of the specific model causing the device to download the specific model from a cloud-based model library and store the downloaded specific model in the device's memory; as well as The specific model is used at one or more processors to process the input signal to generate an output that varies depending on the context.

20. The method of claim 19, further comprising receiving the particular model from the library via wireless transmission.

21. The method of claim 19, wherein the context corresponds to the location of the device, and wherein the particular model includes an acoustic model corresponding to the particular location.

22. The method of claim 21, further comprising: Access to the specific model is granted at least in part based on the device's location matching the specific location.

23. The method of claim 19, further comprising pruning the particular model in response to determining that the context has changed, wherein pruning the particular model includes: Save at least one new sound category associated with the context; as well as Delete the specific model from one or more models stored in the memory.

24. The method of claim 19, wherein the library corresponds to a library of shared models.

25. The method of claim 24, wherein the particular model comprises a trained model uploaded to the library from another user device.

26. The method of claim 25, wherein the library corresponds to a crowdsourcing model library.

27. The method of claim 25, wherein the library is included in a distributed context-aware system.

28. The method of claim 19, wherein the context includes a specific acoustic environment, and the method further comprises: Determine whether the model library includes acoustic models specific to the particular acoustic environment and applicable to the device; as well as In response to the absence of an acoustic model specific to the particular acoustic environment available for the device, it is determined whether a general category of acoustic models for the particular acoustic environment can be used for the device.

29. An apparatus comprising: A device for receiving sensor data, the sensor data including image data corresponding to a visual scene; A means for determining a context based on the sensor data, the context being at least partially based on the visual scene, wherein the context is determined based on a trained classifier trained to determine the context based on training sensor data; A means for receiving the results of a query based on the context; A device for selecting a specific model based on the results of the query, the specific model including a sound event classification (SEC) model, the selection of the specific model causing the device to download the specific model from a cloud-based model library and store the downloaded specific model; as well as A means for using the specific model to process input signals to generate outputs that vary depending on the context.

30. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processor, cause the processor to: Receive sensor data from one or more sensor devices, wherein the sensor data includes image data from a camera, the image data corresponding to a visual scene; The context is determined based on the sensor data, the context being at least partially based on the visual scene, wherein the context is determined based on a trained classifier that is trained to determine the context based on the trained sensor data. The processor receives the results of a query based on the context and selects a specific model based on the query results, the specific model including a sound event classification SEC model. The selection of the specific model causes the processor to download the specific model from a cloud-based model library and store the downloaded specific model in a memory coupled to the processor. as well as The specific model is used to process the input signal to generate an output that varies depending on the context.

Images