Systems and methods for multi-device learning and inference in ambient computing environments

Abstract

Systems and methods for multi-device learning and inference in ambient computing environments. In some aspects, the present technology discloses systems and methods for performing cross-device learning, where a new device can be trained based on supervisory signals from existing devices in an ambient computing environment. In some aspects, the present technology discloses systems and methods for performing multi-device inference across two or more devices in an ambient computing environment. Likewise, in some aspects, the present technology discloses systems and methods for training models that are robust to the addition or removal of one or more devices from an ambient computing environment.

Description

Background Technology

[0001] Today, when a user activates a smart device, it typically operates within a surrounding computing environment that already includes several other smart devices, some of which may have overlapping functionalities. For example, a user might have one or more cameras monitoring areas inside and outside their home, one or more smart speakers and / or smart displays that listen to and respond to voice commands, a smart thermostat monitoring the temperature in one or more rooms, a smart fire alarm monitoring for smoke and / or high heat, and a security system monitoring the opening of various doors and windows, movement within a given room, the sound of breaking glass, and so on. Furthermore, the surrounding computing environment can also include one or more devices that can come and go temporarily or permanently but may have similar functionalities (e.g., smartphones, smartwatches, and personal computers that can also be configured to listen to and respond to voice commands).

[0002] Currently, many smart devices are configured to operate largely independently of each other to perform their designated tasks or sets of tasks. For example, a smart speaker might be configured to run its own audio-based model to detect voice commands, and a doorbell camera might be configured to run its own set of different visual and audio processing models to detect when a person (e.g., homeowner, family member) arrives at the homeowner's front door. While these devices can all be configured to listen to and recognize sounds in the same general environment, they can each be configured to learn and infer what they are hearing. This can lead to a waste of time spent training their models in new environments for new devices, as well as a waste of the total power and computation cycles required to run inference tasks on multiple devices at once. Furthermore, configuring each device to independently monitor and perceive its environment may waste the opportunity for various devices to benefit from the different perspectives and sensing capabilities of other smart devices also present in the surrounding computing environment. Summary of the Invention

[0003] This technology relates to systems and methods for multi-device learning and inference in an ambient computing environment. In some aspects, this technology discloses systems and methods for performing cross-device learning, wherein new devices can be trained based on regulatory signals from existing devices in the ambient computing environment. Furthermore, in some aspects, this technology discloses systems and methods for performing multi-device inference across two or more devices in an ambient computing environment. Similarly, in some aspects, this technology discloses systems and methods for training models that are robust to adding or removing one or more devices from the ambient computing environment.

[0004] In one aspect, this disclosure describes a computer-implemented method for training a model to perform inference in a surrounding computing environment with multiple devices, the method comprising: (i) identifying a multi-device inference paradigm to be used for performing inference using the model based on inputs from one or more of the multiple devices by one or more processors of a processing system; (ii) for each of a plurality of events: generating a first prediction by one or more processors using the model according to the multi-device inference paradigm based on a first input set, the first input set including at least one input from each of the multiple devices; generating a second prediction by said one or more processors using said model according to the multi-device inference paradigm based on a second input set, the second input set including a modified copy of the first input set; generating a loss value by one or more processors based on the first prediction and the second prediction; and (iii) modifying one or more parameters of the model by one or more processors based on the loss value generated for at least one given event. In some aspects, each of the multiple events occurs during a given time period, and the first input set for each given event is based on data sensed by the multiple devices during the given time period. In some aspects, the given time period is a non-contiguous time period consisting of two or more separate time periods. In some aspects, one or more processors are configured to generate each first prediction, each second prediction, and each loss value during a given time period, and for each given event, a first input set is based on data sensed prior to the given time period. In some aspects, for each given event among multiple events, a second input set includes a modified copy of the first input set, which includes one or more selected inputs from the first input set and omits one or more other inputs from the first input set. In some aspects, the method further includes, for each given event among multiple events, one or more processors determining, based on a first value, to omit one or more other inputs. In some aspects, the method further includes, by one or more processors, modifying the first value based on the frequency with which a device associated with the first input has been present in the surrounding computing environment during a time period. In some aspects, the method further includes, by the one or more processors, modifying the first value based on how much power a device associated with the first input has consumed during a time period. In some aspects, for each given event among multiple events, the second input set includes a modified copy of the first input set, wherein at least a first input of the first input set is replaced by a copy of a second input of the first input set. In some aspects, for each given event among a plurality of events, the second input set includes a modified copy of the first input set, wherein at least a first input of the first input set is replaced by an upsampled version of a second input of the first input set. In some aspects, for each given event among a plurality of events, the second input set includes a modified copy of the first input set, wherein at least a first input of the first input set is replaced by a downsampled version of a second input of the first input set.In some aspects, for a given event among the plurality of events, the second input set includes a modified copy of the first input set, in which at least a first input of the first input set is replaced by a synthetic input based on two or more other inputs of the first input set. In some aspects, the synthetic input is the average of two or more other inputs of the first input set.

[0005] In another aspect, this disclosure describes a system for training a model to perform inference in a surrounding computing environment having multiple devices, the system comprising: a memory; and one or more processors coupled to the memory and configured to: (i) identify a multi-device inference paradigm to be used for performing inference using the model based on inputs from one or more of the multiple devices; (ii) for each of a plurality of events: generating a first prediction based on a first input set using the model according to the multi-device inference paradigm, the first input set including at least one input from each of the multiple devices; generating a second prediction based on a second input set using the model according to the multi-device inference paradigm, the second input set including a modified copy of the first input set; generating a loss value based on the first prediction and the second prediction; and (iii) modifying one or more parameters of the model based on the loss value generated for at least one given event. In some aspects, the one or more processors are also configured to, for each of the plurality of events, generate a second prediction based on a second input set including a modified copy of the first input set, the modified copy including one or more selected inputs from the first input set and omitting one or more other inputs from the first input set. In some aspects, one or more processors are further configured to, for each given event of a plurality of events, determine, based on a first value, to omit one or more other inputs. In some aspects, the one or more processors are further configured to modify the first value based on the frequency with which a device associated with the first input has been present in the surrounding computing environment during a time period. In some aspects, the one or more processors are further configured to modify the first value based on how much power a device associated with the first input has consumed during a time period. In some aspects, the one or more processors are further configured to, for each given event of the plurality of events, generate a second prediction based on a second input set including a modified copy of the first input set, in which at least a first input of the first input set is replaced by a copy, upsampled, or downsampled copy of a second input of the first input set. In some aspects, the one or more processors are further configured to, for each given event of the plurality of events, generate a second prediction based on a second input set including a modified copy of the first input set, in which at least a first input of the first input set is replaced by a synthetic input based on two or more other inputs of the first input set. Attached Figure Description

[0006] Figure 1 This is a functional diagram of the surrounding computing environment based on an example of an aspect of this disclosure.

[0007] Figure 2 This is a functional diagram of an example processing system based on aspects of this disclosure.

[0008] Figure 3 This is a flowchart illustrating an exemplary method for training a new device in an ambient computing environment according to aspects of this disclosure.

[0009] Figure 4 This is a flowchart illustrating an exemplary method for performing multi-device inference in an ambient computing environment according to aspects of this disclosure.

[0010] Figure 5 This is a flowchart illustrating an exemplary method for training a model to be robust to adding or removing devices from the surrounding computing environment, according to aspects of this disclosure.

[0011] Figure 6 This is a flowchart illustrating an exemplary method for training a model to be robust to adding or removing devices from the surrounding computing environment, according to aspects of this disclosure. Detailed Implementation

[0012] This technology will now be described with respect to the following exemplary systems and methods.

[0013] Example System

[0014] Figure 1 A high-level system diagram of an exemplary ambient computing environment 100 according to various aspects of this disclosure is schematically illustrated. In this respect, Figure 1 An exemplary ambient computing environment 100 includes a camera 102, a speaker 104, a mobile phone or tablet computer 106, and a personal computer 108, all connected to one or more networks 110. In this example, it is assumed that the camera 102 is capable of recording both audio and video, such as a security camera, doorbell camera, web conferencing camera, etc. Similarly, it is assumed that the speaker 104 is a smart speaker equipped with a microphone (not shown) and configured to listen to and respond to voice-based input (e.g., verbal commands such as playing a given song, checking the weather forecast, etc.). The mobile phone / tablet computer 106 and the personal computer 108 are also assumed to have microphones and are configured to listen to and respond to voice-based input (e.g., verbal commands for an automation assistant, dictation for voice-to-text, etc.).

[0015] One or more networks 110 can be any suitable number of networks and / or combinations thereof capable of providing connectivity between camera 102, speaker 104, mobile phone / tablet computer 106, and personal computer 108. Therefore, in some aspects of this technology, network 110 can include a single local area network (LAN), a combination of two or more LANs, a combination of a LAN and the Internet, etc. Similarly, one or more devices in the surrounding computing environment can gain access to network 110 through another device. For example, camera 102 can be configured to communicate with mobile phone 106 using Bluetooth, which in turn enables camera 102 to communicate with network 110. As will be understood, Figure 1 The exemplary ambient computing environment 100 shown is for illustrative purposes only, and the technology is not limited to any particular topology.

[0016] As will be further explained below, one or more of devices 102-108 can be configured to store and run one or more models for perceiving their environment. Similarly, in some aspects of this technology, one or more models can be stored on separate devices, such as local or remote processing systems (not shown) accessible via one or more networks 110. For example, one or more models can be stored on a remote database, and one or more of devices 102-108 can be configured to retrieve models from the remote database to perform inference and perceive their environment. In some aspects, one or more of devices 102-108 can also be configured to use a remote processing system to perform inference. For example, one or more of devices 102-108 can send data received from their environment (e.g., audio, video) or a partially processed representation of it (e.g., vector representations generated by one or more models on the device) to the remote processing system, which then feeds that data to one or more of its own models and responds with classifications output by those models.

[0017] Figure 2 A high-level system diagram 200 is shown for an exemplary processing system for performing the methods described herein. Processing system 202 may represent any intelligent device (e.g., [device name missing]) that can be included in an ambient computing environment (e.g., ambient computing environment 100). Figure 1 The processing system 202 can be a local or remote processing system configured to store one or more models for use by one or more intelligent devices in the surrounding computing environment. Furthermore, the processing system 202 can be a local or remote processing system configured to process data received from one or more intelligent devices in the surrounding computing environment. Additionally, the processing system 202 can be a distributed system. Therefore, in some aspects of this technology, the processing system 202 can include a collection of intelligent devices in the surrounding computing environment, a cloud computing system, a combination thereof, etc.

[0018] The processing system 202 may include one or more processors 204 and a memory 206 for storing instructions 208 and data 210. The processing system 202 may also optionally include one or more sensors 212 for sensing data in the surrounding computing environment. Where applicable, the data 210 may include one or more models for processing data sensed by at least one sensor 212 and / or data provided to the processing system by one or more other devices in the surrounding computing environment, as further described below.

[0019] The processing system described herein can be implemented on any type of computing device, such as any type of general-purpose computing device, server, or combination thereof, and may also include other components typically found in general-purpose computing devices or servers. Memory 206 stores information accessible by one or more processors 204, including instructions 208 and data 210 that can be executed or otherwise used by processor 204. Memory 206 can be any non-transitory type capable of storing information accessible by processor 204. For example, memory 206 may include non-transitory media such as hard disk drives, memory cards, optical discs, solid-state drives, magnetic tape storage, etc. Computing devices suitable for the role described herein may include different combinations of the foregoing, whereby different portions of instructions and data are stored on different types of media.

[0020] In all cases, the computing device described herein may also include any other components typically used in conjunction with the computing device, such as a user interface subsystem. The user interface subsystem may include one or more user inputs (e.g., a mouse, keyboard, touchscreen, and / or microphone) and one or more electronic displays (e.g., a monitor with a screen or any other electrical device operable to display information). Output devices other than electronic displays, such as speakers, lights, and vibrating, pulsating, or haptic elements, may also be included in the computing device described herein.

[0021] The one or more processors included in each computing device can be any conventional processor, such as a commercially available central processing unit (“CPU”), graphics processing unit (“GPU”), tensor processing unit (“TPU”), etc. Alternatively, the one or more processors can be special-purpose devices, such as ASICs or other hardware-based processors. Each processor can have multiple cores capable of operating in parallel. The processor, memory, and other components of a single computing device can be stored within a single physical enclosure or can be distributed across two or more enclosures. Similarly, the memory of a computing device can include hard disk drives or other storage media located in an enclosure different from the processor enclosure, such as in an external database or networked storage device. Therefore, references to processors or computing devices will be understood to include references to collections of processors or computing devices or memory that may or may not operate in parallel, as well as one or more servers in a load-balanced server cluster or cloud-based system.

[0022] The computing device described herein can store instructions that can be executed directly (such as machine code) or indirectly (such as scripts) by a processor. The computing device can also store data that can be retrieved, stored, or modified by one or more processors according to the instructions. Instructions can be stored as computing device code on a computing device-readable medium. In this regard, the terms "instruction" and "program" are used interchangeably herein. Instructions can also be stored in an object code format for direct processor processing, or in any other computing device language, including scripts or collections of independent source code modules that are interpreted on demand or pre-compiled. As an example, the programming language could be C#, C++, JAVA, or another computer programming language. Similarly, any component of the instructions or program can be implemented in a computer scripting language, such as JavaScript, PHP, ASP, or any other computer scripting language. Furthermore, any of these components can be implemented using a combination of computer programming languages ​​and computer scripting languages.

[0023] Example Method

[0024] Figure 3 An exemplary method 300 for training a new device in an ambient computing environment is described according to aspects of this disclosure.

[0025] In this regard, in step 302, the new device is added to the surrounding computing environment, which includes one or more other devices (e.g., Figure 1 The surrounding computing environment 100, and initiates communication between the new device and one or more other devices. For example, the surrounding computing environment may initially include Figure 1The device shown can be a new camera (e.g., the same as or similar to camera 102) installed in another location. Existing and new devices can be configured to discover each other in any suitable manner, such as using wireless protocols like WiFi, Bluetooth, ultrasound, etc.

[0026] In step 304, the new device and / or one or more other devices determine whether the new device is compatible with the model used by one or more other devices. This can be achieved in any suitable manner. For example, the new device and / or existing device can be configured to recognize similarities between each other. Thus, using the example discussed above where the new device is a new camera, the new camera or existing camera 102 can be configured to recognize that they are both cameras. Based on this, the new camera or existing camera 102 can be configured to determine that the model used by the existing camera 102 is compatible with the new camera. Similarly, if the new device is a camera and the existing device is a smart speaker, it can be determined that the smart speaker's model can still be compatible with the camera, as long as the camera is also equipped with a microphone and configured to perform a speech recognition task similar to that performed by the existing smart speaker. In this case, the smart speaker's model can be combined with another model configured to accept video input to create a model that can accept both audio and video input while still benefiting from the accumulated knowledge of the smart speaker.

[0027] In step 306, one or more other devices send information about the model to the new device. This information can be the model itself, a portion of the model, the type of the model, etc. Thus, continuing with the example where the new device is a new camera and the existing device is camera 102, the existing camera 102 can send a copy of its trained (e.g., initially trained and / or fine-tuned) model to the new camera to be used as the new camera's initial model. This trained model can reflect things that the existing camera 102 has learned over time, such as the sound of a user's voice, the identity of family members, etc. Similarly, in some aspects of this technology, the existing camera 102 can send a portion of its trained model (e.g., a set of initial layers) to generate an initial representation or classification to be shared with the existing camera 102 if the devices will collaboratively perform classification (as discussed further below). Furthermore, in some aspects of this technology, the existing camera 102 can inform the new camera of the type of model it is using, and the new camera can be configured to then (e.g., from a base model pre-stored on the new camera or from a remote processing system) select a copy of that same type of model.

[0028] In step 308, as the new device further trains its model, one or more other devices are selected to supervise the new device. As discussed with respect to step 306, the new device's model may be a copy of a model from one or more of the other devices, a copy of a model pre-stored on the new device, or a copy of a model obtained from a remote processing system. The set of one or more supervising devices may be (or include) the same devices that share information about the new device's model, or may be a separate set of one or more other devices. The selection of which devices will supervise the new device can be based on any suitable criteria, such as: how close they are to the new device; how similar their sensing capabilities are to the sensing capabilities the new device will perform; how many sensors they contain; how much processing power they have; how fine-tuning their own model is; and how their model is personalized (e.g., based on explicit user feedback). Therefore, using Figure 1 In an exemplary surrounding computing environment 100, if the new device is a camera, in some cases, the existing camera 102 can be selected to monitor the new camera based on its similar sensing capabilities. However, the existing camera 102 may not be selected to monitor the new camera in all cases, such as when the existing camera 102 is mounted far from the new camera, and it is unlikely to be exposed to the same field of view and sound as the new camera.

[0029] In step 310, one or more monitoring devices send one or more tags related to the detected event to the new device. In some aspects of this technology, the one or more monitoring devices can be configured to automatically transmit the one or more tags to the new device whenever an event is detected. Similarly, in some aspects of this technology, the new device can be configured to request tags from one or more monitoring devices whenever the new device detects an event. For example, if the new device detects the event first, it can send a request for any tags that the monitoring devices may have already applied to for a given time (or time range).

[0030] In some aspects of this technology, one or more tags may also be accompanied by data about the time (or time range) at which the event was detected. For example, if camera 102 is a monitoring device and identifies the homeowner based on their appearance and voice, camera 102 may be configured to send a tag identifying the homeowner to a new camera. Similarly, if a nearby smart speaker (e.g., speaker 104) is also a monitoring device and also detects the homeowner based on their voice, the smart speaker may also send a tag indicating this to a new camera.

[0031] In step 312, the new device further trains its model using one or more labels and data about information sensed by the new device. Thus, continuing the example illustrated in step 310, the new camera can create training examples by pairing a given sample of audio and / or video data recorded by the new camera with labels it receives from existing camera 102 and existing smart speaker 104, and can then modify one or more parameters of its model based on those labels during backpropagation. Similarly, in some aspects of this technique, the new camera can use its own model to process a given sample of audio and / or video data recorded by the new camera to generate a prediction based on that sample of audio and / or video data. The new camera can then use the labels from existing camera 102 and existing smart speaker 104, along with its own generated prediction, to generate one or more loss values. Those one or more loss values ​​can then be used during backpropagation to modify one or more parameters of the new camera model. In this respect, in the case where the new device receives multiple labels, it can combine the labels in any suitable manner. Therefore, in some aspects of this technology, the new device can combine tags in a weighted manner based on factors such as the proximity of each monitoring device to the new device, the strength of the signal from each monitoring device (e.g., the confidence level of the device's model regarding the prediction), and whether the sensors of the monitoring devices are the same as or different from those of the new device.

[0032] Figure 4 An exemplary method 400 for performing multi-device inference in an ambient computing environment is described according to aspects of this disclosure.

[0033] In this regard, in step 402, the new device is added to the surrounding computing environment, which includes one or more other devices (e.g., Figure 1 The surrounding computing environment 100, and initiates communication between the new device and one or more other devices. For example, the surrounding computing environment may initially include Figure 1 The device shown can be a new camera (e.g., the same as or similar to camera 102) installed in another location. Existing and new devices can be configured to discover each other in any suitable manner, such as using wireless protocols like WiFi, Bluetooth, ultrasound, etc. In some aspects of this technology, the new device and / or one or more other devices in the surrounding computing environment can be configured with restrictions on which devices they can communicate with. For example, a device can be configured to only allow communication with other devices associated with the same account as it.

[0034] In step 404, the new device is paired with one or more other devices. Devices can be configured to pair with each other based on any suitable criteria, such as whether the devices are of the same type (e.g., two cameras), have similar sensors (e.g., both the camera and the smart speaker have microphones), are located close to each other (e.g., both devices are in the same room), and / or perform similar functions (e.g., voice recognition, motion detection). Similarly, the pairing decision can be based, in whole or in part, on the availability of a model that can accept input from all devices if all devices are to be paired.

[0035] In step 406, a model is selected and implemented on the paired devices. In some aspects of this technology, each of the paired devices can implement a complete copy of the selected model. Similarly, in some aspects, one of the paired devices can be selected to host the complete model, and the other paired devices can store only a portion of the model (e.g., a certain number of initial layers of the model). In this case, the devices with partial copies of the model can be configured to process their own data using the partial model and send the output to the device hosting the complete model for further processing.

[0036] The selected model can be chosen, downloaded, or created by any of the paired devices. For example, one or more of the devices may have a library of predefined models for various expected pairings, from which models can be selected. Similarly, such a library of models may be available on a local or remote processing system (e.g., a local networked storage device, a cloud storage system, etc.), from which models can be selected and downloaded.

[0037] In some aspects of this technology, the model can be a model capable of handling input from each paired device without modification. For example, when a new camera is added to an existing surrounding computing environment that includes cameras and smart speakers, one of the devices can access a predefined model that handles input from two cameras (e.g., with audio and video feeds) and a single microphone (e.g., from the smart speaker).

[0038] Similarly, in some aspects of this technology, the model may be one that needs modification to handle input from each paired device. In this case, the selected model can be created by modifying or combining one or more models selected and / or downloaded from a local or remote model library. For example, in some aspects, the surrounding computing environment may initially include a camera and a smart speaker, and they can be operated using a model configured to accept input from one camera and one separate microphone. When an additional camera is added, if a predefined model is not available, one or more of the paired devices can be configured to modify the existing model using one or more initial layers that characterize and pool data from both cameras, and then pass the pooled camera data along with the microphone data from the smart speaker to a pre-existing layer of the original model.

[0039] Similarly, in some aspects of this technology, models can be created from a combination of two or more existing models. For example, if a new camera is added to the surrounding computing environment that initially includes a camera and a microphone, the device can create a new model by combining an existing model (configured to accept input from one camera and one other microphone) with a second model configured to accept input from a single camera and feeding their respective classifications into one or more final layers configured to output the final classification.

[0040] In step 408, an inference paradigm is selected to perform inference based on the input from the paired device. The inference paradigm can be any suitable paradigm for performing inference across multiple devices.

[0041] For example, in some aspects of this technology, each paired device can be configured to continuously capture information and transmit the information (or a representation or classification based on its partial processing) to other devices. In this case, each paired device can be configured to maintain and use a copy of its own selected model to classify any sensed events based on the information it senses and the information it receives from other paired devices.

[0042] Furthermore, in some aspects of this technology, one of the paired devices can be selected as the leader to run continuous classification and provide its classifications to the other devices. This paradigm can consume less power and processing cycles than existing paradigms. The leader device can be selected based on any suitable criteria, such as processing power, connectivity to a continuous power supply, etc. In this case, the leader can solicit information about an event from each of the other paired devices when an event is detected. The information provided by the other paired devices can include any suitable data about the event, such as data sensed by the device, its representation created by processing the sensed data using a partial copy of the selected model, individual device classifications based on the sensed data, etc. The leader device can then perform classification using the full version of the selected model based on its sensed information along with the information it receives from the other paired devices. The leader device can then share its final classifications with the other devices so that each device can perform its respective functions based on the final classifications.

[0043] Similarly, in some aspects of this technology, one of the paired devices can be selected as the leader to run continuous classification using the full model, but may not be configured to solicit information from other paired devices when an event is detected. Instead, each paired device can be configured to continuously capture information, perform single-device classification (e.g., using a partial copy of the selected model, or configured to receive input from a single device), and then share its classification with the leader. As described above, the leader will then perform classification using the full version of the selected model based on the information it senses and its single-device classifications received from the other paired devices. As described above, the leader will then share its final classification with the other devices.

[0044] In step 410, upon detecting an event, inference is performed based on data sensed by the paired devices, according to a selected inference paradigm. As described above, this could involve performing inference on each of the paired devices or entirely on the selected leader device, based on information or classifications provided by other paired devices.

[0045] Apart from Figure 4 Beyond the exemplary inference steps described herein, paired devices can be configured to train (e.g., initial training or fine-tuning) a selected model based on any suitable criteria. For example, a paired device (or a selected leader device) can be configured to train the selected model based on user feedback signals (e.g., explicit input from a person whose identification is recorded in video or audio samples). Similarly, a paired device (or a selected leader device) can be configured to train a model such that layers combining information from other paired devices are modified. For example, these combining layers can be selected to achieve an optimal balance between accuracy and one or more other criteria, such as power consumption or communication link utilization.

[0046] although Figure 4 This paper describes methods for selecting appropriate models and multi-device inference paradigms for a given surrounding computing environment; however, not all surrounding computing environments are static. In this respect, many surrounding computing environments can include one or more mobile devices that can thus momentarily come and go from the surrounding computing environment. For example, laptops, tablets, smartphones, and smartwatches can be configured to connect to a network, interface with other devices on the network (e.g., running applications that provide access to and / or control of cameras, smart speakers, security systems), and / or provide services similar to those provided by other devices in the surrounding computing environment. However, these devices can also periodically appear and disappear from the network and the surrounding computing environment as users carry them in and out of the network's coverage area. Similarly, devices can be added to any surrounding computing environment, and other devices can eventually be permanently removed, such as when a device fails or is replaced. In many cases, changes to the composition of the surrounding computing environment may adversely affect the quality of inference provided by any initially selected model and / or may compromise the selection of a new model.

[0047] in this regard, Figure 5 An exemplary method 500 is described, based on aspects of this disclosure, for training a model that is robust to adding or removing devices (and their associated sensors) from the surrounding computing environment.

[0048] In step 502, a set of devices is added to and paired with the surrounding computing environment, the model is implemented on the paired devices, and multi-device inference paradigms are identified. This can occur in any suitable manner, including according to the above regarding... Figure 4 The options described in steps 402-408. In this regard, as discussed above with respect to step 406, the selected model may also be a model created by modifying an existing model (e.g., by adding layers so that multiple inputs can be pooled before being fed into an existing model) or by combining two or more existing models (e.g., by feeding inputs into two or more existing models and combining their outputs using one or more classification layers).

[0049] In step 504, the model is trained (e.g., initial training and / or fine-tuning) based on one or more events during a given time period, where each event is processed as described in steps 506-512. This training process can be performed by any or all of the paired devices. Thus, in some aspects of this technology, one of the paired devices can be selected as the leader for performing steps 506-512. Similarly, in some aspects of this technology, each of the paired devices can perform steps 506-512 independently. For example, in the paradigm where each device maintains a complete or partial copy of the model, each paired device can train its own copy of the model based on steps 506-512.

[0050] The given time period for training the model can be any suitable length of time, such as an hour, a day, a week, etc. Similarly, the given time period can be a non-contiguous period, such as an hour in the morning, afternoon, and evening of one or more days. Furthermore, the given time period does not have to be measured in terms of time, but can be any length of time required for the model to experience a predetermined number of events or to obtain a predetermined loss on a set of events. In some aspects of this technique, the given time period can be limited to the time all paired devices are present in the surrounding computing environment. Additionally, steps 506-512 can be used to readjust the model after it has been initially trained and / or fine-tuned. For example, in some aspects of this technique, the model can be readjusted periodically (e.g., every Monday, once a month), upon user request, when a device is relocated or permanently removed, etc.

[0051] Furthermore, while models can be trained based on events that occur in real time during a training period, they can also be trained on a pre-existing training dataset (e.g., a set of events previously recorded by the device or another set of similar devices, a set of supervised training examples with human-generated labels, etc.). In the case where the model is trained on a pre-existing training dataset, each training example can constitute an event, and a given time period can be the length of time required for the model to run on that pre-existing training dataset.

[0052] In step 506, for each event, a model is used according to an inference paradigm to generate a first prediction based on a first input set, which includes at least one input from each device currently present in the surrounding computing environment. For example, assuming the surrounding computing environment includes a camera, a smart speaker, and a mobile phone, a model that performs inference based on audio and video input from the camera, audio input from the microphone of the smart speaker, and audio input from the microphone of the smartphone can be selected. In this case, the first prediction can be based on each of these inputs from each of these devices. The first prediction can be any intermediate or final prediction generated by the model. Therefore, the first prediction can be the final vector or final classification output by the model, or the model can use it as an intermediate vector or intermediate classification as the basis for the final prediction.

[0053] In step 508, the model is used according to the inference paradigm to generate a second prediction based on a second input set, which includes a modified set of the first input set. In this respect, the second input set can be identical to the first input set, except that at least one input from at least one device currently present in the surrounding computing environment can be omitted or replaced with some other data. Where the inference paradigm involves a selected leader device, where each other paired device sends a representation or classification to the leader device, the second input set can therefore result in one or more such representations or classifications being replaced, omitted, or zeroed out when the leader device performs inference using the full model. Here, the second prediction can also be any intermediate or final prediction generated by the model. Thus, the second prediction can be the final vector or final classification output by the model, or an intermediate vector or intermediate classification that the model can use as the basis for the final prediction.

[0054] Therefore, using the same example discussed in step 506, the second prediction can be based on inputs from the camera and smart speaker (which are included in the first input set), where inputs from the smartphone are omitted or zeroed out.

[0055] Similarly, the second prediction could be based on input from the camera and smart speaker, where the input from the smartphone is replaced in some way.

[0056] For example, in some aspects of this technology, an input (e.g., audio data collected by a microphone in a smartphone) can be replaced with the same copy, downsampled, or upsampled version of a similar input (e.g., audio data collected by a microphone in a smart speaker in the surrounding computing environment). In this respect, the replacement input can be obtained from a device not included in the first input set. For example, the first input set may include inputs A, B, and C, and can be modified by replacing input C with a similar input D, such that the second input set includes inputs A, B, and D.

[0057] Similarly, in some aspects of this technology, an input (e.g., audio data collected by a microphone in a smartphone) can be replaced with a synthesized input based on multiple similar inputs (e.g., a weighted average of audio data collected by microphones of multiple smart speakers, or a weighted average of audio data collected by microphones of smart speakers and additional audio data collected by microphones of a camera). Here, the replacement signal can also be based on the same copy, downsampled, or upsampled from multiple similar inputs.

[0058] Furthermore, in some aspects of this technology, input from a given device can be replaced with data that occurs slightly before or after the event (e.g., from a predetermined number of seconds before or after the event). This can be useful in situations where the device's signal is only temporarily lost, but previous and subsequent events still provide clues about what happened during the event.

[0059] Determining which input to omit or replace can be based on any suitable criterion. Therefore, in some aspects of this technique, the input to omit or replace can be randomly selected. Similarly, in some aspects of this technique, the input to omit or replace can be selected based on an assumed or calculated dropout probability. Furthermore, in some aspects of this technique, the input to omit or replace can be modified by (e.g., multiplying by) an assumed or calculated dropout probability.

[0060] For example, using the same exemplary ambient computing environment discussed in step 506, the inputs associated with the camera, smart speaker, and smartphone can each be assigned an associated loss probability based on an assumption or calculation about the frequency with which the associated device will be present in the ambient computing environment. Thus, based on the fact that the camera is the type that must be physically mounted to a wall, a preset loss probability of 0.1 can be assigned to the input associated with the camera; based on the fact that the smart speaker is battery-powered and can be moved to areas where network coverage cannot be guaranteed, a preset loss probability of 0.3 can be assigned to the audio input associated with the smart speaker; and the audio input associated with the smartphone can be assigned a preset loss probability of 0.7 based on the expected extended period of time the phone will be carried outside the local network.

[0061] In another example, the probability of loss can be based on assumptions or calculations about how much power each device consumes. Therefore, in some aspects of this technique, under the assumption that a power-saving paradigm might prioritize devices consuming less power over those using more power, the probability of loss for each device can be inversely proportional to how much power each device consumes.

[0062] In another example, the probability of loss can be calculated based on historical data about how frequently each device was present in the surrounding computing environment during a past time period (e.g., based on maintained logs) or how much power each device consumed during a past time period.

[0063] In another example, the drop probability can be preset to a set value regardless of the type of device. Therefore, in this case, the inputs associated with the camera, smart speaker, and smartphone can each initially receive drop probabilities of 0, 0.5, 1, etc.

[0064] Furthermore, in some aspects of this technique, the loss probabilities can be updated over time. In this regard, although initial loss probabilities can be assigned as described above, they can be modified based on how frequently each device actually arrives and leaves the surrounding computing environment. In this way, if the model is periodically readjusted according to steps 506-512, the inputs can be omitted based on loss probabilities that more closely reflect reality.

[0065] In step 510, a loss value is generated based on the first and second predictions. This loss value can be generated according to any suitable loss function (e.g., classification loss, L1 loss, L2 loss, cross-entropy loss, etc.). In some aspects of this technique, the loss value can be an aggregated loss value (e.g., an average loss value) generated and updated over a certain number of events.

[0066] In step 512, one or more parameters of the model are modified based on the loss value generated for at least one of the events. This backpropagation step can be performed at any suitable interval. Thus, in some aspects of this technique, the parameters of the model can be updated after each event. Similarly, in some aspects of this technique, backpropagation can be performed after a certain number of events. For example, after a selected number of events have been processed, one or more parameters can be modified based on each of the loss values ​​generated for those events. Likewise, in some aspects of this technique, each event can be processed to generate an aggregate loss value, as discussed above with respect to step 510, and one or more parameters of the model can be modified based on the aggregate loss value.

[0067] Figure 6 Another exemplary method 600 for training a model to be robust to adding or removing devices from the surrounding computing environment is described according to aspects of this disclosure.

[0068] In step 602, a first set of devices is added to and paired with the surrounding computing environment; a first model is selected and implemented on the first set of paired devices; and a first multi-device inference paradigm is identified. This can occur in any suitable manner, including according to the above regarding... Figure 4The options described in steps 402-408.

[0069] To help illustrate method 600, it will be assumed that the surrounding computing environment initially includes a camera, a smart speaker, and a mobile phone, and that the first model is configured to perform inference based on audio and video input from the camera, audio input from the microphone of the smart speaker, and audio input from the microphone of the smartphone.

[0070] In step 604, one or more devices are added or removed from the surrounding computing environment, resulting in a second set of paired devices existing in the surrounding computing environment. Therefore, continuing the previous example, if the smartphone is removed from the surrounding computing environment, the second set of paired devices will include a camera and a smart speaker.

[0071] In step 606, a second model is selected and implemented on the second set of paired devices. This second model can also be selected, downloaded, or created from any of the paired devices based on any suitable standard, as described above. Figure 4 As discussed in step 406. Therefore, continuing with the previous example, once the smartphone is removed from the surrounding computing environment, one or more of the remaining devices can select a second model that is configured to perform inference solely based on audio and video input from the camera and audio input from the microphone of the smart speaker.

[0072] In some aspects of this technology, the second model can be a model that has already been trained (initial training and / or fine-tuning) together with the first model. In this case, one or more of the first group of devices can be configured to recognize one or more models other than the first model, which will be configured to accept inputs from predictable modifications to the surrounding computing environment. Thus, continuing the previous example, when selecting and implementing the first model, at least one device can be configured to similarly load one or more additional models. For example, the device can load an additional model that will only accept input from a camera, another model that will only accept input from a smart speaker, another model that will accept input from both a camera and a smart speaker, and yet another model that will accept input from both a camera and a smartphone. The device can then be configured to train (e.g., initial training and / or fine-tuning) the additional models in parallel with the training of the first model, such that the trained additional models can be used for implementation when one or more devices leave the surrounding computing environment. Similarly, the device can also be configured to load and train one or more additional models representing possible additions to the first group of devices (e.g., models configured to accept inputs from two cameras, a smart speaker, and a smartphone).

[0073] Furthermore, in this respect, after switching to the second model, one or more devices in the second group can be configured to retain a copy of the first model so that if the composition of the surrounding computing environment returns to its original state (e.g., using the previous example, if the smartphone returns to the surrounding computing environment), they can switch back to the first model. Similarly, in some aspects of this technology, one or more devices can be configured to upload a copy of the first model to a local or remote processing system so that if the composition of the surrounding computing environment returns to its original state, the first model can be downloaded again. Likewise, in terms of the extent to which training continues while using the second model, the first model (and any other additional models) can also continue to be trained in parallel with the second model.

[0074] Unless otherwise stated, the foregoing alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the foregoing description of the exemplary systems and methods should be done illustratively rather than by limiting the subject matter defined by the claims. Furthermore, the provision of examples described herein and terms such as “e.g.,” “comprising,” “including,” etc., should not be construed as limiting the subject matter of the claims to the specific examples; rather, these examples are intended only to illustrate some of the many possible embodiments. Moreover, the same reference numerals in different figures may identify the same or similar elements.

Claims

1. A computer-implemented method for training a model to perform inference in a surrounding computing environment with multiple devices, the method comprising: One or more processors of the processing system identify multi-device inference paradigms to be used for performing inference using the model, based on inputs from one or more of the plurality of devices. For each given event among multiple events: The one or more processors use the model to generate a first prediction based on a first input set, which includes at least one input from each of the plurality of devices. The one or more processors use the model to generate a second prediction based on the multi-device inference paradigm using a second input set, the second input set including a modified copy of the first input set, wherein the modified copy of the first input set differs from the first input set in that at least one input from at least one of the plurality of devices is modified; as well as The loss value is generated by the one or more processors based on the first prediction and the second prediction; as well as The one or more processors modify one or more parameters of the model based on loss values ​​generated for at least one given event.

2. The method according to claim 1, wherein, Each of the plurality of events occurs during a given time period, and the first input set for each given event is based on data sensed by the plurality of devices during the given time period.

3. The method according to claim 2, wherein, The given time period is a non-contiguous time period consisting of two or more separate time periods.

4. The method according to claim 1, wherein, The one or more processors are configured to generate each first prediction, each second prediction, and each loss value during a given time period, and the first input set for each given event is based on data sensed prior to the given time period.

5. The method according to claim 1, wherein, For each of the plurality of events, the modified copy of the first input set includes one or more selected inputs from the first input set and omits one or more other inputs from the first input set.

6. The method according to claim 5, further comprising: For each given event among multiple events: The one or more processors determine, based on a first value, to omit the one or more other inputs.

7. The method according to claim 6, further comprising: The first value is modified by the one or more processors based on the frequency with which the device associated with the first input set has existed in the surrounding computing environment over a period of time.

8. The method according to claim 6, further comprising: The one or more processors modify the first value based on how much power the device associated with the first input set has consumed during a time period.

9. The method according to claim 1, wherein, For each of the plurality of events, in the modified copy of the first input set, at least a first input of the first input set is replaced by a copy of a second input of the first input set.

10. The method according to claim 1, wherein, For each of the plurality of events, in a modified copy of the first input set, at least a first input of the first input set is replaced by an upsampled version of a second input of the first input set.

11. The method according to claim 1, wherein, For each of the plurality of events, in a modified copy of the first input set, at least a first input of the first input set is replaced by a downsampled version of a second input of the first input set.

12. The method according to claim 1, wherein, For a given event among the plurality of events, in a modified copy of the first input set, at least a first input of the first input set is replaced by a composite input based on two or more other inputs of the first input set.

13. The method according to claim 12, wherein, The synthetic input is the average of the two or more other inputs in the first input set.

14. A system for training a model to perform inference in a surrounding computing environment having multiple devices, the system comprising: Memory; as well as One or more processors coupled to the memory and configured to: Identify multi-device inference paradigms to be used for inference using the model, based on inputs from one or more of the plurality of devices; For each given event among multiple events: Based on a first input set, the model is used to generate a first prediction according to the multi-device inference paradigm, the first input set including at least one input from each of the plurality of devices; Based on a second input set, the model is used to generate a second prediction according to the multi-device inference paradigm, the second input set including a modified copy of the first input set, wherein the modified copy of the first input set differs from the first input set in that at least one input from at least one of the plurality of devices is modified; as well as A loss value is generated based on the first prediction and the second prediction; as well as Modify one or more parameters of the model based on the loss value generated for at least one given event.

15. The system of claim 14, wherein for each given event of the plurality of events, the modified copy of the first input set includes one or more selected inputs from the first input set and omits one or more other inputs from the first input set.

16. The system according to claim 15, wherein, The one or more processors are also configured to, for each given event among a plurality of events: The first value determines to omit one or more other inputs.

17. The system according to claim 16, wherein, The one or more processors are further configured to: The first value is modified based on the frequency with which the device associated with the first input set has been present in the surrounding computing environment over a period of time.

18. The system according to claim 16, wherein, The one or more processors are further configured to: The first value is modified based on how much power the devices associated with the first input set have consumed during a certain time period.

19. The system of claim 14, wherein for each given event of the plurality of events, in a modified copy of the first input set, at least a first input of the first input set is replaced by a copy, upsampled, or downsampled of a second input of the first input set.

20. The system of claim 14, wherein for each given event of the plurality of events, in a modified copy of the first input set, at least a first input of the first input set is replaced by a composite input based on two or more other inputs of the first input set.

Images