A self-supervised collaborative approach to machine learning with models deployed on edge devices

Abstract

Presented herein is an approach to developing and deploying machine learning models that addresses the shortcomings of conventional approaches. One goal of the approach described herein is to reduce or eliminate the need for manual classification during the development process. To achieve this, a monitoring system may implement self-supervised learning and knowledge distillation that relies on collaboration between its edge devices and server systems. In summary, self-supervised learning and knowledge distillation ensure that models deployed to edge devices are immediately trained and updated, if necessary, to improve the quality of inferences without any human participation.

Description

[Technical field]

[0001] Various aspects relate to monitoring systems and related techniques for developing and training software-implemented models in edge devices of those monitoring systems.

[0002] Cross-references to related applications This application claims priority to U.S. Application No. 63 / 148,852, entitled “SELF-SUPERVISED COLLABORATIVE APPROACH TO MACHINE LEARNING BY MODELS DEPLOYED ON EDGE DEVICES,” filed February 12, 2021, and incorporated herein by reference for all purposes. [Background technology]

[0003] The word "surveillance" refers to monitoring changes in behavior, activities, and other information for the purpose of protecting people and items in a given environment. Surveillance generally requires that a given environment be monitored using electronic devices such as digital cameras, light sources, locks, motion detectors, etc. Collectively, these electronic devices are referred to as the "edge devices" of a "surveillance system" or "security system."

[0004] One concept that is becoming more common in surveillance systems is edge intelligence (also called "edge analytics"). Edge intelligence refers to the ability of electronic devices included in a surveillance system to process information and make decisions before sending that information elsewhere. As an example, a digital camera (or simply "camera") may be responsible for finding objects contained in digital images (or simply "images") before those images are sent to a destination. The destination may be a computer server system responsible for further analysis of the images.

[0005] Being able to perform tasks locally becomes increasingly important as the scale of information generated by electronic devices continues to grow. For example, assume that a surveillance system designed to monitor a residential environment includes several cameras. Each of these cameras may be capable of generating high-definition images several megapixels (MP) in size. While these high-definition images provide better insight into the residential environment, their larger size makes them difficult to handle due to large bandwidth and storage requirements. For this reason, it may be beneficial to determine which images, if any, generated by the target camera over a defined time frame should be forwarded to other components of the surveillance system for further analysis. Edge intelligence may be used to ensure that only those images deemed important are forwarded by these cameras. This is based on the premise that the surveillance system is more interested in, for example, the minutes during which an unknown individual approaches the door of the residence than the hours during which no one is present at the front door of the residence. [Brief description of the drawings]

[0006] [Figure 1] 1 is a high-level diagram of a centralized monitoring system that includes multiple edge devices disposed in a monitored environment. [Diagram 2] 1 is an illustration of how the environmental context of data generated by edge devices changes over time, and therefore the classification of that information also changes over time. [Diagram 3] FIG. 1 includes an overview for an example of an algorithm designed according to the collaborative framework. [Figure 4] FIG. 1 includes a high-level overview of a monitoring system for self-supervised customization performed for models utilized by edge devices. [Diagram 5] FIG. 1 includes a high-level flowchart showing how the average confidence in the output produced by a local model trained for object detection is calculated in an ongoing manner to determine when the local model is updated. [Figure 6] 1 is a high level illustration of communications involving an edge device responsible for applying local models to data generated during monitoring to identify events of interest. [Figure 7] FIG. 1 includes a flow diagram of a process for creating a local model adapted to the environment in which an edge device is deployed. [Figure 8] FIG. 1 includes a flow diagram of a process for facilitating adaptation of a local model by edge devices deployed in a monitored environment. [Figure 9] FIG. 1 illustrates an example of a processing system in which at least some of the processes described herein may be implemented.

[0007] Various features of the technology described herein will become apparent to those skilled in the art from review of the detailed description in conjunction with the drawings, in which embodiments are illustrated by way of example and not limitation. While the drawings depict various embodiments for illustrative purposes, those skilled in the art will recognize that alternative embodiments may be utilized without departing from the principles of the technology. Thus, while specific embodiments are shown in the drawings, the technology is susceptible to various modifications. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Edge intelligence may require machine learning models (or simply "models") to be applied by at least some of the electronic devices included in the surveillance system. These electronic devices may be referred to as "edge devices" because they represent the points where data enters the surveillance system. Traditional approaches to developing models for edge devices require that data be collected, labeled, and then used for training. Furthermore, the trained models may be tuned to improve their ability to produce appropriate outputs prior to deployment of the edge devices. These outputs may be referred to as "predictions" or "inferences." The trained models may be tuned to ensure that characteristics of the edge devices or their surrounding environment are adequately accounted for.

[0009] Labeling the data used for training is usually done in a planned and regular manner, which tends to inhibit timely model deployment. In other words, labeling tends to be the main bottleneck of the conventional approaches described above. A large amount of labeled data requires different preparations for the deployment of edge devices to train the model. However, the amount of human involvement required makes labeling not only expensive but also time consuming.

[0010] Furthermore, any subsequent model updates require the conventional approach to be fully completed again. For example, assume that a first version of a model trained for object recognition is immediately deployed to a camera. Thereafter, it may be determined (e.g., by a developer or manufacturer) that a model that is better able to recognize objects in images generated by the camera would be desirable. As an example, the first version of the model may have been trained using images acquired under daytime lighting conditions and may therefore experience poor performance when inspecting images acquired under nighttime lighting conditions. In such a scenario, it would be desirable to further train the model using images acquired under nighttime lighting conditions to produce a second version of the model that is better able to recognize objects under nighttime lighting conditions. The second version of the model is then deployed to the camera to replace the first version of the model. This approach is quite cumbersome and makes it difficult for the model to evolve for subsequent deployments.

[0011] Introduced herein is an approach to developing and then deploying models that addresses the shortcomings of conventional methods mentioned above. One objective of the approach described herein is to reduce or eliminate the need for manual labeling during the development process. To accomplish this, a surveillance system may implement self-supervised learning that relies on collaboration between its edge devices and a server system (also called a "cloud system"). Self-supervised learning ensures that the models deployed to those edge devices are trained immediately and then updated as needed without any human intervention.

[0012] At least some edge devices of the monitoring system may implement models for predictive or inferential intent. These models may be modified based on the context, content, or data provided as input to improve performance with respect to localized usage. As shown in FIG. 4, there may be three different models involved in the collaborative framework described herein, namely: A global model (ω) trained to run on various edge devices using the supervised training data available on the server system. As discussed further below, the global model can be distributed to various edge devices where it is intended to be used for inference. Edge model TIFF2024507765000002.tif7150 represents a localized version of the global model. The edge models (called "local models" or "student models") have a similar structure and parameter sizes as the global model. However, using the collaborative framework described herein, the parameters of each edge model can be updated and localized based on data generated from the corresponding edge device. Cloud-based models (Ω) (or simply “cloud models”), which usually have a more complex structure and larger parameter sizes than the global and edge models. Interfaces with cloud models (also called “supervised models”) are created only on server systems due to their high computational requirements. Cloud models can be used in knowledge distillation to distill knowledge for edge models, since they have a richer feature representation than the global and edge models.

[0013] The edge devices and server system represent a distributed surveillance system capable of collecting and augmenting data, as well as facilitating self-supervised training using the data. All these actions can be performed automatically to enhance the quality of the models implemented by the edge devices. At a high level, the above collaborative approach ensures that each edge device is capable of developing customized models specifically adapted for the data it generates, acquires, or otherwise obtains. In some embodiments, training is performed continuously so that the local models are permanently adapted by the edge devices in response to environmental variations in the data used by those local models. In other words, each edge device that is responsible for changes in the data provided as input to the corresponding model is configured to update the corresponding model running there. However, it should be noted that this update does not necessarily have to be realized by the edge device. As will be further discussed below, the update can be realized by the edge device, the server system, or an intermediate device such as a mobile phone, a tablet computer, or a base station of the surveillance system.

[0014] In summary, there are several core aspects of the approach described herein: First, an end-to-end framework for independently adapting edge devices to deploy models to handle data generated by those edge devices according to an edge-cloud collaborative scheme; Second, a training process designed to operate without any human intervention or oversight; and Third, the training process is continuously performed so that data classification on each edge device can immediately adapt to changes in the data.

[0015] The approaches described herein may be particularly suited to models deployed on edge devices such as cameras, light sources, locks, sensors, and the like. However, it should be noted that while embodiments may be described with reference to particular models and edge devices, the techniques may be similarly applied to other models and edge devices. For example, for purposes of explanation, an embodiment may be described in the context of a model designed to recognize instances of objects contained in images generated by a camera. Such models may be referred to as "object recognition models." However, those skilled in the art will recognize that the techniques may be similarly applied to other types of models and other types of edge devices.

[0016] Further, the embodiments may be described in computer executable instructions for illustrative purposes. Aspects of the technology may be implemented by hardware, firmware, or software. For example, an edge device may be configured to generate data representative of a surrounding environment and provide the data as input to a model. The edge device may then determine an appropriate course of action based on the output produced by the model. The edge device may also inspect the data on a periodic or continuous basis. If the edge device determines that there is a meaningful change in the context or content of the data, then the edge device initiates a process to update the model. The term "meaningful change" may refer to a change in the data that affects the usefulness of the output that produced the model. As an example, if the furniture in a residence under surveillance by a camera changes in color or form, the context of the image generated by the camera changes. This change in context may cause the object recognition model present in the camera to endure a degradation in performance. Thus, it may be desirable to adapt the object recognition model to the change in context to ensure that performance remains high. As another example, due to seasonal weather changes, it may be desirable to periodically update object recognition models utilized by cameras responsible for monitoring an exterior environment (eg, a backyard).

[0017] (Terminology) References in this description to "one embodiment" or "some embodiments" mean that the described feature, function, structure, or characteristic is included in at least one embodiment. Appearances of such phrases do not necessarily refer to the same embodiment, nor do they indicate mutually exclusive alternative embodiments.

[0018] Unless the context expressly requires otherwise, the words "comprises," "comprising," and "consisting of" are to be interpreted in an inclusive sense (i.e., "including but not limited to") and not in an exclusive or exhaustive sense. The word "based on" is also to be interpreted in an inclusive sense. Thus, unless otherwise indicated, the word "based on" shall mean "based at least in part on."

[0019] The terms "connected," "coupled," and variations thereof are intended to include any connection or coupling between objects, either direct or indirect. The connection or coupling may be physical, logical, or a combination thereof. For example, objects may be electrically or communicatively coupled to each other despite not sharing a physical connection.

[0020] The term "module" may be used broadly to refer to software, firmware, or hardware. A module is generally a functional entity that produces one or more outputs based on one or more inputs. A computer program may contain one or more modules. Thus, a computer program may contain multiple modules responsible for completing different tasks or a single module responsible for completing all tasks.

[0021] The word "or", when used in connection with a list of multiple items, is intended to cover all of the following interpretations: any of the items in the list, all of the items in the list, and any combination of the items in the list.

[0022] The order in which steps are performed in any of the processes described herein is exemplary. However, steps can be performed in various orders and combinations, unless physically possible. For example, steps can be added to or removed from the processes described herein. Similarly, steps can be interchanged or reordered. Accordingly, the description of any process is intended to be non-limiting.

[0023] (Overview of the surveillance system) 1 includes a high-level illustration of a centralized surveillance system 100 that includes various edge devices 102a-n deployed into an environment 104 for surveillance. While the edge devices 102a-n in FIG. 1 are cameras, other types of edge devices may be deployed in addition to or instead of cameras into the environment 104. Meanwhile, the environment 104 may be, for example, a home or a business.

[0024] In some embodiments, these edge devices 102a-n can communicate directly with a server system 106 comprising one or more computer servers (or simply "servers") via a network 110a. In other embodiments, these edge devices 102a-n can communicate indirectly with the server system 106 via an intermediary device 108. The intermediary device 108 can be connected to the edge devices 102a-n and the server system 106 via respective networks 110b-c. The networks ac can be personal area networks (PANs), local area networks (LANs), wide area networks (WANs), metropolitan networks (MANs), cellular networks, or the Internet. For example, the edge devices 102a-n can communicate with the intermediary device via Bluetooth, Near Field Communication (NFC), or other short-range communication protocols, and the edge devices 102a-n can communicate with the server system 108 via the Internet.

[0025] Generally, computer programs executing on the intermediary device 108 may be supported by the server system 106, thereby facilitating communication with the server system 106. The intermediary device 108 may be, for example, a mobile phone, a tablet computer, or a base station. Thus, the intermediary device 108 may remain in the environment 104 at all times, or the intermediary device 108 may periodically enter the environment 104.

[0026] In a centralized monitoring system, a global model is created and then trained by the server system 106 in the monitoring approach until an acceptable performance is achieved on the test data set. The global model is then deployed to the edge devices 102a-n for inference. As described above, this process involves collecting and then annotating and labeling data, training and then tuning a global model, and deploying the global model to various edge devices. The global model created by this process is deployed to all edge devices of similar type for inference. As an example, the object recognition model trained and tuned by the server system 106 may be realized by all cameras deployed in the environment of interest 104. However, global models are primarily unsuitable because data classification may change dramatically from one edge device to another or even over time for a single edge device. Simply put, the global model cannot capture the nuances of the different locations where the edge devices 102-n are deployed. Thus, performance may be inconsistent and may actually degrade over time across the edge devices 102a-n. Moreover, annotating the data generated by the edge devices 102a-n to train each version of the global model is simply not practical because it is laborious and difficult, if not impossible to achieve in a timely manner.

[0027] The collaborative approach described herein attempts to address these issues by having the edge devices 102a-n adjust the global model based on their respective data classifications. Thus, edge devices can have their own customized models adapted based on their own data. As discussed further below, this can be done in a self-supervised manner to eliminate the need for any human intervention.

[0028] FIG. 2 illustrates how the environmental context of data generated by an edge device may change over time, and therefore the data classification may also change over time. For this reason, it may be desirable to update the model used for the edge model to adapt to these local changes in data classification. In FIG. 2, an edge device (here, a camera) is deployed with a fixed view of the monitored environment. The context of this view (e.g., a camera) may vary dramatically over time, as may the objects of interest. This variation in the classification of the data (e.g., images) generated by the edge device may greatly affect the performance of the model used by the edge device for the data. Thus, it is important to update the model to ensure that it is appropriate for the changing environmental context (and therefore the content of the data).

[0029] One of the main changes to achieve this is that the data generated by the edge devices does not have ground truth labels (also called "ground truth annotations"). Furthermore, the data cannot be used in a supervised manner to update the model using known techniques such as fine tuning. To address this change and to avoid the requirement of any human intervention, self-supervised learning can be combined into a knowledge distillation method to attempt to learn the representation of the data at each edge device while matching the performance against a more powerful model. As discussed further below, this approach relies on a comparison of the output produced by a local model utilized at the edge device and a global model utilized at the server system to which the edge device is communicatively connected. To achieve this, some pseudo-tasks (also called "auxiliary tasks") related to performing supervision based on unlabeled data may need to be defined and used to minimize the loss of certain tasks related to the main goal. The main goal of the model depends on the type of edge device. For example, the main goal of a model deployed by a camera may be object detection. While the goal of the pseudo-tasks is not particularly important, the performance of those tasks helps improve the representation of the data classification at each edge device for the main goal.

[0030] In summary, the collaborative approach described herein was developed to address three problems. First, shifts in the classification of data generated by edge devices that can lead to degradation of the performance of models utilized on edge devices. Second, the costs and delays inherent in human labeling of data used to train models deployed to edge devices. Third, inaccurate feedback from humans labeling the data that inevitably degrades model performance even further.

[0031] (Overview of a collaborative approach to model fitting) There are two main goals of the approach described here. The first (and primary) goal is to have a customized model adapted independently to the edge devices of the surveillance system. The second goal is to achieve the first goal in a self-supervised manner combined with a knowledge distillation method in collaboration with the surveillance system, which includes local components (e.g., edge devices) and remote components (e.g., server systems). Each goal is explored to some extent by different components.

[0032] For the first goal, the most closely related work focuses on personalization in the context of federated learning. In federated learning, the goal is to learn a global model using data distributed to edge devices without direct access to the data generated by those edge devices. In such a setting, edge devices maintain control of their own data but can update the global model provided by the server system using their own data. The goal of personalization in the context of federated learning is to improve the generalization ability of each edge device's local model based on its own test data. However, almost all these approaches to personalization in the context of federated learning rely on (i) labeled data and (ii) updating the local model in a supervised manner. Furthermore, updates to each local model are kept at the corresponding edge device and are not relayed and transported to other parts of the supervised system.

[0033] For the second goal, some components are investigated how to use test data or unlabeled data to improve the performance of the model in completing the inference task in a self-supervised manner. Primarily, these efforts attempt to achieve better results during inference with unclassified data that is not used while training the model. These efforts usually involve the creation of auxiliary tasks to provide other forms of supervision. A loss function may be defined for the auxiliary task, and the model may be updated based on the loss function. For example, rotating an image produced by a camera and adding a classification layer on top of the feature representation produced by the model as output to further classify the angle of rotation is a simple auxiliary task that introduces a form of supervision. As mentioned above, the ultimate goal of these efforts is to improve the representation ability of the model for out-of-distribution samples, and it is not necessary to directly improve the primary task, as in the approach described herein.

[0034] Thus, there have been efforts to achieve the above goals, however the approach described herein is advantageous in several aspects.

[0035] First, knowledge distillation is somewhat widely used for model compression in classification tasks. However, there are some efforts to apply knowledge distillation to other tasks, such as object recognition and object detection. Here, knowledge distillation is combined with self-supervised learning techniques to improve the expressiveness of a student model with the assistance of a teacher model. Note that while the framework is described in the context of object recognition for illustrative purposes, the approach can be realized for other tasks.

[0036] Second, while personalized approaches for self-supervised learning have been considered in the context of some tasks, the approach described herein exploits self-supervised learning in a novel way. At a high level, the approach is designed to ensure that end-to-end development (e.g., creation, training, and tuning) can be accomplished without any human intervention.

[0037] Third, the performance of the model utilized by the edge device can be easily measured, allowing degradation of performance to be immediately detected. When the performance of the model drops below a threshold for a certain amount of time (e.g., hours or days), the model can be automatically adapted to ensure that it adapts appropriately to the data. This adaptation procedure can be run indefinitely, so that the local model is consistently adapted to the classification transformations of the data generated by the corresponding edge device.

[0038] Fourth, collaboration between the student model deployed on the edge device and the teacher model deployed on the server system ensures that high performance can be maintained. In the approach described herein, the teacher model can serve as a reference that can be used to enhance or improve the expressive power of the student model through knowledge distillation and self-monitoring.

[0039] (A. Introduction of a Joint Framework) An edge device can provide data to generate a model to produce an output (also called "prediction" or "inference") related to a task. The task depends on the characteristics of the edge device itself. For example, if the edge device is a camera, then the task may be to detect objects in images generated by the camera. To do this, the camera may use a model trained to detect those objects and localize each detected object using a bounding box. To train the model, images of those objects in different contexts may be given to the model as training data. However, no matter how diverse the training data is, it cannot capture all the classifications of images that may be generated from the camera that is used with the model. Thus, the performance of the model will degrade for cameras that generate images that are disparate in content terms from the training data.

[0040] Introduced here is a collaborative framework developed to address this problem. Rather than simply relying on improved training and tuning before deployment, the collaborative framework adapts the model to the next deployment based on data generated by the corresponding edge device. For example, assume that a global model is trained by a server system for object detection. The global model is then provided to cameras deployed throughout the environment of interest. Using the collaborative framework described herein, each local version of the global model can be adapted to account for the content of the image generated by the corresponding camera.

[0041] To update the local version of the global model, the edge device typically needs to have access to labels (e.g., tags and bounding boxes) for each sample (e.g., image) used for the update. However, these labels are not available without human intervention, and thus supervised learning cannot be used to update the local version of the global model. For this reason, the collaborative framework can be useful for self-supervised learning in adding to the distillation of knowledge to adapt the local version of the global model. Figure 3 includes an overview for an example of an algorithm designed according to the collaborative framework. As an example, assume that a local version of the global model implemented on the edge device is adapted. As mentioned above, the local version of the global model can be referred to as a "local model". To do this, the edge device must first collect data to use in the adaptation process. This "new data" can be fed to the local model and the teacher model, and the outputs produced at different layers of the local model can be calculated. Two different losses are then used to update the local model.

[0042] The first loss is called "knowledge distillation loss". At a high level, the idea of ​​knowledge distillation in this setting is that the teacher model becomes more powerful than the local model during feature representation. Thus, in scenarios where a new classification of the data causes the local model to become less reliable, a global model with more valuable feature representations can help the local model to adapt to the new classification. This can be done using knowledge distillation loss, which captures how the local and global model representations of the new data differ. Gradient updates from the knowledge distillation loss can be used to improve the fit of the local model.

[0043] The second loss is called "self-supervised task loss" or "self-supervised loss". Since there are no labels available for new data in this scenario, a self-supervised task must be defined to update the local model. A self-supervised task has its own loss that can be used to update the parameters of the local model. There is a variety of self-supervised tasks for visual convergence models, and any of these self-supervised tasks can be used uniformly in the collaborative framework.

[0044] These losses and corresponding gradients may be used to update the local model, as shown in Figure 3. In particular, these losses and corresponding gradients may be used to update parameters of the local model, thereby adjusting the local model after deployment of the edge device.

[0045] For knowledge distillation losses, feature representations can be used at different levels and output classification probabilities (also called "class classification probabilities") can be used to match local model features to corresponding teacher model features. The losses can include, but are not limited to, knowledge distillation losses such as feature maps for output classification probabilities and Euclidean distance for Kullback-Leibler (KL) divergence.

[0046] For self-supervised losses, pseudo-tasks that are different from the main goal of the model can be introduced as a means of supervision. The pseudo-tasks can be based on the characteristics of the edge device, the characteristics of the data generated by the edge device, or the characteristics of the local model. For example, pseudo-tasks can be chosen for each device using metrics such as gradient diversity or Shapley value to determine the quality of the updates from each pseudo-task. Each pseudo-task corresponds to a loss function, and the overall loss of the pseudo-tasks can be used to update the global model so that its representation is adapted to account for the generation of data generated by the edge device. For example, A different pseudo task for TIFF2024507765000003.tif7150 When used with a loss function such as TIFF2024507765000004.tif7150, then the total loss for a self-supervised task can be written as

number

[0047] The edge device can then use the adapted local model to derive insights from the generated data. In other words, the edge device can use the adapted local model to derive inferences from the data. FIG. 4 includes a high-level overview of a monitoring system in which self-supervised customization is performed for the models utilized by the edge device. In this context, the teacher model TIFF2024507765000013.tif7150 is maintained in a cloud, e.g., a server system accessible via the Internet. Global Model TIFF2024507765000014.tif7150 is used as a teacher model by edge devices, each of which is a local model A less computationally intensive version can be run, referred to as TIFF2024507765000015.tif7150. Each local model TIFF2024507765000016.tif7150 may be adapted based on data generated by the corresponding edge device, thereby It produces a fitted local model represented as TIFF2024507765000017.tif7150. The backpropagation of gradients and the calculations for updating the local model can be handled in any of the following places: Edge devices themselves can compute those gradients and update the model based on the data generated. This option is desirable because it minimizes communication costs and preserves data privacy. A server system may be responsible for computing updates for the local models deployed to the edge devices. This is done in a siloed manner so that the server system adapts each local model using only the data generated by the corresponding edge device. In this scenario, data is communicated by each edge device to the server system, and then a customized version of the global model can be sent back to each edge device. The intermediary device may be responsible for computing updates for the local model deployed to the edge device. The intermediary device may be communicatively connected to the server system and / or the edge device included in the monitoring system. Examples of intermediary devices include mobile phones, tablet computers, and base stations. Generally, the intermediary device has greater computing power than the edge device and is therefore better able to handle the computations required to update the local model.

[0048] A collaborative framework may be implemented so that local models can be continuously updated to account for changes in the classification of data generated by the corresponding edge device. However, due to the significant computational resources required for the continuous updating of local models, the collaborative framework has a mechanism to ensure that local models are updated only when necessary. A significant challenge is to define how often updates need to occur. Updates should occur regularly enough to keep the local models accurate, but not so frequently that the performance of the monitoring system suffers due to bandwidth limitations or processing resources. To automatically determine the optimal time to initiate an update procedure, the edge device monitors the reliability of the models of the outputs produced over time. When the reliability falls below a threshold for a given amount of time and remains below the threshold, the edge device may initiate the update procedure. Alternatively, the edge device may initiate the update procedure in response to a determination that a reliable amount of data (e.g., a reliable number of images) has been generated since the local model was last trained. As mentioned above, the update procedure may also be triggered by a server system or an intermediate device. Thus, to initiate an update procedure, the average reliability level of the models at the i edge devices ( TIFF2024507765000018.tif7150) is a threshold for a given time interval ( TIFF2024507765000019.tif7150) to remain below the threshold. FIG. 5 includes a high-level flowchart illustrating how the average confidence in the output produced by a local model trained for object detection can be calculated in an ongoing manner to establish when the local model should be updated.

[0049] (B. Methodology for realizing the collaborative framework) 6 includes a high level illustration of communications associated with an edge device 600 responsible for applying a local model to data generated while monitoring an environment to identify events of interest. By way of example, the edge device can be a camera responsible for applying a model to images of a residence to detect objects contained in those images. In FIG. 6, the edge device 600 completes the procedure for adapting the model by communicating with a server system 650. However, those skilled in the art will recognize that the actions performed by the server system 650 can also be performed by an intermediary device to which the edge device 600 is communicatively connected.

[0050] Initially, the edge device 600 may obtain the global model from the server system 650 (step 601). In some embodiments, the global model is obtained by the server system 650 before deployment of the edge device 600 (e.g., during production or calibration process). In some embodiments, the global model is obtained by the server system 650 after deployment of the edge device 600. For example, upon deployment in an environment of interest, the edge device 600 may establish a connection, either direct or unconnected, with the server system 650. In such a scenario, the server system 650 sends the latest version of the global model to the edge device 600.

[0051] The edge device 600 may then apply the global model to the data generated by the edge device 600 (step 602). Each time the global model is applied to the data, an output may be produced that is representative of a prediction or inference. Referring again to the example above, if the edge device is a camera that generates images of an environment over time, it may indicate the presence of objects in each image (e.g., using labels and bounding boxes).

[0052] Over time, the edge device 600 can monitor the performance of the global model. For example, the edge device 600 can calculate a metric indicative of the reliability of any time the global model is applied. Thus, the edge device 600 can continually track the performance of the global model with respect to whether the output is accurate. If the metric exceeds a threshold, then the edge device 600 can infer that the global model is performing effectively enough (hence, no change is required). However, if the metric does not exceed the threshold, the edge device 600 can infer that the performance of the global model is insufficient to merit adaptation. Thus, the edge device 600 can determine that adaptation is required based on an analysis of the output produced by the global model (step 603).

[0053] The edge device 600 can then create a local version of the global model adapted for the data generated by the edge device 600 (step 604). Again, assume that the edge device 600 is a camera that generates images of the surroundings. In such a scenario, the edge device can acquire a series of images generated over time and can apply the local version of the global model to generate a first series of outputs. As described above, these outputs can be labels and bounding box representations that indicate certain objects in the series of images. Moreover, the edge device 600 can transmit the series of images to the server system 650. The server system 650 can apply its version of the global model to the series of images to generate a second series of outputs. Typically, the teacher model implemented by the server system 650 will be more computationally intensive than the local version of the global model implemented by the edge device 600. These models can differ, for example, due to differences in computing resources available to the edge device 600 and the server system 650. Moreover, the global model implemented by the server system 650 can be further trained and tuned after it is initially acquired by the edge device 600. By comparing the first and second series of outputs, the edge device 600 can determine how to adjust its local version of a global model that represents the environment in which the edge device 600 is deployed (and therefore the data that the edge device 600 generates).

[0054] In some embodiments, the edge device 600 transmits information about the local version of the global model to the server system 650 (step 605). For example, the server system 650 simply stores this information in a digital profile associated with the surveillance system of which the edge device is a part. Alternatively, the server system 650 may use this information to improve other edge devices. For example, this information may be incorporated into the training of the global model, or this information may be used to adjust other edge devices that are part of a similar surveillance system, such as the edge device 600. Thus, improvements may be integrated across the entire surveillance system of which the edge device 600 is a part. For example, if the edge device 600 is a camera deployed in the backyard of a house, then some insight gained through adaptation of the local version of the global model may be applied by other cameras deployed in the front yard of the house.

[0055] 6 has been (and likely will be) performed iteratively to continually adapt the local version of the global model to data generated by edge device 600. Thus, steps 602-604 may be performed repeatedly as data is generated by edge device 600 over time.

[0056] 7 includes a flow diagram of a process 700 for creating a local model adapted to the environment in which an edge device is deployed. Initially, the edge device may obtain a version of the model from a server system that stores it locally (step 701). This version of the global model may be referred to as the local model because it is stored locally (e.g., on the edge device).

[0057] The edge device can then adjust parameters of the local model based on the data generated by the edge device to ensure that the local model is adapted to the environment in which the edge device is deployed (step 702). For example, the edge device can adjust the parameters based on an analysis of the output produced by the local model as it is adapted to the data. As another example, the edge device can adjust the parameters based on a comparison of (i) the output produced by the local model as it is applied to the data, and (ii) the output produced by the teacher model as it is applied to the data. Typically, the data is always sent back to the server system so that the teacher model can be applied. However, if the edge device has sufficient processing resources, the edge device can apply the local model and the teacher model to the data, even though only the output by the local model can be used for inference purposes.

[0058] Therefore, the edge device can monitor the generated data over time to discover variations in classification that are not transitory among the attributes (step 703). In other words, the edge device can monitor the generated data to discover variations in the context (even content). To accomplish this, the edge device can examine the output produced by the local model as it is applied to the data rather than its own data. For example, the edge device can track the confidence in the output produced by the local model to determine whether performance is improving, deteriorating, or remaining the same. If the edge device discovers that performance is deteriorating, then the edge device can initiate an adaptation procedure, for example by comparing the confidence of the output to a threshold. Thus, the edge device can adjust the local model in response to discovering that variations in the environment affect the performance of the local model (step 704). Step 704 of FIG. 7 can be substantially similar to step 604 of FIG. 6.

[0059] 8 includes a flow diagram of a process 800 for adapting a local model by an edge device deployed in a monitored environment. First, a server system can identify a model trained to perform a task (step 801). The task (and thus the model) is based on the edge device. For example, if the edge device is a camera, the model can be one that can detect objects from an image.

[0060] The server system can provide training data to the model to create a global model trained to perform the task (step 802). Typically, the training data includes samples and corresponding labels identified by an individual. Referring again to the example above, if the model is trained to detect objects in images, then the training data may include a series of images with associated bounding boxes that define the location of the objects in each image. The training data is provided as input to the model to allow the model to learn characteristics that indicate the presence of objects from the training data.

[0061] The server system can then provide a version of the global model to the edge device (step 803). In some embodiments, this occurs during a production or calibration process that occurs prior to sale to a user. In some embodiments, this occurs following deployment in an environment monitored by a user. For example, after deployment, the edge device initiates a connection with the server system and then requests the most recent version of the global model. This local version of the global model may be referred to as the "local model." The edge device may then apply the local model to the generated data to produce an output that is representative of an inference or prediction.

[0062] In some embodiments, the server system receives an input indicating a request from the edge device to apply the global model (step 804). For example, assume that the edge device determines that a local model should be adapted based on data generated by the edge device. In such a scenario, the edge device may determine how to adapt the local model based on a comparison of the outputs produced by the local model to the outputs produced by the global model, as described above. Thus, the edge device may request that the server system apply the global model to the data generated by the edge device. The server system may provide the outputs produced by the global model, if any, to the edge device (step 805). As described above with reference to FIG. 7, the edge device may adapt the local model based on those outputs.

[0063] Alternatively, the server system may be responsible for adapting the local model as described above. In such an embodiment, the edge device may also transmit the outputs produced by the local model, if any, to the server system. The server system may use (i) the outputs produced by the local model and (ii) the outputs produced by the global model to adapt a version of the global model that can be provided to the edge system for use as the local model.

[0064] Except where contrary to the possibility, these steps may be performed in various orders and combinations. For example, an edge device may simultaneously apply models for inference purposes and examine the output produced by the local model to determine performance. As another example, some steps of the process of Figures 6-8 may be repeatedly performed to ensure that the local model is adapted whenever analysis of the output indicates that performance has deteriorated past an appropriate point. Moreover, one skilled in the art will recognize that the process of Figures 6-7 may be simultaneously performed by different edge devices of a similar monitoring system. For example, assume that a monitoring system includes multiple edge devices deployed at different locations in an environment of interest. In such a scenario, one edge device may determine that its own local model should be adapted, while the other edge device may determine that its own local model does not require any changes. Similarly, multiple instances of the process of Figure 8 may be independently and simultaneously performed by a server system for different edge devices.

[0065] Other steps may also be included in some embodiments. As one example, information gleaned piecemeal through adapting local models may be applied in an integrated manner as described above. As an example, adaptations made to a local model utilized by one edge device of a given environment may also be made to local models utilized by other edge devices of the given environment. As another example, information regarding the adaptation of local models may be surfaced for review by individuals. For example, an owner of a surveillance system may be notified (e.g., by a computer program executing on a computing device such as a mobile phone or tablet computer) whenever a local model utilized by an edge device included in the surveillance system is updated.

[0066] (Processing System) 9 is a block diagram illustrating an example of a processing system 900 capable of implementing at least some of the processes described herein. For example, components of the processing system 900 may be hosted on an edge device, an intermediary device, or a server system.

[0067] The processing system 900 may include one or more central processing units (processors) 902, a main memory 906, a non-volatile memory 910, a network adapter 912, a video display 918, input / output devices 920, a control device 922 (e.g., a keyboard or pointing device), a drive unit 924 including a storage medium 926, and a signal generating device 930 communicatively coupled to a bus 916. The bus 916 is illustrated as an abstraction representing one or more physical buses or point-to-point connections connected by appropriate bridges, adapters, or controllers. The bus 916 may thus include a system bus, a Peripheral Component Interconnect (PCI) bus or PCI-Express bus, a HyperTransport or Industry Standard Architecture (ISA) bus, a Small Computer System Interface (SCSI) bus, a Universal Serial Bus (USB), an Inter-Integrated Circuit (I2C) bus, or an Institute of Electrical and Electronics Engineers (IEEE) standard 1394 bus (also referred to as "Firewire").

[0068] The processing system 900 may share a similar processor architecture with a desktop computer, a tablet computer, a mobile phone, a game console, a music player, a wearable electronic device (e.g., a watch or fitness tracker), a network-connected ("smart") device (e.g., a television or home assistant device), a virtual / augmented reality system (e.g., a head-mounted display), or other electronic device, which can execute a set of instructions (sequentially or differently) that specify actions that the processing system 900 takes.

[0069] While main memory 906, non-volatile memory 910, and storage medium 926 are depicted as a single medium, the terms "machine-readable medium" and "storage medium" may include a single medium or multiple media (e.g., a centralized or distributed database or associated caches and servers) that store one or more sets of instructions 928. Additionally, the terms "machine-readable medium" and "storage medium" are intended to include any medium capable of storing, encoding, or transmitting a set of instructions for execution by processing system 900.

[0070] Generally, the procedures performed to implement the embodiments of the present disclosure may be implemented as part of an operating system or a specific application, component, program, object, module, or sequence of instructions (collectively referred to as a "computer program"). A computer program typically comprises one or more instructions (e.g., instructions 904, 908, 928) set at various times in various memory and storage devices of an electronic device. When read and executed by the processor 902, the instructions cause the processing system 900 to perform operations to implement elements associated with various aspects of the present disclosure.

[0071] Moreover, while the embodiments have been described in the context of a fully-functional electronic device, those skilled in the art will recognize that some aspects of the techniques may be distributed as a program product in various forms, and the present disclosure applies regardless of the particular type of machine or computer-readable medium used to effect the classification.

[0072] Further examples of machine- and computer-readable media include store-able media such as volatile and non-volatile memory devices 910, removable disks, hard disk drives, and optical disks (e.g., Compact Disk Read-Only Memories (CD-ROMS) and Digital Versatile Disks (DVDs)), as well as transparent media such as digital and analog communication links.

[0073] The network adapter 912 enables the processing system 900 to broker data over a network 914 with components external to the processing system 900 through any communication protocol supported by the processing system 900 and the external components. The network adapter 912 may include a network adapter card, a wireless network interface card, a router, an access point, a wireless router, a switch, a multi-layer switch, a protocol converter, a gateway, a bridge, a bridge router, a hub, a digital media receiver, a repeater, or any combination thereof.

[0074] The network adapter 912 may include a firewall that governs and / or manages access permissions or proxy data in the network. The firewall may also track changing levels of trust between different machines and / or applications. The firewall may be any number of modules with any combination of hardware, firmware, or software components that can enforce a set of predefined access permissions between a set of machines and applications, between machines, or between applications (e.g., governing the flow of traffic and resources shared between these components). In addition, the firewall may manage and / or have access to access control lists that detail the permissions, including the rights of individuals, machines, or applications to access and operate objects, and the circumstances under which the permissions are valid.

[0075] (Notes) The various embodiments of the claimed subject matter set forth above have been provided for purposes of illustration and description. It is not intended to be exhaustive or to limit the claimed subject matter to the precise form disclosed. Many modifications and variations will become apparent to those skilled in the art. The embodiments have been selected and described in order to best explain the principles of the invention and its practical application so that others skilled in the art can understand the claimed subject matter, various embodiments, and various modifications that are suitable for the particular use contemplated.

[0076] Although the detailed description of the invention describes certain embodiments and the best mode contemplated, the technology can be practiced in many ways, no matter how detailed the detailed description of the invention is. The embodiments may vary significantly in their implementation details until covered by the specification. The specific terminology used when describing certain features or aspects of the various embodiments should not be understood to mean that the terminology is redefined here to limit the technology to any particular characteristic, feature, or aspect of the technology to which the terminology is associated. In general, the terms used in the following claims should not be interpreted to limit the technology to the specific embodiments disclosed herein, unless those terms are expressly defined in the text. Thus, the actual scope of the technology encompasses not only the disclosed embodiments, but all similar ways of practicing and implementing the embodiments.

[0077] The language used in this specification has been selected primarily for ease of reading and instructional intent. It may not have been selected to delineate or define the subject matter. Thus, it is intended that the scope be limited not by the detailed description, but rather by any claims issuing in the prosecution based on this specification. Thus, the disclosure of various embodiments is intended to be illustrative, but not limiting, of the scope as defined by the following claims.

Claims

1. obtaining an image labeled to indicate inclusion of a defined object; training the model to detect instances of the defined object by providing the images to the model as training data; Transmitting the trained model to a camera placed in the monitored environment. A server system configured as above; generating a first series of images of the environment; Adjusting parameters of the trained model based on analysis of the first series of images to create a local version of the trained model adapted to the environment. and the camera configured to

2. The camera further comprises: generating a second series of images of the environment; applying the local version of the trained data to each image in the second set of images to produce a set of outputs; Compute a metric indicative of confidence in the set of outputs produced by the local version of the trained model; Comparing the metric to a threshold; Readjusting the parameters in response to determining that the metric falls below the threshold. The monitoring system according to claim 1 , configured to:

3. 2. The surveillance system of claim 1, wherein the camera is configured to perform the adjustment in response to determining that a predetermined number of images of the environment have been captured since the local version of the trained model was last adjusted.

4. The surveillance system of claim 1 , wherein the camera is further configured to transmit information regarding the adjusted parameters to the server system.

5. 2. The surveillance system of claim 1, wherein the camera is one of a plurality of cameras to which the server system sends the trained data, and each camera independently creates a different local version of the trained model.

6. The surveillance system of claim 5 , wherein the plurality of cameras are positioned in the environment to be monitored.

7. obtaining from a server system, by an edge device, a model trained to identify events of interest when applied to data generated by the edge device; and adjusting, by the edge device, parameters of the model to create a local version of the model adapted to the environment in which the edge device is located; A method comprising:

8. 8. The method of claim 7, further comprising monitoring the data generated by the edge devices over time to discover content variations that are non-transient in nature.

9. Furthermore, the edge device identifying a portion of the data that corresponds to a variation in the content; applying a cloud-based model to the portion of the data to produce a first output, causing the cloud-based model to be more powerful than a local version of the model; applying the local version of the model to the portion of the data to produce a second output; calculating a metric indicative of similarity between the first and second outputs; and modifying the parameters of the local version of the model based on the metrics; adjusting a local version of the model in response to discovering a variation in the content. The method of claim 8 , comprising:

10. The causing includes: transmitting said portion of said data to said server system; and receiving from the server system the first output generated by the model when applied to the portion of the data.

10. The method of claim 9, comprising:

11. The method of claim 7 , wherein the edge device is a camera and the data includes an image of the environment.

12. identifying, by a server system, a model to be trained to perform a task; providing training data to the model to produce a global model that is trained by the server system to perform the task; serving, by the server system, a version of the global model to edge devices responsible for monitoring an environment of interest; receiving, by the server system, an input indicating a request from the edge device to apply the global model to data generated by the edge device; and Providing, by the server system, an output generated by the global model when the data is applied to the edge device. A method comprising:

13. The method of claim 12 , wherein the providing is performed prior to deployment of the edge devices in the environment of interest.

14. The method of claim 12 , wherein the providing is performed after placement of the edge device in the environment of interest.

15. the edge device is a camera; the global model is trained to detect instances of objects in images; the training data includes a sequence of images and associated labels, each identifying the location of the object in the corresponding image; The method of claim 12.

Images