Environment-specific training of machine learning models

Abstract

A method for environment-specific training of a machine learning model includes receiving a data stream for a local environment that includes a plurality of sequential data segments. Programmatic labels are generated for each data segment using a student version of the machine learning model. A portion of the data segments and associated programmatic labels are selected for evaluation by a teacher version of the machine learning model and uploaded to a server-side computing device. An environment-specific training update is received from the server-side computing device. The training update is based on a comparison of the selected programmatic labels and pseudo labels generated by the teacher version for the selected portion of the data segments. The environment-specific training update is applied to the student version to generate an updated student version. The updated student version machine learning model is then used to generate programmatic labels for newly received data segments.

Description

Background Technology

[0001] Powerful machine learning models can be used to perform tasks such as image analysis, speech recognition, natural language processing, and data analysis. Given sufficient time and computational resources to perform such tasks, trained machine learning models can provide accurate and reproducible analysis. For scenarios requiring real-time analysis and / or when local computing resources are limited, knowledge from large machine learning models can be distilled into smaller models for local deployment without sacrificing effectiveness. Summary of the Invention

[0002] A method for environment-specific training of a machine learning model includes: receiving a data stream comprising multiple sequential data segments for a local environment; generating programming tags for each data segment using a student version of the machine learning model; selecting a portion of the data segment for evaluation by a teacher version of the machine learning model and its associated programming tags, and uploading it to a server-side computing device; receiving environment-specific training updates from the server-side computing device. These updates are based on a comparison of the selected programming tags with pseudo-tags generated by the teacher version for the selected portion of the data segment; applying the environment-specific training updates to the student version to generate an updated student version; and then using the updated student version of the machine learning model to generate programming tags for newly received data segments.

[0003] This summary is provided to present a simplified description of selected concepts, which will be further described in the detailed description below. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. Furthermore, the claimed subject matter is not limited to implementations that address any or all of the disadvantages pointed out in any part of this disclosure. Attached Figure Description

[0004] Figure 1 An example local environment for deploying machine learning models is shown.

[0005] Figure 2 An example computing system for deploying and updating machine learning models in a local environment is shown.

[0006] Figure 3 This is an example method for environment-specific training of machine learning models.

[0007] Figure 4 This is an example method for retraining machine learning models for local environments.

[0008] Figure 5 This is an example workflow for iterative active distillation of machine learning models.

[0009] Figure 6 It shows what can be used in it. Figure 2 A schematic diagram of an example computing environment for computer devices. Detailed Implementation

[0010] Machine learning models can be used to perform a wide variety of functions in various environments. They can be assigned tasks such as image classification, speech recognition, text interpretation, human pose and gesture estimation, people detection, and object counting. These tasks can be performed using data collected from any suitable environment via one or more sensors or any other suitable collection mechanism.

[0011] Figure 1 An example local environment 100 is depicted, including a retail store 101 and an adjacent parking lot 102. Several surveillance cameras 105 are pointed at the interior of the retail store 101 and the parking lot 102. Each surveillance camera may include one or more 2D imaging devices. Image data acquired by the surveillance cameras can be used to detect objects (such as people 110 and vehicles 112) and / or other items related to store management (such as bicycles, shopping carts, assistive mobile devices, etc.). Image data from the surveillance cameras 105 can also be used to identify individuals or analyze human movement or other behaviors, individually or in related groupings.

[0012] Additional environmental sensors 115 may include 3D imaging devices, microphones, etc., for human-computer interaction. For example, data from 2D and / or 3D imaging devices can facilitate pose and / or gesture detection for use in video games or other natural language input applications, such as interactive self-service terminals 120, which may also use data from microphones to provide voice commands, transcription services, etc.

[0013] The machine learning models deployed in the example environments mentioned above are typically general-purpose and trained on public datasets. However, there may be specific local factors that can obfuscate the machine learning models, factors that may not exist in other environments. For example, parking lot 102 may have specific lighting patterns throughout the day (e.g., sunlight, shade, artificial lighting), which could benefit from location-specific processing to enable accurate image analysis. The location of retail store 101 may influence customer demographics, which in turn means a higher prevalence of certain accents / dialects / consonants, etc. Deploying machine learning models trained on all possible environmental factors can result in bloated software that may struggle to respond quickly and accurately to real-time data from environmental sensors. Training each machine learning model for its own local environment is preferable, but also challenging.

[0014] In traditional teacher-student models, the teacher and student versions operate on the same model (often a general model). Both versions are trained on the same static benchmark dataset for evaluation. Nothing can be distilled, so the student version cannot be improved except by examining the teacher version's inference on the dataset. This traditional active learning approach relies on human effort in labeling selected data. Generally, the student version is trained on all available data, resulting in slow training and inefficient model updates. Similarly, traditional distillation methods distill knowledge from the teacher version to the student version based on all available data. This process is also relatively slow and therefore unsuitable for continuous training in real-world scenarios where the amount of real-time data received and evaluated can be enormous.

[0015] Here, systems and methods for an active distillation framework are described, enabling small student-version machine learning models to achieve automatic, scalable, and cost-effective performance improvements within a specific environment. In some embodiments, a large teacher-version machine learning model uses actively selected training data as the basis for a rapid, incremental training process, thereby selecting training data from data received in the deployment environment. The student version can operate online, generating programmed labels for data snippets in real time. The teacher version can operate offline, using additional time and complexity to generate pseudo-labels for portions of data snippets already programmed by the student version, and then iteratively deploying training updates based on a comparison of pseudo-labels and programmed labels. This workflow enables incremental, feedback-driven training schemes that do not require manual labeling to improve the student version. In fact, a server hosting the teacher version can continuously and in parallel train multiple student versions from different environments, providing environment-specific training updates to each client based on data from the deployment environment of each student version. Therefore, each student version can be tailored to its own environment without becoming excessively large and complex. Traditional methods of manually collecting environment-specific data may be practical for a single client, but are not practical for large-scale deployment scenarios.

[0016] Figure 2 An example system 200 for deploying and training machine learning models for a local environment is illustrated schematically. System 200 includes a server-side computing device 202 located in a remote environment 204 and operating a teacher-side machine learning model 206. System 200 also includes a client-side computing device 208 located in a local environment 210 and configured to operate a student-side machine learning model 212.

[0017] The remote environment 204 can be a physical location (such as a group of servers) and / or a cloud computing environment. In some examples, the server-side computing device 202 can be located in or tightly linked to the local environment 210. The local environment 210 can be any suitable environment from which data can be collected and then analyzed by the student version 212.

[0018] Both the teacher version of machine learning model 206 and the student version of machine learning model 212 can be any suitable machine learning model, such as deep neural networks, deep learning neural networks, deep convolutional neural networks, recurrent neural networks, etc. This document also references... Figure 6 Additional examples of machine learning models and neural networks are described.

[0019] Teacher version 206 and student version 212 can be the same machine learning model, such as the same neural network. Teacher version 206 can be configured to generate and deploy student version 212. Student version 212 can be viewed as a smaller version of the machine learning model, while teacher version 206 can be considered a larger version. For example, teacher version 206 may include deeper layers, wider layers, more neurons, etc. Furthermore, while student version 212 can operate in real-time or near real-time, teacher version 206 can efficiently operate offline by leveraging the additional time required to run inference on the data. Therefore, the deployed student version machine learning model 212 can be smaller and scaled down.

[0020] Teacher version 206 can initially be trained, at least partially, under human supervision. Additional updates to the teacher version can be generated through supervised training and / or through higher-level active distillation training via a larger, primary version of the machine learning model. Teacher version 206 can be configured to provide distillation training services for student version 212, thereby incrementally fine-tuning student version 212 within the client-side computing device for its environment.

[0021] Server-side computing device 202 includes a server-side network interface 214 configured to communicate with a client-side network interface 216 that operates as part of client-side computing device 208. The server-side network interface 214 and the client-side network interface 216 are operable to facilitate communication between server-side computing device 202 and client-side computing device 208, including uploading and downloading data fragments, associated programming tags, and training updates.

[0022] Environmental sensor 218 can collect data from the local environment 210, and then it can generate and / or input a data stream 220. Environmental sensor 218 may include a passive sensor that continuously collects data and / or a cue sensor that collects data for a duration that meets data collection conditions. Environmental sensor 218 may include 2D imaging devices, 3D imaging devices, microphones, seismic sensors, weather sensors, pressure sensors, radiation sensors, and / or any sensor configured to identify, collect, and store data about its surrounding environment.

[0023] Data stream 220 may include data output by environmental sensor 218, and data transcriptions that are neither generated from nor indirectly generated via local environment 210. Although this is directed to an application in which data is generated by environmental sensor 218 within local environment 210 and processed by client-side computing device 208 located in the same environment, consideration has been given to positioning the sensors and processor in separate sub-environments.

[0024] Data stream 220 includes multiple sequential data segments. Each data segment may have a fixed size or duration, or it may have a variable size or duration. For example, in a scenario where data stream 220 includes one or more 2D image streams, each data segment may include one or more image frames. In a scenario where data stream 220 includes one or more 3D image streams, each data segment may include one or more 3D image frames. In a scenario where data stream 220 includes one or more audio streams, each data segment may include a finite number of audio data chunks. For image detection or counting algorithms, each collected frame may be sent to data stream 220, periodic frames may be sent to data stream 220, and / or cue frames may be sent to data stream 220. For example, a face recognition algorithm may only receive frames acquired within a certain period after a door opening cue. Similarly, a natural language recognition program may send continuous data streams or data chunks where sensors have identified an individual, and / or when a gesture or gesture may have been initiated. For speech recognition and transcription applications, each data segment may include one or more identified utterances characterized by a pause or amplitude below a threshold on either side of the utterance. For example, each data segment can include words, phrases, sentences, etc. In some examples, a data segment can include a fixed-length duration of an audio recording.

[0025] Local data interface 222 can be configured to receive a data stream 220 comprising multiple consecutive data segments for a local environment 210 and to provide the data stream 220 to student version 212. Student version 212 can be configured to perform inference on the input data stream 220. Student version 212 can be configured to generate and output a programmed tag for each data segment based on the inference results, and thus output programmed-tagged data segments 224. In this scenario, a “programmed tag” refers to a tag automatically applied to a data segment after analysis by student version 212. For example, if the task of student version 212 is to identify each person in a 2D image frame, a programmed tag can be attached to coordinate locations on the 2D image, where the inference run by student version 212 indicates that these coordinate locations are above a threshold probability of being occupied. Another programmed meta-tag can be used to count the total number of people inferred by student version 212 for an image frame (e.g., a data segment). The nature of the programmed tags can depend on the task assigned to student version 212. Speech recognition tasks can lead to programmed tags indicating which word may have been spoken in a data segment, or whether a specific word or sequence of words was spoken in the data segment. In other words, the programmed tags represent the best guess or inference made by the student version 212 about whether the data segment includes certain data elements described in the machine learning model. As further described herein, the student version 212 can be trained for local environment 210 based on data stream 220 and programmed tagged data segments 224.

[0026] The student version of the machine learning model 212 can also be configured to provide at least some of the programmed labeled data segments 224 to the local output 226. For example, the local output 226 could be a local computing device that presents data from one or more programmed labeled data segments 224 to a human operator or uses information from the programmed labeled data segments 224 to initiate automated tasks. The local output 226 can translate the programmed labeled data segments for immediate use within the local environment 210. For example, labeled 2D images can be used to represent objects and / or people within the local environment 210. This enables the computing device or human operator to identify individuals and objects within the local environment and / or to count people or objects. Figure 1Within the environment 100 shown, this allows the store to track the number of people 110 in retail store 101 or parking lot 102, the number of vehicles 112 or shopping carts in the parking lot, etc. Additionally or alternatively, the store may be able to track the movement of specific individuals as they pass through store aisles and departments. This data can be tracked over time to inform store management about customers, customer shopping patterns, etc., over time. For natural language purposes, the recognition of performed gestures can indicate commands at self-service kiosks, gaming devices, or other computing devices, or whether the data stream is 2D or 3D image. For speech recognition applications, data may include spoken commands, transcriptions, etc. Programmed tags can indicate what the student version believes it is saying, and / or whether it is uttering a word or command, and at what frequency.

[0027] Selector 228 can be configured to select a portion of the data segments used for evaluation of the distillation training service of student version 212 by teacher version machine learning model 206, instead of uploading all programmatically labeled data segments. Selector 228 can employ a selection process that reduces the amount of data uploaded to the server-side computing device, thereby reducing training and student version update time. Selector 228 may include one or more computing devices, and / or may be installed as a program, module, or engine on one or more computing devices. Selector 228 can be configured to receive a stream of data segments and output a portion of the data segments (e.g., less than all received data segments). As described herein, the selection of the output portion of the data segments can be active or passive. The selected data segments 230 and associated labels 232 can be uploaded to teacher version 206 for evaluation and bundled into episodes.

[0028] Selector 228 can receive programmed tagged data segments 224, which may include data segments and their associated programmed tags, or as raw or filtered feeds to data stream 220. Here, "raw" data stream may refer to unprocessed image data, while the data stream may undergo some processing or filtering before generating data segments that can be tagged by student version 212. Selector 228 can then specify a portion of the programmed tagged data segments 224. Selector 228 can be delegated to obtain a representative sample of data segments, including a variety of signals and information that teacher version 206 can use to improve the performance of student version 212 when encountering similar signals and information.

[0029] In some examples, selector 228 may use random selection of data segments (e.g., 1% of all input data segments) without considering their associated programmed tags. This frequency may be adjusted based on dynamic information included in data stream 220 or based on predetermined rules. In other examples, selected data segments 230 may be selected at least in part based on their associated tags 232. In some examples, selector 228 may be informed of the presence or absence of a specific tag. For example, if the task of student version 212 is to count pickup trucks in a parking lot, data segments may be preferentially selected to include a wide variety of pickup truck to total vehicles ratios, some of which appear to indicate a majority of pickup trucks, while others appear to indicate a lack of pickup trucks. For voice command or gesture recognition tasks, a certain number of data segments with positive event recognition may be selected, and a certain number of data segments with negative event recognition may be selected as a control.

[0030] In some examples, labels and / or other information related to recent training updates can be given greater weight. For instance, if a people counting algorithm performs poorly in images taken under sunny conditions, an update can be generated to improve people counting under such conditions. Thus, data segments with these conditions may be over-presented during the selection process. In the speech recognition module, if the algorithm misidentifies certain words (e.g., due to accents common in the local environment), updates can target these speech patterns. Thus, data segments with labels indicating corrected words and / or labels indicating potentially incorrect guesses can be over-presented at selector 228.

[0031] In some examples, selector 228 can use intermediate outputs from student version 212 to generate selected data segments 230. As an example, the confidence intervals of the markers output by the student model can inform the selection, as student version 212 can effectively understand what kind of problems they are encountering over time. Other data included in the data segments (such as motion or rate of change in video frames) can be used to increase the selection rate during periods of high activity and decrease it during periods of relative stillness between frames of the image data. Furthermore, if the markers fluctuate at a high rate, this may indicate that the student version is facing challenging inference, and therefore the data segments for that time period can be over-represented in the selected data segments 230.

[0032] In some examples, data segments can be selected based on their similarity or difference to the selected data segment 230 to provide positive and / or negative controls for evaluation. The selected data segment 230 and its associated tag 232 can be uploaded one at a time, bundled into a fixed-size data group, a variable-size data group, or a data section (described further herein).

[0033] Teacher version 206 receives a selective data segment 230 and may additionally receive associated programming tags 232. Teacher version 206 then generates pseudo-labeled data 234 for the received portion of the selected data segment 230, for example, by performing inference on that portion of the selected data segment. Similar to programming tags, pseudo-labels can be considered as tags applied to the data segment based on a machine learning algorithm or task assigned to teacher version 206. Neither the programming tags nor the pseudo-labels are verified by real humans. Thus, these tags are merely inferences. However, it can be assumed that teacher version 206 (with additional time, computational power, and experience) is more accurate than student version 212 in performing the same task. Thus, for the same data segment, the pseudo-labels applied by teacher version 206 will be considered more accurate than the programming tags applied by student version 212. In some examples, teacher version 206 may generate pseudo-labels for a portion of the selected data segment 230 without involving the associated programming tags 232. Additionally or alternatively, teacher version 206 may specifically determine whether the associated programming tags have been correctly applied.

[0034] For 2D image data, for example, Teacher version 206 can use higher resolution images compared to Student version 212, and therefore can identify more features of the target object. Conversely, Student version 212 (which has limited time available for data segment inference) can quickly process lower resolution images and may not have been extended to train to identify target objects from a reasonable amount of pixel data.

[0035] Student version 212 may only be able to target a single or a small number of solutions, while teacher version 206 has the time, ability, and experience to process multiple solutions (whether simultaneously or sequentially). One such approach is called multi-scale testing or multi-scale inference. For example, if student version 212 and teacher version 206 are executing the same basic machine learning model for 2D image analysis, teacher version 206 can operate 4x per frame at different resolutions, zoom in on the image, zoom out on the image, segment the frame, etc., to allow for the identification of multiple solutions. Accordingly, teacher version 206 can generate pseudo-labels for each selected data segment.

[0036] Comparator 236 can receive pseudo-labels 234 and programmed labels 232 for selected data segments 230 and perform comparisons. The comparator can record instances in which the student version 212 mislabels, underlabels, or otherwise incorrectly labels data segments. For example, programmed labels 232 may be inaccurate, or the teacher version 206 can apply pseudo-labels that identify objects not identified by the student version 212. The pseudo-labeled data 234 and associated labels 232 for the selected data segments 230 can be used to train the student version 212 using a distillation process. In this way, the student version 212 can receive training specific to the local environment 210, because the teacher version 206 can identify more accurate inferences for each selected data segment 230 and can identify commonalities between mislabeling and missed targets that may be caused by environmental features. For example, the teacher version 206 can identify patterns in lower-resolution data segments that the student version 212 can use to generate higher-confidence labels or to actively label completely missed objects within the data segments.

[0037] The data segment compiler 238 can compile selected data segments 230, associated tags 232, and pseudo-tags 234 into discrete data segments and determine when a data segment is sufficient to form the basis for retraining the student version 212. A data segment can include a predetermined number of data segments or a dynamically determined number of data segments. A data segment can be viewed as a collection or batch of data used to generate training updates. For example, depending on the available data segments, each data segment can have a fixed or variable length. Data segments can be compiled and distributed sequentially.

[0038] Once the data segment has been compiled, update generator 240 can generate training updates 242 specific to student version 212 and local environment 210. Training updates can add features, parameters, and / or layers to the student version to improve its performance within the local environment. The improved performance of student version 212 compared to the pseudo-labels applied to the same data segment by teacher version 206 can include increased accuracy in the programmed labels. In some examples, training update 242 can adjust parameter weights, such as giving more emphasis to certain parameters. Alternatively, training updates can remove unused or invalid features, parameters, layers, etc.

[0039] Once generated and configured through distillation training, training update 242 can be deployed to client-side computing device 208 to generate an updated student version 212. Training update 242 may include selection parameters, based on which selector 228 selects data segments 230. For instances where selector 228 uses random selection, the selection frequency can be increased or decreased based on the performance of student version 212 on previous data segments. For example, if teacher version 206 marks 75% of the selected data segments 230 as including the target object, but student version 212 marks only 62% of the same data segments as including the target object, the sampling frequency can be increased until student version 212 reaches a threshold for target object identification.

[0040] For instances where directional or target-based (e.g., intelligent) selection is used to select data fragment 230, the frequency and / or threshold can also be adjusted. For example, specific data fragments can be selected based on a threshold confidence level in the programmed labels and / or based on label ambiguity (e.g., two or more labels have similar confidence levels). Data fragments that include new training update features specific to the selected data fragments can be selected.

[0041] Specifically, for data segments of variable length (e.g., pose determination, speech recognition), a combined selection can be applied, where some data segments are targeted based on applied labels, and some data segments are selected randomly or pseudo-randomly, for example, data segments that do not include applied labels but do include some feature similarities (such as content, duration, background, etc.). In some examples, this combined selection method can be used to generate matching pairs or sets of data segments, where labeled data segments are matched with one or more different labeled and / or unlabeled data segments (still including some similar features). In this way, the teacher version 206 can use a case-control model to evaluate the student version 212.

[0042] In some examples, a server-side computing device 202 can be communicatively coupled to multiple local clients running in local environments (such as local environment (b) 244 and local environment (c) 246). Initially, the same student version 212 can be deployed to local environment (b) 244 and local environment (c) 246. However, since each student version is trained specifically for its local environment, training updates provided to each local environment may cause the student version to deviate over time. If local environments 210, (b) 244, and / or (c) 246 share characteristics, a curriculum from training one student version can be applied to other student versions to accelerate their training. For example, rare events can be aggregated across local environments to improve the detection scheme for all student versions.

[0043] Figure 3 A method 300 for environment-specific training of a machine learning model is described. Method 300 can be executed by a client-side computing device (such as client-side computing device 208) operating in a local environment.

[0044] At 310, method 300 includes receiving a data stream comprising multiple sequential data segments specific to a local environment. For example, a localized environmental sensor (such as an imaging sensor, microphone, etc.) may output a continuous stream of data segments received at a local data interface. Each data segment may be an evaluable block of data, such as one or more image frames, audio data blocks, etc.

[0045] At 320, method 300 includes using a student version of the machine learning model to generate programmed labels for each data segment. The student version of the machine learning model may be assigned one or more tasks that cause programmed labels to be applied to some data segments. For example, such tasks may include identifying objects in image frames and / or counting them, identifying poses or gestures in a series of image frames, recognizing verbal commands and utterances, etc. Given its current training capabilities, the student version may perform inference on the data segments as appropriate and label the data segments accordingly.

[0046] At 330, method 300 includes selecting a portion of a data segment for evaluation by the teacher-version machine learning model. In some examples, the data segment selected for evaluation is randomly selected. Alternatively, the data segment selected for evaluation may be selected at least in part based on programming tags. As an example, the data segment selected for evaluation is selected at least in part based on confidence ratings assigned to one or more programming tags. At 340, method 300 includes uploading the selected portion of the data segment and the associated programming tags to a server-side computing device including the teacher-version machine learning model.

[0047] At 350, method 300 includes receiving environment-specific training updates from a server-side computing device, the environment-specific training updates being based at least on a comparison of programmed labels associated with selected data segments and pseudo-labels generated for the selected data segments by the teacher-version machine learning model. In some examples, the environment-specific training updates may also be based on data comparisons performed in other, similar environments.

[0048] At 360, method 300 includes applying an environment-specific training update to the student version of the machine learning model to generate an updated student version of the machine learning model. In some examples, the environment-specific training update includes one or more selection parameters that can be used to determine a portion of the newly received data fragment for selection for evaluation by the teacher version of the machine learning model. For example, one or more selection parameters included in the training update may include selection frequencies.

[0049] At 370, method 300 includes generating programming tags for the new received data fragment using an updated student version of the machine learning model. At 380, method 300 includes providing at least some of the new received data fragment and the generated programming tags to a local output for use in evaluating events within a local environment. For example, the new received data fragment and the generated programming tags can be evaluated to determine the frequency of the event, the presence / absence of the event, etc.

[0050] Optionally, method 300 may include iteratively performing an update procedure, which includes: selecting a portion of a newly received data segment and uploading the selected portion of the newly received data segment to a server-side computing device for evaluation by the teacher version of the machine learning model; receiving new environment-specific training updates from the server-side computing device; and applying the new environment-specific training updates to generate an updated student version of the machine learning model.

[0051] In this way, the student version can iteratively report to the teacher version, providing information about how well the model performed after each update. The student version can be continuously adapted to the local environment, even as the environment itself changes over time.

[0052] Figure 4 A method 400 for retraining a machine learning model for a local environment is described. Method 400 can be performed by a server-side computing device (such as server-side computing device 202) that operates a teacher-version machine learning model in a remote environment.

[0053] At 410, method 400 includes receiving data fragments and associated programming tags from a client-side computing device employing a student version of the machine learning model. For example, the student version may have been deployed by the teacher version for use in a local environment where the client-side computing device operates. In some examples, data fragments and associated programming tags may be received from two or more client-side computing devices operating in different local environments.

[0054] At 420, method 400 includes generating pseudo-labels for each received data segment using a teacher-version machine learning model. (See also: Regarding...) Figure 2As described, the teacher version may have additional complexity, available time and resources, and additional training that are unavailable in the student version. Therefore, even if the pseudo-tags are not verified by human operators, they may still be more accurate than the programming tags generated by the student version.

[0055] At 430, method 400 includes performing a comparison of pseudo-tags with programmed tags for each received data segment. Continuing at 440, method 400 includes compiling two or more performed comparisons into a data section. The resulting data section may include a predetermined number of data segments, or may include a number of data segments determined based on at least some of the pseudo-tags and / or programmed tags.

[0056] At 450, method 400 includes generating environment-specific training updates for the student version of the machine learning model based on the data section. In some examples, generating environment-specific training updates is based on a difference greater than a threshold between pseudo-labels and programmed labels.

[0057] In an example where multiple client-side computing devices are operating the student version, different environment-specific training updates can be generated for each client-side computing device, at least based on data fragments received for that client-side computing device and associated programming tags. At least some aspects of each different environment-specific training update can be based on data fragments received from one or more other client-side computing devices and associated programming tags. In other words, a training update for one student version can be generated based on data collected from multiple student versions operating in separate local environments.

[0058] At 460, method 400 includes publishing the generated environment-specific training update to the client-side computing device. The generated environment-specific training update may include one or more selection parameters that control which newly generated data fragments are selected for uploading and evaluation. Optionally, at 470, method 400 includes iteratively training a student version of the machine learning model. Iterative training includes receiving new data fragments and associated programming tags from the client-side device, and generating new environment-specific training updates for the student version of the machine learning model based at least on pseudo-tags generated for the received new data fragments. Thus, the teacher version can continuously evaluate the student version based on the output of the student version at each update.

[0059] In the described implementation, even when the teacher and student versions operate on the exact same model, the teacher will spend more time on data fragments and run different kinds of data inference (e.g., multi-scale inference on high-resolution input data), thereby generating improvements that can be implemented for the student version in that local environment.

[0060] The teacher version can use the same machine learning model and the same type of neural network, or, for example, a larger model. For instance, in person detection from 2D images, different network architectures exist, including large and small networks. The teacher version can use larger, more complex networks because these are generally more accurate, but such networks are impractical for real-time or near-real-time use in the student version.

[0061] Figure 5 An example workflow 500 is illustrated for deploying and retraining a local environment-specific machine learning model over time. A teacher version of the machine learning model 505 can operate in a remote environment 510. The teacher version 505 can deploy a student version(t) of the machine learning model 515 to a local environment 520. The student version 515 can be a generic model. For example, the same student version can be deployed to multiple local environments.

[0062] The teacher version can be a deep, multi-layered neural network that can generate pseudo-labels for data snippets from a wide variety of local environments. Each student version 515 can have a much narrower, more specific environment in which it operates in real-time or near real-time. The student version 515 is inferring from data snippets, typically continuously, possibly multiple snippets per second. However, the student version 515 does not receive the labels for these data snippets and therefore does not know in real-time whether the inference is correct.

[0063] Such as about Figure 2 The described method can compile a first data segment (t) 525, characterized by a subset of data segments evaluated by the student version 515 and associated programming tags, as well as pseudo-tags applied to those data segments by the teacher version 505. A comparison between the programming tags and the pseudo-tags can be used as the basis for the first training update (t+1) 530.

[0064] Real, traditional, authentic tags are generated by humans. These tags can be used when training the teacher version 505. In this workflow, pseudo-tags are generated by the teacher version 505 in the remote environment 510. The pseudo-tags may not be perfectly accurate, but should be more accurate than the programming tags generated by the student version 515.

[0065] Then, the first training update (t+1) can be based on details of the environment: shadow and lighting patterns, background noise, user's language, dialect and accent, user-specific identity, etc. Figure 1 In the example, there may be multiple surveillance cameras inside the store, all pointing at the parking lot, but each store's parking lot has its own characteristics and challenges. These surveillance cameras are typically placed in static locations. If a generic people detection model is deployed to the store, the initial student version 515 of the model may miss people when only given generic training.

[0066] For 2D image detection and classification, the student version 515 typically operates online at speeds on the order of 15 frames per second. The teacher version 505 can operate offline, and therefore may take longer to perform inference on each frame in order to generate pseudo-tags that are more accurate than the programmed tags output by the student version 515.

[0067] Once the system is operating in the local environment 520, the student model 515 is trained at those camera locations. This may require time-dependent training (e.g., changes in light and shadow that occur throughout the day). The first training update 530 can also allow the model to use information from multiple camera angles to arrive at a single conclusion or label. By comparing the best guesses of the student version 515 and the teacher version 505, the analysis of environmental characteristics by the student version 515 can be improved. For example, if a tree in a parking lot casts a predictable shadow at a certain time of day, the people detection task can take this shadow into account and adjust the weights of the people detection parameters to compensate for variations in detectable colorization.

[0068] The first training update 530 is applied to the student version 515 to generate the second student version MLM(t+1) 535. Training updates can be frequent at deployment time or shortly thereafter. For example, for continuous 2D image capture, training updates might initially be released approximately weekly. As the student version is retrained to the local environment, training updates may become less frequent. If the student version labels data segments within the threshold accuracy of the teacher version, retraining may not be necessary. If the student version's performance fails to meet the threshold, a second training update may be released.

[0069] Inaccuracies can be checked over time to form data sections with sufficient information to generate training updates. For example, rare event detection can use a large number of instances of the rare event to generate patterns that can be used to generate training updates.

[0070] Version 535 can generate enough labeled data to form the second data section (t+1) 540, which can be used as the basis for the second training update (t+2) 545. The second training update (t+2) 545 can be applied to student version 535 to generate student version machine learning model (t+2) 550. This process can occur iteratively, whereby the data section is compiled, and then the student model is updated and tuned for the local environment 520.

[0071] If the teacher version 505 performs more accurate labeling than the student version, the student version will automatically improve during retraining without requiring manual labeling of data segments. Compared to time-consuming manual training, which typically has a lower retraining update rate, this system allows for inexpensive retraining, which can be achieved simply by identifying threshold differences in labeling at the (data) section compiler.

[0072] In some embodiments, the methods and processes described herein can be associated with a computing system of one or more computing devices. In particular, these methods and processes can be implemented as computer applications or services, application programming interfaces (APIs), libraries, and / or other computer program products.

[0073] Figure 6 A non-limiting embodiment of a computing system 600 is illustrated schematically, which may employ one or more of the methods and processes described above. The computing system 600 is shown in a simplified form. The computing system 600 may be embodied in the above description and... Figure 2 The computer device 200 shown in the figure. The computing system 600 may take the form of one or more personal computers, server computers, tablet computers, home entertainment computers, network computing devices, gaming devices, mobile computing devices, mobile communication devices (e.g., smartphones) and / or other computing devices, as well as wearable computing devices (such as smartwatches and head-mounted augmented reality devices).

[0074] The computing system 600 includes a logic processor 602, volatile memory 604, and non-volatile storage device 606. Optionally, the computing system 600 may include a display subsystem 608, an input subsystem 610, a communication subsystem 612, and / or Figure 6 Other components not shown.

[0075] The logic processor 602 includes one or more physical devices configured to execute instructions. For example, the logic processor may be configured to execute instructions that are part of one or more applications, programs, routines, libraries, objects, components, data structures, or other logical constructs. These instructions may be implemented to perform tasks, implement data types, transition the state of one or more components, achieve technical effects, or otherwise achieve desired results.

[0076] A logic processor may include one or more physical processors (hardware) configured to execute software instructions. Alternatively, a logic processor may include one or more hardware logic circuits or firmware devices configured to execute hardware-implemented logic or firmware instructions. The processor of logic processor 602 may be single-core or multi-core, and the instructions executed thereon may be configured for sequential, parallel, and / or distributed processing. Optionally, the various components of the logic processor may be distributed across two or more separate devices, which may be located remotely and / or configured for coordinated processing. Aspects of the logic processor may be virtualized and executed by remotely accessible, networked computing devices configured for cloud computing configurations. In this context, it will be understood that these virtualized aspects run on different physical logic processors on various different machines.

[0077] The non-volatile storage device 606 includes one or more physical devices configured to store instructions executable by a logic processor to implement the methods and processes described herein. When these methods and processes are implemented, the state of the non-volatile storage device 606 can be transformed—for example, to store different data.

[0078] Non-volatile storage device 606 may include removable and / or built-in physical devices. Non-volatile storage device 606 may include optical memory (e.g., CD, DVD, HD-DVD, Blu-ray Disc, etc.), semiconductor memory (e.g., ROM, EPROM, EEPROM, FLASH memory, etc.), and / or magnetic memory (e.g., hard disk drive, floppy disk drive, tape drive, MRAM, etc.) or other high-capacity storage device technologies. Non-volatile storage device 606 may include non-volatile, dynamic, static, read / write, read-only, sequential access, location-addressable, file-addressable, and / or content-addressable devices. It should be understood that non-volatile storage device 606 is configured to retain instructions even when power to it is cut off.

[0079] Volatile memory 604 may include a physical device containing random access memory. Volatile memory 604 is typically used by logic processor 602 to temporarily store information during the processing of software instructions. It should be understood that when power to volatile memory 604 is cut off, volatile memory 604 typically does not continue storing instructions.

[0080] Various aspects of the logic processor 602, volatile memory 604, and non-volatile storage device 606 can be integrated together into one or more hardware logic components. For example, such hardware logic components may include field-programmable gate arrays (FPGAs), application-specific integrated circuits (PASICs / ASICs), application-specific standard products (PSSPs / ASSPs), system-on-a-chip (SOCs), and complex programmable logic devices (CPLDs).

[0081] The terms "module," "program," and "engine" can be used to describe an aspect of computing system 600, typically implemented in software by a processor to perform specific functions using portions of volatile memory. These functions involve specially configured processors to perform translational processing. Thus, a module, program, or engine can be instantiated via a portion of volatile memory 602, which executes instructions stored in non-volatile storage device 606, using portions of volatile memory 604. It should be understood that different modules, programs, and / or engines can be instantiated from the same applications, services, code blocks, objects, libraries, routines, APIs, functions, etc. Similarly, the same module, program, and / or engine can be instantiated from different applications, services, code blocks, objects, routines, APIs, functions, etc. The terms "module," "program," and "engine" can encompass individuals or groups of executable files, data files, libraries, drivers, scripts, database records, etc.

[0082] The logical subsystem and storage subsystem can collaborate to instantiate one or more logical machines. As used herein, the term "machine" is used collectively to refer to the combination of hardware, firmware, software, instructions, and / or any other components that collaborate to provide the functionality of a computer. In other words, "machine" is never an abstract concept but always has a concrete form. A machine can be instantiated by a single computing device, or a machine can include two or more sub-components instantiated by two or more different computing devices. In some implementations, a machine includes local components (e.g., software applications executed by a computer processor) that collaborate with remote components (e.g., cloud computing services provided by a network of server computers). Optionally, the software and / or other instructions that give a particular machine its functionality can be stored as one or more unexecuted modules on one or more suitable storage devices.

[0083] The machine can be implemented using any suitable combination of existing and / or future machine learning (ML), artificial intelligence (AI), and / or natural language processing (NLP) technologies. Non-limiting examples of techniques that can be incorporated into implementations of one or more machines include support vector machines, multilayer neural networks, convolutional neural networks (e.g., including spatial convolutional networks for processing images and / or videos, temporal convolutional neural networks for processing audio signals and / or natural language sentences, and / or any other suitable convolutional neural networks configured to convolve and pool features in one or more temporal and / or spatial dimensions), recurrent neural networks (e.g., long short-term memory networks), associative memories (e.g., lookup tables, hash tables, Bloom filters, neural Turing machines, and / or neural random access memories), word embedding models (e.g., GloVe or Word2Vec), unsupervised spatial and / or clustering methods (e.g., nearest neighbor algorithms, topological data analysis, and / or k-means clustering), graphical models (e.g., (hidden) Markov models, Markov random fields, (hidden) conditional random fields, and / or AI knowledge bases), and / or natural language processing techniques (e.g., tokenization, stemming, region selection and / or dependency parsing and / or intent recognition, segmentation models and / or super-segmentation models (e.g., hidden dynamic models)).

[0084] In some examples, the methods and processes described herein can be implemented using one or more differentiable functions, wherein the gradient of the differentiable function can be computed and / or estimated with respect to the input and / or output of the differentiable function (e.g., with respect to training data and / or with respect to a target function). Such methods and processes can be determined at least in part by a set of trainable parameters. Therefore, the trainable parameters for a particular method or process can be tuned by any suitable training procedure to continuously improve the functionality of the method or process.

[0085] Non-limiting examples of training procedures for tuning trainable parameters include supervised training (e.g., using gradient descent or any other suitable optimization method), zero-shot, few-shot, unsupervised learning methods (e.g., classification based on categories derived from unsupervised clustering methods), reinforcement learning (e.g., feedback-based deep Q-learning) and / or generative adversarial neural network training methods, belief propagation, RANSAC (random sample consensus), contextual gambling machine methods, maximum likelihood methods, and / or expectation maximization. In some examples, multiple methods, procedures, and / or components of the system described herein can be trained simultaneously with respect to an objective function that measures the performance of the collective function of multiple components (e.g., with respect to reinforcement feedback and / or with respect to labeled training data). Simultaneous training of multiple methods, procedures, and / or components can improve this collective function. In some examples, one or more methods, procedures, and / or components can be trained independently of other components (e.g., offline training on historical data).

[0086] Language models can use lexical features to guide the sampling / search of words for speech recognition. For example, a language model can be defined at least in part by the statistical distribution of words or other lexical features. For instance, a language model can be defined by the statistical distribution of n-grams, defining the transition probabilities between candidate words based on lexical statistics. Language models can also be based on any other suitable statistical features and / or by processing the results of those statistical features using one or more machine learning and / or statistical algorithms (e.g., confidence values ​​generated by such processing). In some examples, such as based on the assumption that the words in the audio signal come from a specific vocabulary, a statistical model can constrain which words can be recognized for an audio signal.

[0087] Alternatively or additionally, the language model can be based on one or more neural networks previously trained to represent audio input and words in a shared latent space, for example, a vector space learned by one or more audio and / or word models (e.g., wav2letter and / or word2vec). Therefore, finding candidate words can include searching the shared latent space based on vectors encoded by the audio model for the audio input to find candidate word vectors for decoding with the word model. The shared latent space can be used to evaluate the confidence of a candidate word's features in the speech audio for one or more candidate words.

[0088] Language models can be used in conjunction with acoustic models configured to evaluate the confidence of candidate words included in the speech audio within the audio signal, based on the acoustic features of the words (e.g., Mel-frequency cepstral coefficients, formants, etc.). Optionally, in some examples, language models can be incorporated into acoustic models (e.g., the evaluation and / or training of the language model can be based on the acoustic model). For example, based on labeled speech audio, the acoustic model defines a mapping between acoustic signals and basic sound units such as phonemes. The acoustic model can be any suitable combination of state-of-the-art or future machine learning (ML) and / or artificial intelligence (AI) models, such as: deep neural networks (e.g., long short-term memory, temporal convolutional neural networks, restricted Boltzmann machines, deep belief networks), hidden Markov models (HMMs), conditional random fields (CRFs) and / or Markov random fields, Gaussian mixture models and / or other graphical models (e.g., deep Bayesian networks). The audio signal to be processed by the acoustic model can be preprocessed in any suitable manner (e.g., encoded with any suitable sampling rate, Fourier transform, bandpass filter, etc.). The acoustic model can be trained to recognize the mapping between acoustic signals and sound units based on labeled audio data. For example, the acoustic model can be trained on labeled audio data including speech audio and corrected text to learn the mapping between speech audio signals and sound units labeled by the corrected text. Therefore, the acoustic model can be continuously improved to enhance its practicality in accurately recognizing speech audio.

[0089] In some examples, language models can be incorporated into any suitable graphical model, such as a Hidden Markov Model (HMM) or a Conditional Random Field (CRF), in addition to statistical models, neural networks, and / or acoustic models. Considering the speech audio and / or other words identified so far, the graphical model can use statistical features (e.g., transition probabilities) and / or confidence values ​​to determine the probability of recognizing a word. Therefore, the graphical model can use statistical features, previously trained machine learning models, and / or acoustic models to define the transition probabilities between states represented in the graphical model.

[0090] When the methods and processes described herein are incorporated into ML and / or AI components, the ML and / or AI components can make decisions based at least in part on training the components on training data. Therefore, ML and / or AI components can and should be trained on diverse, representative datasets that include sufficient relevant data for diverse users and / or user groups. In particular, the training dataset should contain information about different human individuals and groups so that, when training the ML and / or AI components, their performance in terms of user experience for users and / or user groups is improved.

[0091] ML and / or AI components can be additionally trained to make decisions in order to minimize potential biases toward individuals and / or groups of people. For example, when AI systems are used to evaluate any qualitative and / or quantitative information about individuals or groups of people, they can be trained to remain invariant to differences between individuals or groups that are not intended to be measured through qualitative and / or quantitative evaluations, such that any decision is not unintentionally influenced by differences between individuals and groups.

[0092] When the methods and processes described herein are incorporated into ML and / or AI components, the ML and / or AI components can make decisions based at least in part on training the components on training data. Therefore, ML and / or AI components can and should be trained on diverse, representative datasets that include sufficient relevant data for diverse users and / or user groups. In particular, the training dataset should contain information about different human individuals and groups so that, when training the ML and / or AI components, their performance in terms of user experience for users and / or user groups is improved.

[0093] ML and / or AI components can be additionally trained to make decisions in order to minimize potential biases toward individuals and / or groups of people. For example, when AI systems are used to evaluate any qualitative and / or quantitative information about individuals or groups of people, they can be trained to remain invariant to differences between individuals or groups that are not intended to be measured through qualitative and / or quantitative evaluations, such that any decision is not unintentionally influenced by differences between individuals and groups.

[0094] ML and / or AI components can be designed to provide context about how they operate, allowing implementers of ML and / or AI systems to be accountable for the decisions / evaluations made by the system. For example, ML and / or AI systems can be configured for reproducible behavior; for instance, when they make pseudo-random decisions, a random seed can be used and recorded to enable later reproduction of that decision. As another example, data used to train and / or test ML and / or AI systems can be curated and maintained to facilitate future investigations into the behavior of ML and / or AI systems with respect to data. Furthermore, ML and / or AI systems can be continuously monitored to identify potential biases, errors, and / or unexpected results.

[0095] When included, the display subsystem 608 can be used to present a visual representation of the data stored by the non-volatile storage device 606. This visual representation may take the form of a graphical user interface (GUI). When the methods and processes described herein change the data stored by the non-volatile storage device and thus change the state of the non-volatile storage device, the state of the display subsystem 608 can also be changed to visually represent the changes in the underlying data. The display subsystem 608 may include one or more display devices that use any type of technology virtually. Such display devices may be combined with the logic processor 602, the volatile memory 604, and / or the non-volatile storage device 606 in a shared housing, or such display devices may be peripheral display devices.

[0096] When included, the input subsystem 610 may include or be coupled to one or more user input devices, such as a keyboard, mouse, touchscreen, or game controller. In some embodiments, the input subsystem may include or be coupled to a selected Natural User Input (NUI) element portion. Such element portions may be integrated or peripheral, and the translation and / or processing of input actions may be processed on-board or off-board. Example NUI element portions may include: a microphone for speech and / or voice recognition; an infrared, color, stereo, and / or depth camera for machine vision and / or gesture recognition; a head tracker, eye tracker, accelerometer, and / or gyroscope for motion detection and / or intent recognition; an electric field sensing element portion for assessing brain activity; and / or any other suitable sensor.

[0097] When included, the communication subsystem 612 can be configured to communicatively couple the various computing devices described herein to each other and to other devices. The communication subsystem 612 may include wired and / or wireless communication devices compatible with one or more different communication protocols. As a non-limiting example, the communication subsystem may be configured to communicate via a wireless telephone network, or a wired or wireless local area network or wide area network (such as an HDMI connection via Wi-Fi). In some embodiments, the communication subsystem may allow the computing system 600 to send messages to and / or receive messages from other devices via a network such as the Internet.

[0098] In one example, a method for environment-specific training of a machine learning model includes: receiving a data stream comprising multiple sequential data segments for a local environment; generating programming tags for each data segment using a student version of the machine learning model; selecting a portion of the data segments for evaluation by a teacher version of the machine learning model; uploading the selected portion of the data segments and the associated programming tags to a server-side computing device including the teacher version of the machine learning model; receiving environment-specific training updates from the server-side computing device, the environment-specific training updates being based at least on a comparison of the programming tags associated with the selected portion of the data segments with pseudo-tags generated by the teacher version of the machine learning model for the selected portion of the data segments; applying the environment-specific training updates to the student version of the machine learning model to generate an updated student version of the machine learning model; generating programming tags for newly received data segments using the updated student version of the machine learning model; and providing at least some of the newly received data segments and the generated programming tags to a local output for use in evaluating events within the local environment. In this example or any other example, the method additionally or alternatively includes: performing an update procedure that includes selecting a portion of newly received data fragments and uploading the selected portion of the newly received data fragments to a server-side computing device for evaluation by a teacher-side machine learning model; receiving new environment-specific training updates from the server-side computing device; and applying the new environment-specific training updates to generate an updated student-side machine learning model. In any of the foregoing examples or any other example, the environment-specific training updates additionally or alternatively include one or more selection parameters that can be used to determine a portion of the newly received data fragments to be selected for evaluation by the teacher-side machine learning model. In any of the foregoing examples or any other example, one or more selection parameters included in the training update additionally or alternatively include a selection frequency. In any of the foregoing examples or any other example, additionally or alternatively, the portion of data fragments selected for evaluation is randomly selected. In any of the foregoing examples or any other example, additionally or alternatively, the portion of data fragments selected for evaluation is selected at least in part based on programmed tags. In any of the foregoing examples or any other example, additionally or alternatively, the portion of data fragments selected for evaluation is selected at least in part based on confidence levels assigned to one or more programmed tags.

[0099] In another example, a system for deploying a machine learning model in a local environment includes: a local data interface configured to receive a data stream comprising multiple sequential data segments for a local environment; a student-version machine learning model configured to generate programming tags for each data segment; a selector configured to select a portion of the data segment for evaluation by a teacher-version machine learning model; and a client-side network interface configured to upload the selected portion of the data segment and its associated programming tags to a server-side computing device and receive environment-specific training updates from the server-side computing device, the environment-specific training updates being based at least on a comparison of the programming tags associated with the selected portion of the data segment with pseudo-tags generated by the teacher-version machine learning model for the selected portion of the data segment. In this example or any other example, the client-side computing device also additionally or alternatively includes one or more sensors configured to generate the data stream within the local environment. In any or any other example of the foregoing examples, the data stream additionally or alternatively includes one or more 2D image streams, and wherein each data segment includes one or more image frames. In any or any other of the foregoing examples, the data stream additionally or alternatively includes one or more 3D image streams, wherein each data segment includes one or more image frames. In any or any other of the foregoing examples, the data stream additionally or alternatively includes one or more audio streams, wherein each data segment includes finite chunks of audio data. In any or any other of the foregoing examples, the student version of the machine learning model is additionally or alternatively configured to provide at least some of sequential data segments and associated programming tags to the local output.

[0100] In yet another example, a method for retraining a machine learning model for a local environment includes: receiving data segments and associated programming tags from a client-side computing device employing a student version of the machine learning model; generating pseudo-tags for each received data segment using a teacher version of the machine learning model; comparing the pseudo-tags and programming tags for each received data segment; compiling two or more performed comparisons into a data section; generating an environment-specific training update for the student version of the machine learning model based on the data section; and publishing the generated environment-specific training update to the client-side computing device. In this example or any other example, generating the environment-specific training update is additionally or alternatively based on the comparison of pseudo-tags and programming tags exhibiting a difference greater than a threshold. In any or any other example of the foregoing examples, the teacher version of the machine learning model is additionally or alternatively initially trained at least partially under human supervision. In any or any other example of the foregoing examples, the method additionally or alternatively includes iteratively training the student model of the machine learning model, the iterative training including: receiving new data segments and associated programming tags from the client-side device, and generating new environment-specific training updates for the student version of the machine learning model based at least on the pseudo-tags generated for the received new data segments. In any or any of the foregoing examples, the method additionally or alternatively includes receiving data fragments and associated programming tags from two or more client-side computing devices operating in different local environments. In any or any of the foregoing examples, the method additionally or alternatively includes generating different environment-specific training updates for each client-side computing device, based at least on the data fragments and associated programming tags received for that client-side computing device. In any or any of the foregoing examples, at least some aspects of each different environment-specific training update are additionally or alternatively based on data fragments and associated programming tags received from one or more other client-side computing devices.

[0101] It should be understood that the configurations and / or methods described herein are exemplary in nature, and these specific embodiments or examples should not be considered limiting, as many variations are possible. The specific routines or methods described herein may represent one or more of any number of processing strategies. Thus, the various actions illustrated and / or described may be performed in the illustrated and / or described order, in another order, in parallel, or omitted. Similarly, the order of the above processes may be changed.

[0102] This disclosure includes all novel and non-obvious combinations and sub-combinations of the various processes, systems and configurations disclosed herein, as well as any and all their equivalents.

Claims

1. A method for environment-specific training of a machine learning model, the method comprising: Receive a data stream for a local environment, comprising multiple sequential data segments, wherein the data stream includes one or more 2D or 3D image streams, and wherein each data segment includes one or more image frames; Using a client-side computing device located in the local environment, the client-side computing device is configured to run a student version of a machine learning model to generate programming tags for each data fragment; A subset of data segments is selected for evaluation by the teacher-based machine learning model based on a threshold confidence level for the programmed tags and / or based on the tag ambiguity for the programmed tags. Selected portions of the data fragment and associated programming tags are uploaded to a server-side computing device that includes the teacher-version machine learning model. Receives environment-specific training updates from the server-side computing device, the environment-specific training updates being based at least on a comparison of the programming tags associated with the selected portion of the data fragment and pseudo-tags generated by the teacher version machine learning model for the selected portion of the data fragment, wherein at least some aspects of the environment-specific training updates are based on the data fragment and associated programming tags received from one or more other client-side computing devices different from the client-side computing device running the student version machine learning model; The environment-specific training update is applied to the student version of the machine learning model to generate an updated student version of the machine learning model; Using the updated student version of the machine learning model, programmatic tags are generated for the newly received data fragments; as well as At least some of the newly received data fragments and generated programming tags are provided to the local output for use in evaluating events within the local environment.

2. The method according to claim 1, further comprising: An update procedure is executed iteratively, comprising selecting a portion of a newly received data segment and uploading the selected portion of the new received data segment to the server-side computing device for evaluation by the teacher version machine learning model, receiving new environment-specific training updates from the server-side computing device, and applying the new environment-specific training updates to generate an updated student version machine learning model.

3. The method according to claim 1, wherein, The environment-specific training update includes one or more selection parameters that can be used to determine a portion of newly received data fragments for selection for evaluation by the teacher version of the machine learning model.

4. The method according to claim 3, wherein, The one or more selection parameters included in the training update include selection frequency.

5. The method according to claim 1, wherein, The portion of the data segment selected for evaluation was also randomly chosen.

6. A system for deploying machine learning models in a local environment, the system comprising: A local data interface configured to receive a data stream for the local environment comprising multiple sequential data segments, wherein the data stream includes one or more 2D or 3D image streams, and wherein each data segment includes one or more image frames; A client-side computing device, located in the local environment and configured to run a student version of a machine learning model, the student version of the machine learning model being configured to generate programming tags for each data fragment; A selector configured to select a portion of a data segment for evaluation by a teacher-based machine learning model based on a threshold confidence level for the programmed tag and / or based on the tag fuzziness for the programmed tag; as well as A client-side network interface configured to upload selected portions of a data fragment and associated programming tags to a server-side computing device, and to receive environment-specific training updates from the server-side computing device, the environment-specific training updates being based at least on a comparison of the programming tags associated with the selected portions of the data fragment and pseudo-tags generated by the teacher version machine learning model for the selected portions of the data fragment, wherein at least some aspects of the environment-specific training updates are based on the data fragment and associated programming tags received from one or more other client-side computing devices different from the client-side computing device running the student version machine learning model.

7. The system according to claim 6, wherein, The client-side computing device also includes one or more sensors configured to generate the data stream within the local environment.

8. The system according to claim 6, wherein, The data stream includes one or more audio streams, and each data segment includes a finite number of audio data blocks.

9. The system according to claim 6, wherein, The student version of the machine learning model is also configured to provide at least some of the sequential data fragments and associated programming tags to the local output.

10. A method for retraining a machine learning model for a local environment, comprising: The system receives data segments and associated programming tags included in a data stream from a client-side computing device employing a student version of a machine learning model located in the local environment, wherein the student version of the machine learning model is configured to generate programming tags for each data segment, and the data stream includes one or more 2D or 3D image streams, and wherein each data segment includes one or more image frames. Using a teacher-version machine learning model, pseudo-labels are generated for a portion of a data fragment, wherein the selected portion of the data fragment is selected based on a threshold confidence level for the programmed label and / or based on the label fuzziness for the programmed label; The pseudo-tags and programming tags associated with the portions of the data fragment are compared; Compile two or more performed comparisons into a data section; Based on the data segment, an environment-specific training update is generated for the student version of the machine learning model, wherein at least some aspects of the environment-specific training update are based on the data segment and associated programming tags received from one or more other client-side computing devices different from the client-side computing device running the student version of the machine learning model. as well as The generated environment-specific training updates are published to the client-side computing device.

11. The method according to claim 10, wherein, The environment-specific training update is based on a comparison between the pseudo-label and the programmed label, which shows a difference greater than a threshold.

12. The method according to claim 10, wherein, The teacher-trained machine learning model was initially trained, at least in part, under human supervision.

13. The method of claim 10, further comprising: The student version of the machine learning model is trained iteratively, the iterative training comprising: receiving new data fragments and associated programming tags from the client-side computing device, and generating new environment-specific training updates for the student version of the machine learning model based at least on pseudo-tags generated for the received new data fragments.

14. The method of claim 10, further comprising: Receive data fragments and associated programming tags from two or more client-side computing devices operating in different local environments.

15. The method of claim 14, further comprising: Different environment-specific training updates are generated for each client-side computing device, based at least on the data fragments received for the client-side computing device and the associated programming tags.

Images