Anomaly Detection Using Feedback Training

Abstract

Techniques for anomaly detection are described. An exemplary method includes receiving one or more requests to train an anomaly detection machine learning model using feedback-based training, the requests indicating one or more of a type of analysis to be performed, a model selection indication, and a configuration of a training data set; using the training data to train the anomaly detection machine learning model according to the one or more requests; performing feedback-based training on the trained anomaly detection machine learning model; and retraining the anomaly detection machine learning model using the training data.

Description

Background Art

[0001] Figuring out whether a part has been manufactured correctly or if a machine component is broken is critical to quality. Today, making this determination typically requires manually inspecting objects like machine parts, equipment, and production lines, either physically or in photographs, to spot any deviations (i.e., anomalies) from what is considered "good." Manual inspections are slow and often miss errors due to inconsistent judgment, limited experience and training, labor availability and cost, and even factors like poor eyesight and distraction while working. In the manufacturing industry alone, production line downtime due to overlooked errors results in millions of dollars in cost overruns and lost revenue each year. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Various embodiments according to the present disclosure will be described with reference to the accompanying drawings, in which:

[0003] Figure 1 An implementation of an anomaly detection service or component is shown.

[0004] Figure 2 A process flow diagram is shown for performing inference using anomaly detection services / components according to some embodiments.

[0005] Figure 3 A process flow diagram is shown for performing training of one or more ML models using feedback, according to some embodiments.

[0006] Figure 4 An implementation of training of a model graph according to some embodiments is shown.

[0007] Figure 5 An embodiment of a GUI for creating an anomaly detection project is shown.

[0008] Figure 6 Shown is an embodiment of a GUI for creating a data set.

[0009] Figure 7 An embodiment of a GUI for manipulating a data set is shown.

[0010] Figure 8 An embodiment of a GUI for manipulating a data set is shown.

[0011] Figure 9 An embodiment of a GUI for manipulating a data set is shown.

[0012] Figure 10 An embodiment of a GUI for manipulating a data set is shown.

[0013] Figure 11 An embodiment of a GUI showing details of a trained model is shown.

[0014] Figure 12 An embodiment of a GUI for configuring the use of a model to perform anomaly detection is shown.

[0015] Figure 13 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown.

[0016] Figure 14 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown.

[0017] Figure 15 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown.

[0018] Figure 16 An embodiment of a GUI for performing more detailed annotation of an image is shown.

[0019] Figure 17 An embodiment of a GUI for performing more detailed annotation of an image is shown.

[0020] Figure 18 An embodiment of a GUI for performing annotation of a verification image is shown.

[0021] Figure 19 An embodiment of a GUI illustrating a dashboard for an anomaly detection service is shown.

[0022] Figure 20 is a flow diagram illustrating the operation of a method for utilizing anomaly detection services according to some embodiments.

[0023] Figure 21 is a flow diagram illustrating the operations of a method for creating a dataset to be used for training and / or testing anomaly detection services, according to some embodiments.

[0024] Figure 22 is a flow diagram illustrating the operation of a method for creating at least a subset of a training data set according to some embodiments.

[0025] Figure 23 An embodiment of a graphical user interface for creating a dataset from an image is shown.

[0026] Figure 24 is a flow chart illustrating the operation of a method for performing feedback training according to some embodiments.

[0027] Figure 25 Embodiments of components for feedback training and components for post-inference feedback training are shown.

[0028] Figure 26is a flow diagram illustrating the operation of a method for performing inference in an anomaly detection service according to some embodiments.

[0029] Figure 27 An exemplary provider network environment is shown according to some embodiments.

[0030] Figure 29 is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers according to some embodiments.

[0031] Figure 29 is a block diagram illustrating an exemplary computer system that may be used in some embodiments.

[0032] Figure 30 is a block diagram of an illustrative operating environment for training and hosting machine learning models according to some embodiments. DETAILED DESCRIPTION

[0033] The present disclosure relates to methods, apparatus, systems, and non-transitory computer-readable storage media for performing anomaly detection, describing feedback-based training and / or feedback-based anomaly detection. In some embodiments, one or more cameras are installed at a remote location (such as a production line), and the images they capture and their metadata (such as timestamps and camera identifiers) are sent to a supplier network for anomaly detection.

[0034] Figuring out whether a part has been manufactured correctly or if a machine component is broken is critical to quality. Today, making this determination typically requires manually inspecting objects like machine parts, equipment, and production lines, either physically or in photographs, to spot any deviations (i.e., anomalies) from what is considered "good." Manual inspections are slow and often miss errors due to inconsistent judgment, limited experience and training, labor availability and cost, and even factors like poor eyesight and distraction while working. In the manufacturing industry alone, production line downtime due to overlooked errors results in millions of dollars in cost overruns and lost revenue each year.

[0035] To overcome the challenges of manual inspection, some companies have applied automated image analysis (a technology known as machine vision) to detect anomalies. These expensive, purpose-built camera and computer systems must be calibrated for lighting and viewing angle and use hard-coded rules about what is and is not a defect to support a single, specialized task and environment. While useful, these systems are difficult and expensive to maintain. Humans have an innate flexibility that allows us to understand that differences in viewing angle or lighting are not differences in the object itself. However, machine vision systems cannot adapt to new information and environmental changes like humans do. Therefore, over time, the rule engines become a tangled web of hard-coded exceptions, and even minor changes in the environment require constant recalibration.

[0036] This document details embodiments of methods, etc., for detecting image differences of objects. As detailed, a simple API (or GUI) can be used to generate a set of baseline "good" images of one or more objects of interest, and feedback about the model trained from those good images can be used to improve the original model. In some embodiments, the training data can include images with defects that are marked at the location of the defect they want to identify (e.g., the location of a stain on a tank) and / or at designated areas where defects are expected (such as the expected location of a machine part). A custom machine learning (ML) model is generated to compare subsequent images to the baseline and, through this comparison, identify additional defects (e.g., more stains on a tank or missing machine parts). Additionally, feedback on the ML model's predictions (e.g., images that were incorrectly marked as anomalies, or marked areas in an image where no defects were identified) can be used to improve the model's accuracy, thereby learning from human expertise and getting better over time. Note that the images can be 2-D, 3-D, color, black and white, grayscale, etc.

[0037] Figure 1 An embodiment of an anomaly detection service or component is shown. Specifically, Figure 1 An anomaly detection service (e.g., provided as part of a provider network 100) or component 102 (e.g., provided as one or more software components external to the provider network (such as edge servers, edge devices, etc.), but generally referred to as computing system 101) and entities that can communicate with the anomaly detection service / component 102 are shown.

[0038] In some embodiments, the anomaly detection service / component 102 uses one or more instances of anomaly detection machine learning (ML) models 103 to analyze the image t (either as a whole or in chunks thereof) for potential anomalies. In some embodiments, a model graph structure is used to configure anomaly detection. Examples of models 103 used by the anomaly detection service / component 102 may include, but are not limited to, models for performing anomaly detection, object detection, and / or alignment (orientation) detection. Note that alignment can be in 2-D or 3-D form.

[0039] Exemplary use cases include, but are not limited to, detecting texture anomalies (examples include analysis of wood finishes, flooring, wall coatings, and fabrics), finding differences (e.g., a given test image is then compared to a reference set, and the system should respond with a pixel-level (soft) binary mask indicating anomalous regions or with an image-level anomaly score), relative placement (e.g., given a test image, the system responds with a homography that maps each part to its canonical position), object analysis (e.g., where each item needs to meet some normality criteria specified with the aid of a large reference set of items, and anomalies may include shape, color, and / or texture anomalies), and / or manifest verification (e.g., to ensure that all items in a box are specified in a product parts list). Thus, at least the detection of visual defects, alignment defects, and placement defects is possible.

[0040] Local object analysis can produce a pixel-level anomaly score mask for each object analyzed. In some embodiments, the user is required to map the pixel-level anomaly score mask produced by this method to an image-level anomaly score, and this is supported by a user-provided object mask drawn on a reference object, which is transferred to the test object by aligning the test object with the reference sample. The image-level anomaly score can then be obtained by summing the pixel-level anomaly scores supported by the object mask. Global object analysis can produce an image-level anomaly score for each object analyzed. Texture analysis can produce a pixel-level mask of anomalies, which can similarly be aggregated into global or local anomaly scores. Relative placement and shape analysis can produce an output that is a homography H for each object part i. i List verification produces per-object counts and supports high-speed requirements with potentially large numbers of objects, but focuses on cases with strong priors (e.g., simple objects with low intra-class variability, nearly fixed scale, and required invariance to very specific changes like breakage).

[0041] The ML model is typically trained using a feedback training module 109, described in detail below. Feedback allows the user to provide corrections to predictions, etc. The ML model, the images on which inference is to be performed, one or more training datasets, and / or one or more test datasets are stored in a storage device 111.

[0042] The console 106 provides one or more graphical user interfaces (GUIs) to assist users in configuring the anomaly detection service / component 102 , performing anomaly detection, training one or more anomaly detection ML models 103 , and the like.

[0043] Front end / coordinator 105 receives input from users (eg, requesters 143, feedback devices 145, etc.) and / or image / video capture devices 141 and provides the input to the appropriate components. Additionally, front end / coordinator 105 coordinates the various methods detailed herein.

[0044] Image / video capture device 141 captures still images or video images for analysis of anomalies. Exemplary image / video capture devices include, but are not limited to, still cameras, video cameras, smartphones, x-ray machines, scanning electron microscopes, infrared cameras (shortwave, mediumwave, and longwave), thermal imaging cameras, and the like.

[0045] The provider network 100 (or "cloud" provider network) provides users with the ability to utilize one or more of various types of computing-related resources, such as computing resources (e.g., executing virtual machine (VM) instances and / or containers, executing batch jobs, executing code without provisioning servers), data / storage resources (e.g., object storage, block-level storage, data archive storage, databases and database tables, etc.), network-related resources (e.g., configuring virtual networks (including multiple groups of computing resources), content delivery networks (CDNs), domain name services (DNS)), application resources (e.g., databases, application build / deployment services), access policies or roles, identity policies or roles, machine images, routers, and other data processing resources. These and other computing resources may be provided as services, such as hardware virtualization services that can execute computing instances, storage services that can store data objects, etc. Users (or "customers") of the provider network 100 may utilize one or more user accounts associated with a customer account, but these terms may be used somewhat interchangeably depending on the context of use. A user may interact with the provider network 100 across one or more intermediate networks 121 (e.g., the Internet) via one or more interfaces, such as by using application programming interface (API) calls, via a console implemented as a website or application, and the like. An API refers to an interface and / or communication protocol between a client and a server such that if a client issues a request in a predefined format, the client should receive a response in a specific format or initiate a defined action. In the context of a cloud provider network, an API provides a gateway for customers to access the cloud infrastructure by allowing them to obtain data from the cloud provider network or cause actions within the cloud provider network, thereby enabling the development of applications that interact with resources and services hosted in the cloud provider network. The API may also enable different services of the cloud provider network to exchange data with each other. One or more interfaces may be part of or act as a front end to a control plane for the provider network 100, which includes "back-end" services that support and implement services that may be provided more directly to customers.

[0046] For example, a cloud provider network (or just "the cloud") generally refers to a large pool of accessible virtualized computing resources (such as computing, storage and networking resources, applications, and services). The cloud provides convenient, on-demand network access to a shared pool of configurable computing resources that can be programmatically provisioned and released in response to customer commands. These resources can be dynamically provisioned and reconfigured to adjust to variable loads. Thus, cloud computing can be considered applications delivered as services over a publicly accessible network (e.g., the Internet, cellular communication networks), and the hardware and software in the cloud provider's data centers that provide those services.

[0047] A cloud provider network may be formed into multiple zones, where a zone is a geographic area in which a cloud provider clusters its data centers. Each zone includes multiple (e.g., two or more) availability zones (AZs) connected to each other via a dedicated high-speed network (e.g., a fiber optic communication connection). An AZ (also called an availability domain, or simply a "zone") provides an isolated fault domain that includes one or more data center facilities that have power, networking, and cooling separate from those in another AZ. A data center refers to the physical building or cabinet that houses the servers of the cloud provider network and provides them with power and cooling. Preferably, the AZs within a zone are far enough apart from each other so that a natural disaster (or other failure-inducing event) does not affect more than one AZ simultaneously or take more than one AZ offline.

[0048] Customers can connect to the AZs of the cloud provider network via a publicly accessible network (e.g., the Internet, a cellular communication network), for example, through a transit center (TC). The TC is the primary backbone location that connects customers to the cloud provider network and can be co-located at other network provider facilities (e.g., an Internet service provider (ISP), a telecommunications provider) and securely connected (e.g., via VPN or direct connection) to the AZs. Each zone can operate two or more TCs for redundancy. The zones are connected to a global network that includes a dedicated network infrastructure (e.g., a fiber connection controlled by the cloud provider) that connects each zone to at least one other zone. The cloud provider network can deliver content from points of presence (or "POPs") outside of these zones but networked with them through edge locations and zone edge cache servers. This partitioning and geographical distribution of computing hardware enables the cloud provider network to provide customers with low-latency access to resources worldwide with a high degree of fault tolerance and stability.

[0049] To provide these and other computing resource services, the provider network 100 typically relies on virtualization technology. For example, virtualization technology can be used to provide users with the ability to control or utilize computing resources (e.g., "compute instances," such as VMs using a guest operating system (O / S), the VMs operating using a hypervisor that may or may not further operate on top of an underlying host O / S, containers that may or may not operate within the VMs, compute instances that can execute on "bare metal" hardware without an underlying hypervisor), where one or more computing resources can be implemented using a single electronic device. Thus, users can directly utilize computing resources hosted by the provider network (e.g., provided by hardware virtualization services) to perform various computing tasks. Additionally or alternatively, users can indirectly utilize computing resources by submitting code for execution by the provider network (e.g., via an on-demand code execution service), which in turn utilizes one or more computing resources to execute the code, typically without the user having any control or knowledge of the underlying compute instance or instances involved.

[0050] For example, in various embodiments, a "serverless" function may comprise code that is executable on demand and provided by a user or other entity (such as the provider network itself). The serverless function may be maintained within the provider network 100 by an on-demand code execution service and may be associated with a specific user or account or generally accessible to multiple users / accounts. The serverless function may be associated with a uniform resource locator (URL), uniform resource identifier (URI), or other reference that can be used to invoke the serverless function. The serverless function may be executed by a computing resource (such as a virtual machine, container, etc.) when triggered or invoked. In some embodiments, the serverless function may be invoked via an application programming interface (API) call or a specially formatted hypertext transfer protocol (HTTP) request message. Thus, a user can define a serverless function that can be executed on demand without requiring the user to maintain dedicated infrastructure to execute the serverless function. Alternatively, the serverless function may be executed on demand using resources maintained by the provider network 100. In some embodiments, these resources may be maintained in a "ready" state (e.g., with a pre-initialized runtime environment configured to execute the serverless function), thereby allowing the serverless function to be executed in near real time.

[0051] In some embodiments, the provider network 100 includes a model training system 120 that can be used to perform training and retraining and / or a model hosting system 140 for hosting trained (and retrained) models.

[0052] The circles with numbers therein indicate an exemplary high-level flow: At circle 1 , the requester 143 requests that the anomaly detection service / component 102 be configured to perform anomaly detection on unlabeled images from the image / video capture device 141 .

[0053] At circle 2, a training dataset and a test dataset are generated and stored in storage device 111. At circle 3, initial training of model 103 is performed using the training dataset, and unlabeled data from the test dataset is subjected to the trained model 103. At circle 4, a request is sent to feedback device 145 (e.g., a user provides feedback via this device) to evaluate the results of the trained model 103 on the unlabeled test dataset, and the anomaly detection service / component 102 receives the feedback. In some embodiments, the request is in the form of a GUI provided by console 106, which the user uses to provide input.

[0054] The responses based on the updated model 103 participate in feedback training 109 , which when deployed can perform anomaly detection on images captured by the image / video capture device 141 .

[0055] Figure 2 A process flow diagram for performing inference using anomaly detection service / component according to some embodiments is shown. This flow diagram uses aspects of anomaly detection service / component 102 in conjunction with Figure 1 However, this is for illustrative purposes only, and other implementations of the anomaly detection service / component may be used.

[0056] In this example, an image / video capture device 141 (image source) provides source data, such as an image, one or more appropriate subsets of an image (eg, cropped, extracted, etc.), to a storage location 111 or model 103 to be analyzed.

[0057] The model 103 is applied to the source data and generates results that are provided to the console 106 for presentation to the user 143. Examples of how the results may be presented are described in detail later, but may include, but are not limited to: the image evaluated, a score for the image, an indication of whether it is abnormal or not, an indication that an abnormality may exist (e.g., a mask on a potential problem), a means for indicating whether the prediction was correct (e.g., a means for validating the prediction), etc.

[0058] The user 143 may then provide feedback to the console 106 to participate in feedback retraining of the model 103 (the feedback loop indicated by the large rectangular box). The feedback is provided to the front end / coordinator 105 and / or the storage device 111 (e.g., added to the training dataset), which is then used to retrain the model 103.

[0059] Figure 3 A process flow diagram is shown for performing training of one or more ML models using feedback according to some embodiments. This flow diagram uses aspects of the anomaly detection service / component 102 in conjunction with Figure 1However, this is for illustrative purposes only, and other implementations of the anomaly detection service / component may be used.

[0060] In this example, a user creates a project through the front-end / coordinator. A project is a logical grouping of resources (e.g., images, labels, models, etc.) and operations (e.g., training, evaluation, and detection). User 143 also creates a training dataset for training model 103. Examples of how to use one or more GUIs to create this training dataset will be detailed later. The initial training dataset can include only unlabeled data, labeled data, or a mixture of labeled and unlabeled data.

[0061] Front-end / coordinator 105 begins training model 103 and stores the trained model in storage device 111 for later use. User 143 also creates a test dataset. This test dataset may include labeled and unlabeled data. User 143 requests testing using the test dataset via front-end / coordinator 105, which interfaces with model 103 to identify anomalies in the test dataset. In some embodiments, front-end / coordinator 105 is also responsible for processing received annotations / labels, generating training and / or test datasets, and the like.

[0062] The results of those anomaly detection attempts are provided to the console 106, which then requests a request similar to Figure 2 The user 143 can then provide feedback to the console 106 to participate in the feedback retraining of the model 103 (the feedback loop indicated by the large rectangular box). The feedback is provided to the front end / coordinator 105 and / or the storage device 111 (e.g., added to the training dataset), which is then used to retrain the model 103.

[0063] In some embodiments, the trained "model" consists of multiple models, where each model solves a different problem type. Example problem types may include, but are not limited to: i) object detection, ii) alignment, iii) unsupervised anomaly detection, and iv) supervised anomaly detection. Figure 4 Implementations of training of a model graph according to some implementations are shown. Note that the subsequent discussion of feedback training or initial training can be applied to the training of a single model or a collection of models.

[0064] At 401, a configuration (or selection) of a model graph is received. For example, a user may specify what models to use and in what order. In other embodiments, selections are made for the user based on input from the user, such as answers to questions about one or more objects. In some embodiments, for local object analysis, the model graph includes one or more of an object detection model, an alignment model, and an anomaly detection model. In some embodiments, for global object analysis, the model graph includes one or more of an object detection model and an anomaly detection model. In some embodiments, for text analysis, the model graph includes an anomaly detection model. In some embodiments, for relative placement analysis, the model graph includes one or more of an object detection model and an alignment model. In some embodiments, for manifest validation, the model graph includes an object detection model. The object detection and alignment models are supervised models, and the anomaly model is (semi-) unsupervised.

[0065] At 403, feedback training of the model graph is performed. Specifically, feedback-based training of a non-anomaly detection supervised model of the model graph is performed. The details of feedback training will be described in detail later. In some embodiments, when training a given model, the user is required to provide annotations only for this model. Therefore, in some embodiments, when performing feedback training of an object detection model, the user will be shown a set of image-level views that are optimal for finding bounding box errors by default. When feedback training is performed on an alignment model, the user will instead be shown a single object with annotations by default.

[0066] At 405, one or more unsupervised anomaly models are learned. Having partially trained the model graph (excluding all anomaly modules) means that the inputs to all anomaly modules can now be calculated, and therefore, the unsupervised anomaly models can be trained. While the first training run is unsupervised, it is noted that future iterations may include partial supervision that the anomaly models will utilize through semi-supervised learning.

[0067] At 407, training is performed on the supervised anomaly detector. At 409, the supervised anomaly detector (the entire graph) is evaluated against test data and metrics are shown. The evaluation can include several actions such as set exploration, anomaly tagging (annotation), etc. Set exploration means that after having all the inference outputs (and the recorded image metadata), the user will use multiple ranking and filtering criteria to explore the set of captured images. These include (i) model-generated criteria (e.g., local or global anomaly scores, number of bounding boxes, anomalies calculated based on pose estimation), (ii) selection of images with inferences that have changed relative to a previous learning step, and / or (iii) image timestamps or camera identifiers. Anomaly tagging sets up a way to effectively explore the set during the search for anomalous samples, which allows the user to build an annotation set of anomalous and normal images that can be utilized during model graph evaluation or semi-supervised retraining. The user may also encounter false positive anomalies where an earlier model graph stage is problematic. Thus, the user can choose to expand each image to visualize all intermediate steps and provide feedback for the next iteration of the retraining phase to utilize new annotations.

[0068] At 411, the model graph is deployed.

[0069] Note that training can have different forms. In the joint model training method, the user provides feedback of all annotation types for all the images they annotate. Then they perform training iterations on all the models {M1, …, M N} in the model graph pipeline and repeat. In the individual model training method, the user instead trains each model M i . When training model M i , the user performs all the iterations of the feedback training process on this model until the user is no longer able to find any model inferences to correct.

[0070] Note that when training the model graph pipeline, feedback training is used to train each model M j in the pipeline individually, while assuming that the upstream models M i , i < j have been trained previously such that the inputs to M j derived from the entire training set can be used for feedback training. This can be important because the mining of informative examples (i.e., determining useful ranking R j ) assumes that a large pool of examples is available to mine from.

[0071] The following describes what can be used with an anomaly detection service / component such as Figure 1Multiple embodiments of GUIs for interacting with anomaly detection services / components 102 are provided. Note that user input can be used to generate specific API calls. In some embodiments, a user can use a command line interface to generate similar or identical API calls. Note that not all aspects are shown in all embodiments. Furthermore, the content and / or functionality of these GUIs can be condensed (using fewer GUIs) or expanded (using more GUIs).

[0072] Figure 5 An embodiment of a GUI for creating an anomaly detection project is shown. The user provides a project name 501, and in some embodiments, a type of project selection mechanism 503 is provided to the user. For example, a user can choose between image-level anomaly detection (indicating that the image content is generally abnormal) and pixel-level anomaly detection (indicating that a specific area or areas of the image content are abnormal). Typically, one project is completed for each assessment. For example, in an automotive production line, there may be a separate project for each step in the process (e.g., installing an engine, installing a body panel, etc.). In particular, it is often beneficial to use a model trained on data similar to the data being analyzed.

[0073] In some embodiments, during project creation, the user can also use model selection 505 to select how to use the model for anomaly detection. In this illustration, automatic selection is used. The user can provide information such as the problem to be solved, and the anomaly detection service / component 102 will determine which model to use. For example, different models can be used for local object analysis, global object analysis, texture analysis, relative placement and shape, etc. A collection of models can be used when multiple problems are to be solved. For manual selection, the user provides a model path, the model itself, an algorithm path, or the algorithm itself.

[0074] Each project is trained and / or tested using a dataset. Note that in some embodiments, the test dataset is a subset of the unlabeled training dataset. Figure 6 An embodiment of a GUI for creating a dataset is shown. In some embodiments, this GUI can be used to allow a user to select a configuration option for creating a single dataset or creating separate training and test datasets in the dataset configuration 600 portion of the GUI. In some embodiments, the single dataset will be split into separate training and test datasets by the anomaly detection service / component 102.

[0075] The GUI allows the user to provide a configuration 602 of what image source to use for the dataset. For example, the user can use the Select Import Image 603 function to select the location of the images for the dataset. Illustrative but non-limiting examples of such sources include provider network storage (e.g., Figure 1storage device 111), a location on the user's local computer, a location containing images to be copied, and imports from different sources (such as an online storage location not associated with the provider's network).

[0076] In this example, the provider network is selected as the location from which to import images. The GUI allows the user to specify a storage location QA05 (in this example, a path). Additionally, in some embodiments, the GUI allows the user to select automatic tagging of images 607. For example, if a user marks an image as having unusual content after viewing it, it is placed in an unusual folder.

[0077] In some embodiments, items in a dataset can be created from a single item. For example, a single image can be made into multiple images for testing and / or training. Option 609 allows the user to configure such use for the image at location 611. The image may have one or more of the following: detected and extracted objects, extracted meshes, extracted blocks of different sizes, cropped, geometric transformations applied (e.g., rotation, noise addition, transform compression), normalized, pre-processed, post-processed, photometrically altered, etc.

[0078] Figure 7 An embodiment of a GUI for manipulating a data set is shown. As shown, this GUI includes an indication 700 of what type of data set is being manipulated (training or testing) (if necessary). The GUI may include one or more actions that the anomaly detection service / component 102 takes in response to input in this GUI. These actions can be selected using one or more inputs, including inputs for adding a label 701, adding an image 702, deleting an image 704, and / or training a model based on one or more of the selected images 704. In this example, an image will be added to the data set being shown.

[0079] In some embodiments, the GUI includes information about the image 706, such as a description of the image (which should already be provided), text indicating how many images are labeled, how many images need to be labeled for training (if necessary), and the like.

[0080] In some embodiments, the GUI includes one or more filters 731. Examples of filters include, but are not limited to, showing all images, showing all labeled images, showing all unlabeled images, showing images labeled as normal, and / or showing images labeled as abnormal.

[0081] In some embodiments, for displayed images, the GUI includes a plurality of images shown 707, functionality for searching within those images 709, functionality for sorting within those images 711 (e.g., by size, name, date, abnormality status, etc.), functionality for classifying one or more selected images as abnormal 713, and functionality for classifying one or more selected images as normal 715. Specifically, the various functions shown invoke functions to be performed.

[0082] Additionally, the image itself is shown as a separate image GUI 720. The separate image GUI 720 can be just the image itself, and can also include additional information, such as one or more of the file name and / or how the user marked it as normal 721 or abnormal 723. Note that in some embodiments, the user can use a classification function (e.g., across multiple images) rather than a separate classification. In some embodiments, a GUI is provided that allows the user to perform more complex labeling, such as providing the user with a way to highlight abnormal areas, orientations, objects not to be evaluated, specific areas of the image to be evaluated (e.g., in a wide image where only a portion of the image is viewed), etc.

[0083] Figure 8 An embodiment of a GUI for manipulating a data set is shown. This GUI uses Figure 7 Same number, but at a different point in time. Specifically, the three images shown previously have two to be labeled normal (those labels can be attached using the Save Changes function 801). Exit function 803 leaves this view.

[0084] Figure 9 An embodiment of a GUI for manipulating a data set is shown. This GUI uses Figure 8 Same number, but at a different point in time. Specifically, the three images shown previously had two images marked as normal (and therefore marked and not shown) and one image now marked as abnormal.

[0085] Figure 10 An embodiment of a GUI for manipulating a data set is shown. This GUI uses Figure 7 Same number, but at different points in time. In this view, all images are shown, including their labels.

[0086] Figure 11An embodiment of a GUI showing details of a trained model is shown. In some embodiments, this GUI includes information about performance metrics 1101 of the model. These metrics may include how long training took 1107, information about the test images 1109 (such as the total number, the number of anomalies found), a precision value 1113, a score value 1111, a recall value 1115, and a threshold value 1117 for anomaly detection. Additional information such as the test status 1103 and the training date 1105 may also be provided.

[0087] In some embodiments, advanced metrics 1119 such as receiver operating characteristic (ROC) curves, area under the curve (AOC), etc. are available for viewing.

[0088] In some embodiments, high-level test results are shown 1120, such as the number of tested images correctly predicted as normal, correctly predicted as abnormal, and so on.

[0089] In some embodiments, the GUI includes one or more filters 1131. Examples of filters include, but are not limited to, showing all images, showing all labeled images, showing all unlabeled images, showing images labeled as normal, and / or showing images labeled as abnormal.

[0090] In some embodiments, for the displayed images, the GUI includes one or more of the following: multiple images 1131 shown, a function 1133 to search within those images, a function 1135 to sort within those images (e.g., by size, name, date, abnormality status, etc.), etc.

[0091] In addition, the image itself is shown as a separate image GUI 1141. The separate image GUI 1141 can be just the image itself, and can also contain additional information, such as one or more of the file name, an indication of whether the prediction was correct, an indication of the prediction, and / or a score for the prediction. Note that the anomaly image shows the locations where the anomalies were detected (black circles and black squares). In some embodiments, this is shown using a pixel-level mask on top of the image. In some embodiments, the mask can be more general, such as having a grid overlaid on the image and an indication of which block of the prediction grid has a problem. As shown later, in some embodiments, a GUI is provided to highlight problems that should have been found but were not, etc.

[0092] Figure 12An embodiment of a GUI for configuring the use of a model to perform anomaly detection is shown. Specifically, this GUI shows aspects of a job configuration 1201 request. These aspects include selecting the model to be used 1203 and an indication of where the unlabeled images will come from for testing 1205, such as from a provider network storage device (e.g., uploaded to a storage device from a camera, etc.), from a local file, or from one or more external devices (such as streaming). In this example, the images will come from the provider network storage device as given from storage location 1207. In some embodiments, the user can select how to perform anomaly notification, such as via SMS, email, phone call, chat, etc. A function 1211 of starting the detection process is provided via input. The notification may include one or more of an indication of the image with the anomaly, the time the image was taken, the camera that took the image, the type of anomaly (orientation, etc.), the anomaly image with a mask covering showing the potential anomaly location, etc.

[0093] Figure 13 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown. In some embodiments, the GUI includes instructions on what validation to perform. In some embodiments, high-level test results are shown 1302, such as the number of tested images correctly predicted as normal, correctly predicted as anomaly, etc.

[0094] In some embodiments, the GUI includes one or more filters 1331. Examples of filters include, but are not limited to, showing all images, showing all labeled images, showing all unlabeled images, showing images labeled as normal, and / or showing images labeled as abnormal.

[0095] In some embodiments, the user can sort the displayed images using sorting functionality 1323. In this example, the user can sort by confidence, but other types of sorting may also be supported, such as image size, time to detect anomaly, number of anomalies, number of objects detected, orientation, etc.

[0096] In some embodiments, for the displayed images, the GUI includes one or more of the following: multiple images shown 1331, a function to search within those images 1333, a function to sort within those images 1335 (e.g., by size, name, date, abnormal status, etc.), verification of correctness 1337 or incorrectness 1337.

[0097] Additionally, the image itself is shown as a separate image GUI 1341. The separate image GUI 1341 can be just the image itself and can also include additional information, such as one or more of the file name, prediction indication, and / or predicted score. Additionally, in some embodiments, functionality is provided to indicate whether the prediction was correct 1343 or incorrect 1345.

[0098] Figure 14 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown. Figure 11 Same numbering, but with some additional functionality added. Specifically, the user can verify an image and then use the functionality to add it to the dataset 1401. Images added to the dataset can be used for retraining later.

[0099] Figure 15 An embodiment of a GUI for performing anomaly detection using a model to validate inferred results is shown. Figure 11 Same numbering, but with some additional functionality. Specifically, the user can use functionality to verify the predictions 1501 for the image.

[0100] Figure 16 An embodiment of a GUI for performing more detailed annotation of images is shown. This annotation can be performed based on the results of the inference (to correct inappropriate detections), training images (e.g., to add alignment information, to indicate regions of interest in the image for performing anomaly detection, to indicate anomalies, etc.). Typically, test images are not corrected.

[0101] As shown, the detailed annotation may include several different views (such as an image gallery, a single image (or object), and a detailed view of a single image (or object)); indications of what layers to apply (such as detected objects, alignment, and / or anomaly masks); and / or indications of how to sort the images (such as, by number of objects, ranking, etc.).

[0102] In this example, a single anomaly image is shown, and the user has added an anomaly mask (circle and box), as well as alignment indicators (arrows) showing that the image is incorrectly oriented. Note that the same or similar GUI can be used to show what a "normal" image should look like with respect to orientation, regions of interest, etc. Other annotations can include regions of interest, objects, object shapes, object geometry, and object positions.

[0103] A function 1601 to cancel a comment and / or a function 1603 to accept a comment may also be provided.

[0104] Figure 17An embodiment of a GUI for performing more detailed annotation or labeling of an image is shown. This GUI shows multiple images that are first sorted by the number of objects and then ranked. In some embodiments, a user can select an individual one of the images to be evaluated in a detailed view.

[0105] Figure 18 An embodiment of a GUI for performing annotation of validation images is shown. This annotation can be performed based on the results of inference (to correct inappropriate detections), training or test images (e.g., to add alignment information, to indicate regions of interest in the image for performing anomaly detection, to indicate anomalies, etc.). Typically, test images are not corrected.

[0106] As shown, the detailed annotation may include several different views (such as an image gallery, a single image (or object), and a detailed view of a single image (or object)); indications of what layers to apply (such as detected objects, alignment, and / or anomaly masks); and / or indications of how to sort the images (such as, by number of objects, ranking, etc.).

[0107] In this example, a single anomaly image is shown, and two anomalies are detected. In this GUI, the user can use functions 1805 or 1807, respectively, to verify whether this is correct or incorrect, and correct the prediction (e.g., highlighting where the detected object exists, alignment errors, etc.). A function 1801 to cancel verification or a function 1803 to accept verification can also be provided.

[0108] Figure 19 An embodiment of a GUI for a dashboard of anomaly detection service is shown. The dashboard provides a way for users to see how the anomaly detector performs over a set time period (set using the time period function 1901). Note that users can also request that anomalies be sent via SMS, email, stored in a specific location, etc.

[0109] The dashboard allows the user to view an overview 1903 showing the total number of anomalies detected 1905, the total number of images processed 1907, and / or the anomaly rate of processed images 1909. This allows the user to determine if there are significant issues with the underlying source of the image content. For example, if 100 out of 100 images of a production line indicate abnormal behavior, there may be an issue that requires urgent resolution. Note that an overview can contain multiple items.

[0110] The project section QA011 allows the user to view (or search for) a specific project. This section includes a way to sort the projects 1913, such as by most recent exception, exception rate, most exceptions, etc. The user can request to use 1915 to display the search for the project and / or one or more sorted projects.

[0111] Examples of individual projects are also shown. Each project display 1921 includes the project name, information about the project over time, such as total anomalies, total images processed, total anomaly rate, etc. Note that one or more of those pieces of information may be shown. In some embodiments, clicking on a project name takes the user to a more detailed GUI (such as a verification GUI that allows the user to see a GUI with similar content for a specific image with a problem, such as a Figure 11 Model performance GUI shown, etc.).

[0112] Figure 20 is a flowchart illustrating the operation of a method for utilizing anomaly detection services according to some embodiments. Some or all of the operations (or other processes described herein or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed together on one or more processors. The code is stored on a computer-readable storage medium in the form of a computer program including instructions that can be executed by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the anomaly detection service / component 102 of the other figures.

[0113] At 2002, one or more requests are received to configure and train an anomaly detection system. For example, such as Figure 5 、 Figure 6 、 Figure 7 One or more of the GUIs in the example above are used to generate API calls to configure and train at least one ML model. Of course, the requests can be command line based, where the API calls are made directly.

[0114] At 2004, the anomaly detection system is configured according to the one or more requests. For example, in some embodiments, a project is created at 2006, and a training and / or test dataset is created at 2008. Note that in some embodiments, the creation of one or more datasets also includes user input, such as labeling normal, abnormal, appropriate (or inappropriate) orientation, identifying regions of interest in the image to find anomalies, identifying objects to detect, identifying a model for performing object detection, image extraction parameters, cropping parameters, etc.

[0115] In some embodiments, a model diagram is generated based on the problem to be solved as indicated by the user at 2009. Examples of model diagrams may include, but are not limited to, model diagrams for local object analysis, global object analysis, texture analysis, relative placement and shape, checklist validation, or custom model diagrams.

[0116] In some embodiments, the user provides further configuration, which may include determining a directional stability mode (horizontal, vertical, none, etc.) and / or specifying a range of valid image changes (illumination, geometric deformation, tonal changes, etc.). This can be done in an ad hoc manner through a visual interface in which the user can apply image transformations to the test sample.

[0117] At 2010, one or more models of the anomaly detection system are trained using at least an appropriate subset of the training data. For example, the model indicated as indicated via the GUI is trained using the generated training data. In some embodiments, at 2012, one or more models of the one or more trained models are selected. For example, in some embodiments, a "zoo" of models is trained, and the model with the best test metric is selected. Exemplary test metrics include classification metrics such as accuracy, precision, recall, ROC, AUC, etc. Of course, other metrics may be used (and are user-selectable in some embodiments), such as detection time, detection cost, etc. It should be noted that training may include training of supervised models (e.g., for object detection, alignment, or segmentation) and (semi-) unsupervised models (for anomaly detection).

[0118] Once one or more models are trained (and selected as needed), feedback-based training is performed at 2014 based on the training dataset. Details of different methods for performing this training will be described later. In some embodiments, the user is asked to review the output of the model for the training images and provide feedback on the accuracy of the predictions. The training dataset is then updated to reflect the predictions. In some embodiments, one or more of the previously discussed GUIs are used for feedback.

[0119] Once one or more models have been (re)trained, at 2015, inference is performed on the training dataset using the one or more (re)trained models, and feedback training is performed based on the results of those inferences. For example, a user is asked to view the output of the model for a test image and provide feedback on the accuracy of the predictions. The training dataset is updated to include correctly predicted data. Incorrectly predicted data may be retried.

[0120] At some later point in time, in 2016, unlabeled data is received. This unlabeled data is the object on which anomaly detection is to be performed. For example, one or more models are deployed and ready to begin evaluating the data.

[0121] At 2018, inference is performed on the received unlabeled data using the deployed one or more models. The results of the inference can be stored, provided in real time (or near real time), provided in an alert (e.g., a call, text, chat message, etc.), etc.

[0122] At some point, at 2020, at least one GUI is provided that shows one or more results of at least one inference. As described above, the at least one GUI allows a user to determine how a collection of projects performed, how an individual project performed, view results for individual models, annotate errors in the results, etc.

[0123] In some embodiments, one or more actions are performed in response to the detected anomaly at 2021. For example, stopping the assembly line, activating an actuator to remove a part, etc. These actions may be programmable by the user.

[0124] In some embodiments, when needed or desired, feedback training is performed at 2022. For example, the user provides feedback including one or more of an indication of a missed anomaly, an indication of an inappropriate alignment, an indication of an appropriate anomaly result, and an indication of an inappropriate anomaly result to initiate a retraining cycle.

[0125] Figure 21 is a flowchart illustrating the operations of a method for creating a data set to be used for training and / or testing anomaly detection services according to some embodiments. Some or all of the operations (or other processes described herein or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that executes together on one or more processors. The code is stored on a computer-readable storage medium in the form of, for example, a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the anomaly detection service / component 102 of the other figures. Specifically, in some embodiments, this flow diagram represents the flow diagram of the process diagram. Figure 20 Create training and / or test datasets at 2008.

[0126] In some embodiments, one or more requests to create a training and / or test dataset are received at 2100. The request may include one or more of: at least one indication of at least one image from which to generate training data, at least one indication of an operation to be performed on at least one image to generate a plurality of images from the at least one image, an indication of where to store the generated images, etc.

[0127] At 2102, a graphical user interface is provided showing at least one image of a data set. For example, a graphical user interface similar to Figure 6 、 Figure 7 etc. GUI.

[0128] At 2104, images of the data set are accessed as potentially viewable. As described above, in some embodiments, the images are filtered so that not all are shown.

[0129] In some embodiments, at 2106, aspects are extracted from the accessed image. Details of such extraction are detailed in subsequent figures. However, generally, one or more of an area of ​​interest, an object of interest, a portion of a grid overlaid on the image, and the like are extracted. This allows the model to be trained so that it only sees the most relevant content. For example, in an image with a wide-angle view of a manufacturing plant, only a portion of this view may be relevant and can be extracted.

[0130] At 2108, at least one label for at least an appropriate subset of the images (and / or aspects) of the dataset is received. This label may be determined prior to accessing the image (e.g., as metadata describing the image) or may be provided by a user via a GUI (such as one of those previously described). The at least one label may be one of an abnormal indication or a normal indication.

[0131] In some embodiments, additional information may be provided as part of the annotation of the anomaly-marked image at 2112. For example, the image or an orientation, color, mask, etc. specified in the image.

[0132] In some embodiments, additional information may be provided as part of the normal annotation of the tagged image at 2114. For example, information about the image or a specified orientation, region, object, etc. in the image.

[0133] Figure 22 Flowchart illustrating the operation of a method for creating at least one subset of a training data set according to some embodiments. Some or all of the operations (or other processes described herein or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) executed together on one or more processors. The code is stored on a computer-readable storage medium in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the anomaly detection service / component 102 of the other figures.

[0134] In some embodiments, this flow chart is Figure 20 At 2010, a training and / or test dataset is created 2008 / aspects are extracted from the accessed images. Note that this can allow a single image to provide multiple images for training.

[0135] In some embodiments, one or more requests to create a training and / or test dataset are received at 2200. The request may include one or more of: at least one indication of at least one image from which to generate training data, at least one indication of an operation to be performed on at least one image to generate a plurality of images from the at least one image, an indication of where to store the generated images, etc.

[0136] In some embodiments, at 2202, an indication of at least one region of a visual output to be analyzed using at least one machine learning (ML) model is received. For example, a user can determine which regions of an image should be evaluated and which regions should be ignored. This would allow, for example, a user to indicate that only the top quarter of an image should be analyzed for anomalies. In some embodiments, such indication of a region of interest results from utilizing one or more ML models on the image. For example, an object detection model can be applied to find at least one object of interest, and one or more regions where the object (or objects) of interest were found are indicated.

[0137] At 2204, the accessed images are pre-processed. For example, the accessed images may be subjected to geometric transformations (e.g., one or more of rotation, compression, etc.), photometric transformations (e.g., one or more of magnification, reduction, hue adjustment, etc.), normalization, etc. In some embodiments, the normalization of the first subset of images is performed based on default characteristics of the camera that captured those differences relative to default characteristics of one or more other cameras used to capture the second subset of images.

[0138] In some embodiments, an extraction of identically sized, non-overlapping blocks from each image is performed at 2206. Such extraction does not require explicit indication of the region of interest.

[0139] In some embodiments, at 2208, blocks of different sizes are extracted from each image. These blocks may also overlap. Blocks of different sizes may reduce the chance of cutting an object of interest into multiple blocks.

[0140] In some embodiments, at 2210, extraction or cropping based on the received at least one region is performed. For example, if an image that cannot be clearly decomposed has 12 regions of interest that all should contain the same part, the image can be decomposed into 12 separate images. In some embodiments, at 2212, object detection is performed on the image, and the objects of interest are extracted as blocks.

[0141] In some implementations, the extracted blocks are post-processed at 2214. For example, the blocks may be subjected to geometric transformations (e.g., one or more of rotation, compression, etc.), photometric transformations (e.g., one or more of magnification, reduction, hue adjustment, etc.), normalization, etc.

[0142] At 2214, the extracted patches are stored as images to be added to the training or test dataset. Note that the entire image can be saved, but a post-processed or pre-processed variant is saved.

[0143] In a typical model training system, the user is asked to provide a set of labeled samples, and then the labeled samples are used to train the model. However, the number of labeled samples may not be enough to obtain the required performance of the model. In such cases, the user can introduce unlabeled samples that belong to the problem to be solved and let the model generate output about the unlabeled samples. The user can then provide feedback on the model output, which can be used to generate additional labeled samples to improve model performance. This feedback loop can be executed multiple times until all unlabeled samples are exhausted or the model performance reaches the desired level. Other annotations can include regions of interest, object shapes, object geometries, objects, and object positions.

[0144] Figure 23 An embodiment of a graphical user interface for creating a data set from an image is shown. As shown, the GUI provides the user with the ability to select one or more actions to be taken for a particular image 2301 (e.g., Figure 22 Depending on the action to be taken, tools for providing annotations are provided (e.g., selection of an object, etc.).

[0145] Figure 24 is a flowchart illustrating the operations of a method for performing feedback training according to some embodiments. Some or all of the operations (or other processes described herein or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors. The code is stored on a computer-readable storage medium in the form of, for example, a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the anomaly detection service / component 102 of the other figures. Note that feedback training is not required for anomaly detection. However, in some embodiments, the flowchart representation of Figure 20 2012 or 2012 feedback training.

[0146] Specifically, this process describes a method for improving a system that can perform inference on unlabeled samples by obtaining feedback about the model's output and using the feedback to update the model. The system uses the importance of samples on the unlabeled set to select samples for feedback, thereby maximizing model performance improvement.

[0147] In some embodiments, a request to perform feedback training is received. This request may include one or more of the following: an identifier of one or more models to be (re)trained, an identifier of a dataset to be used for training, an identifier of a dataset to be used for testing, an indication of a threshold for anomalies, an indication of how to display items to be verified (e.g., use of scores, confidence levels, rankings, or a combination thereof), an indication of where to store historical information (e.g., data points, scores, confidence levels, and / or rankings), etc.

[0148] In some embodiments, at 2401 , a model of a model zoo is selected to perform feedback training on it.

[0149] At 2402, the trained model is applied to the project's dataset to generate predictions (e.g., scores) and, in some embodiments, confidence levels in those predictions. For example, an anomaly detection model is applied to an unlabeled dataset to find anomalies and provide confidence levels in its predictions. In some embodiments, the dataset includes enhanced class information (such as lighting conditions) that should be invariant. In some embodiments, the dataset is at least partially annotated. In some embodiments, the dataset is unlabeled.

[0150] At 2404, an importance ranking score is calculated for each item of the unlabeled dataset based on the results of applying the trained model. The ranking of the training set items is performed based on how informative those items are likely to be when re-annotated.

[0151] In some embodiments, the importance ranking score is a weighted anomaly score based on the likelihood that the item is correctly classified. A high probability indicates that the classification is correct. The higher the importance score, the more important the item being viewed (e.g., the most anomaly). Other rankings include confidence scores.

[0152] At 2406, the prediction information, confidence, and ranking scores are stored in a history. This history allows the system to consider the ranking of items at a specific time (e.g., a (re)training cycle). Additionally, the history can be used during inference to try to correct the model output.

[0153] At 2408, the items (e.g., images) of the dataset are displayed to the user along with the predictions. In some embodiments, the items are displayed by rank. In some embodiments, the user can select how to sort using the ranking scores (e.g., showing higher rankings first, etc.). For example, in some embodiments, the top-ranked images are then displayed along with the corresponding model inferences A. M are shown to the user together.

[0154] At 2410, feedback is received regarding the displayed item. Exemplary feedback has been detailed above.

[0155] At 2412, for those items that are verified to be accurate based on the received feedback, those items are added to the training set for use in later training. In some embodiments, errors in the predictions are corrected (e.g., using annotations as described above) and the corrected annotations are provided.

[0156] At 2414, the model is retrained using the training set.

[0157] Figure 25 An embodiment of a component for feedback training and a component for post-inference feedback training is shown. Specifically, a training set 2500 available at time t is used to train a set of models, and the best model 2521 is selected from the set by a model trainer and selector 2501. The selected model 2521 is then applied to unlabeled data 2502 to calculate sample importance scores, ranking information, and / or predictions for the unlabeled set stored in a history 2507. A feedback mechanism 2509 from the user selects samples from the unlabeled data and enhances the training set for the next iteration based on the feedback. For example, the feedback mechanism can show images and corresponding predictions to the user in a manner specified by one or more of the importance scores and ranking information.

[0158] The deployed inference system (shown in the lower half) receives unlabeled data 2523 and applies model 2521 to determine abnormal behavior. In some embodiments, further post-processing (such as sorting results, providing results to specific services or individuals, generating alerts, etc.) can be performed by post-processor 2525.

[0159] Figure 26is a flowchart illustrating the operations of a method for performing inference in an anomaly detection service according to some embodiments. Some or all of the operations (or other processes described herein or variations and / or combinations thereof) are performed by hardware or a combination thereof under the control of one or more computer systems configured with executable instructions and implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors. The code is stored on a computer-readable storage medium in the form of, for example, a computer program including instructions that can be executed by one or more processors. The computer-readable storage medium is non-transitory. In some embodiments, one or more (or all) of the operations are performed by the anomaly detection service / component 102 of the other figures. In some embodiments, this process is represented in Figure 20 2018 to perform inference on unlabeled data.

[0160] In some embodiments, aspects are extracted from unlabeled data (such as images) at 2602. This extraction may include multiple actions. In some embodiments, the accessed data is pre-processed at 2604. For example, the accessed images may be magnified, color-adjusted, normalized, etc. In some embodiments, the first subset of images is normalized based on the default characteristics of the camera that captured those differences relative to the default characteristics of one or more other cameras used to capture the second subset of images.

[0161] In some embodiments, an extraction of identically sized, non-overlapping blocks from each image is performed at 2606. Such extraction does not require explicit indication of the region of interest.

[0162] In some embodiments, at 2608, blocks of different sizes are extracted from each image. These blocks may also overlap. Blocks of different sizes may reduce the chance of cutting an object of interest into multiple blocks.

[0163] In some embodiments, at 2610, extraction or cropping based on the received region indication is performed. For example, if an image that cannot be clearly decomposed has 12 regions of interest that all should contain the same part, the image can be decomposed into 12 separate images. In some embodiments, at 2612, object detection is performed on the image, and the objects of interest are extracted as blocks.

[0164] In some embodiments, at 2614, the block is post-processed. Examples of post-processing and pre-processing are described in detail above. Note that post-processing and / or pre-processing can also be applied to the entire image.

[0165] At 2614, the ML model is applied to one or more images or patches thereof to generate predictions. In some embodiments, the predictions are corrected based on historical data.

[0166] Figure 27 An exemplary provider network (or "service provider system") environment is shown in accordance with some embodiments. A provider network 2700 may provide resource virtualization to customers via one or more virtualization services 2710 that allow customers to purchase, lease, or otherwise obtain instances 2712 of virtualized resources (including, but not limited to, computing resources and storage resources) implemented on devices within the provider network or network in one or more data centers. A local Internet Protocol (IP) address 2716 may be associated with the resource instance 2712; the local IP address is the internal network address of the resource instance 2712 on the provider network 2700. In some embodiments, the provider network 2700 may also provide public IP addresses 2714 and / or public IP address ranges (e.g., Internet Protocol version 4 (IPv4) or Internet Protocol version 6 (IPv6) addresses) that customers may obtain from the provider 2700.

[0167] Conventionally, the provider network 2700 can allow a customer of the service provider (e.g., a customer operating one or more client networks 2750A-2750C including one or more customer devices 2752) via the virtualization service 2710 to dynamically associate at least some public IP addresses 2714 assigned or allocated to the customer with a specific resource instance 2712 assigned to the customer. The provider network 2700 can also allow the customer to remap a public IP address 2714 previously mapped to one virtualized computing resource instance 2712 allocated to the customer to another virtualized computing resource instance 2712 also allocated to the customer. For example, a customer of a service provider (e.g., an operator of one or more customer networks 2750A-2750C) can use the virtualized computing resource instances 2712 and public IP addresses 2714 provided by the service provider to implement customer-specific applications and present the customer's applications over an intermediate network 2740, such as the Internet. Other network entities 2720 on the intermediate network 2740 can then generate traffic to the destination public IP address 2714 published by one or more customer networks 2750A to 2750C; the traffic is routed to the service provider data center and, at the data center, is routed via the network underlay to the local IP address 2716 of the virtualized computing resource instance 2712, which is now mapped to the destination public IP address 2714. Similarly, response traffic from the virtualized computing resource instance 2712 can be routed back onto the intermediate network 2740 via the network underlay to the source entity 2720.

[0168] As used herein, a local IP address refers to an internal or "private" network address of a resource instance, for example, in a provider network. A local IP address may be within an address block reserved by Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918 and / or have an address format specified by IETF RFC 4193, and may be variable within the provider network. Network traffic originating from outside the provider network is not routed directly to the local IP address; instead, the traffic uses a public IP address that is mapped to the local IP address of the resource instance. The provider network may include network devices or appliances that provide network address translation (NAT) or similar functionality to perform the mapping from public IP addresses to local IP addresses, and vice versa.

[0169] A public IP address is an Internet-variable network address assigned to a resource instance by a service provider or a customer. Traffic routed to a public IP address is translated and forwarded to the corresponding local IP address of the resource instance, for example, via 1:1 NAT.

[0170] Some public IP addresses may be assigned to specific resource instances by the provider network infrastructure; these public IP addresses may be referred to as standard public IP addresses, or simply standard IP addresses. In some embodiments, the mapping of standard IP addresses to the local IP addresses of resource instances is the default startup configuration for all resource instance types.

[0171] At least some public IP addresses may be allocated to or obtained by customers of the provider network 2700; the customers may then assign their allocated public IP addresses to specific resource instances assigned to the customers. These public IP addresses may be referred to as customer public IP addresses, or simply customer IP addresses. Instead of being assigned to resource instances by the provider network 2700 as in the case of standard IP addresses, customer IP addresses may be assigned to resource instances by the customer, for example, via an API provided by the service provider. Unlike standard IP addresses, customer IP addresses are assigned to customer accounts and can be remapped to other resource instances by the respective customers as needed or desired. Customer IP addresses are associated with a customer account, not a specific resource instance, and the customer controls the IP address until the customer chooses to release it. Unlike conventional static IP addresses, customer IP addresses allow customers to shield against resource instance or availability zone failures by remapping the customer's public IP address to any resource instance associated with the customer account. For example, customer IP addresses enable customers to resolve issues with customer resource instances or software by remapping the customer IP address to a replacement resource instance.

[0172] Figure 2828 is a block diagram of an exemplary provider network that provides storage services and hardware virtualization services to customers according to some embodiments. Hardware virtualization service 2820 provides multiple computing resources 2824 (e.g., computing instances 2825 such as VMs) to customers. For example, computing resources 2824 can be rented or leased to customers of provider network 2800 (e.g., customers implementing customer network 2850). Each computing resource 2824 can be provided with one or more local IP addresses. Provider network 2800 can be configured to route packets from the local IP addresses of computing resources 2824 to public Internet destinations, and to route packets from public Internet sources to the local IP addresses of computing resources 2824.

[0173] The provider network 2800 can provide a customer network 2850 coupled to the intermediary network 2840, for example, via a local network 2856, with the ability to implement a virtual computing system 2892 via a hardware virtualization service 2820 coupled to the intermediary network 2840 and the provider network 2800. In some embodiments, the hardware virtualization service 2820 can provide one or more APIs 2802 (e.g., web service interfaces) via which the customer network 2850 can access functionality provided by the hardware virtualization service 2820, for example, via a console 2894 (e.g., a web-based application, a standalone application, a mobile application, etc.). In some embodiments, at the provider network 2800, each virtual computing system 2892 at the customer network 2850 can correspond to a computing resource 2824 that is leased, rented, or otherwise provided to the customer network 2850.

[0174] A customer may access functionality of the storage service 2810, for example, from an instance of a virtual computing system 2892 and / or another customer device 2890 (e.g., via a console 2894), via one or more APIs 2802, to access and store data from storage resources 2818A through 2818N of virtual data stores 2816 (e.g., folders or “buckets,” virtualized volumes, databases, etc.) provided by the provider network 2800. In some embodiments, a virtualized data storage gateway (not shown) may be provided at the customer network 2850, which may cache at least some data (e.g., frequently accessed or critical data) locally and may communicate with the storage service 2810 via one or more communication channels to upload new or modified data from the local cache, such that a primary store of data (the virtualized data store 2816) is maintained. In some embodiments, a user via a virtual computing system 2892 and / or on another client device 2890 can mount and access virtual data storage 2816 volumes via a storage service 2810 acting as a storage virtualization service, and these volumes can appear to the user as local (virtualized) storage devices 2898.

[0175] Although Figure 28 Although not shown, one or more virtualized services may also be accessed from resource instances within the provider network 2800 via one or more APIs 2802. For example, a customer, device service provider, or other entity may access a virtualized service via API 2802 from within a corresponding virtual network on the provider network 2800 to request allocation of one or more resource instances within the virtual network or within another virtual network.

[0176] Descriptive System

[0177] In some embodiments, a system implementing part or all of the techniques described herein may include a general-purpose computer system that includes or is configured to access one or more computer-accessible media, such as Figure 29 In the embodiment shown, the computer system 2900 includes one or more processors 2910 coupled to a system memory 2920 via an input / output (I / O) interface 2930. The computer system 2900 also includes a network interface 2940 coupled to the I / O interface 2930. Although Figure 29 Computer system 2900 is shown as a single computing device, but in various embodiments, computer system 2900 may include one computing device or any number of computing devices configured to work together as a single computer system 2900 .

[0178] In various embodiments, the computer system 2900 can be a uniprocessor system including one processor 2910 or a multiprocessor system including several processors 2910 (e.g., two, four, eight, or another suitable number). The processor 2910 can be any suitable processor capable of executing instructions. For example, in various embodiments, the processor 2910 can be a general-purpose or embedded processor that implements any of a variety of instruction set architectures (ISAs), such as the x86, ARM, PowerPC, SPARC, or MIPS ISAs or any other suitable ISAs. In a multiprocessor system, each of the processors 2910 can typically (but not necessarily) implement the same ISA.

[0179] The system memory 2920 can store instructions and data that can be accessed by one or more processors 2910. In various embodiments, the system memory 2920 can be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated embodiment, program instructions and data that implement one or more desired functions (such as those methods, techniques, and data described above) are shown as being stored within the system memory 2920 as anomaly detection service code 2925 (e.g., executable to implement all or part of the anomaly detection service 102) and data 2926.

[0180] In one embodiment, the I / O interface 2930 can be configured to coordinate I / O traffic between the processor 2910, the system memory 2920, and any peripheral devices in the device (including the network interface 2940 or other peripheral interfaces). In some embodiments, the I / O interface 2930 can perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., the system memory 2920) into a format suitable for use by another component (e.g., the processor 2910). In some embodiments, the I / O interface 2930 may include support for devices attached via various types of peripheral buses (e.g., such as variations of the Peripheral Component Interconnect (PCI) bus standard or the Universal Serial Bus (USB) standard). In some embodiments, the functionality of the I / O interface 2930 can be split into two or more separate components, such as, for example, a north bridge and a south bridge. Furthermore, in some embodiments, some or all of the functionality of the I / O interface 2930 (such as the interface to the system memory 2920) can be incorporated directly into the processor 2910.

[0181] The network interface 2940 may be configured to allow the computer system 2900 to communicate with other devices 2960 attached to one or more networks 2950 (e.g., such as Figure 1 In various embodiments, the network interface 2940 may support communication via any suitable wired or wireless general-purpose data network, such as an Ethernet network type, for example. Additionally, the network interface 2940 may support communication via a telecommunications / telephone network, such as an analog voice network or a digital fiber-optic communication network, via a storage area network (SAN), such as a Fibre Channel SAN, or via any other suitable type of network and / or protocol for I / O.

[0182] In some embodiments, the computer system 2900 includes one or more offload cards 2970A or 2970B (including one or more processors 2975 and may include one or more network interfaces 2940) that are connected using an I / O interface 2930 (e.g., a bus that implements a version of the Peripheral Component Interconnect Express (PCI-E) standard or another interconnect such as Quick Path Interconnect (QPI) or Ultra Path Interconnect (UPI). For example, in some embodiments, the computer system 2900 may act as a host electronic device that hosts computing resources (such as computing instances) (e.g., operating as part of a hardware virtualization service), and the one or more offload cards 2970A or 2970B execute a virtualization manager that can manage computing instances executing on the host electronic device. As an example, in some embodiments, the one or more offload cards 2970A or 2970B may perform computing instance management operations, such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In some embodiments, these management operations may be performed by the offload card(s) 2970A or 2970B in cooperation with (e.g., pursuant to requests from) hypervisors executed by the other processors 2910A through 2910N of the computer system 2900. However, in some embodiments, the virtualization manager implemented by the offload card(s) 2970A or 2970B may accommodate requests from other entities (e.g., from the compute instances themselves) and may not cooperate with (or service) any separate hypervisor.

[0183] In some embodiments, the system memory 2920 can be an embodiment of a computer-accessible medium configured to store program instructions and data as described above. However, in other embodiments, program instructions and / or data can be received, sent, or stored on different types of computer-accessible media. Generally speaking, the computer-accessible medium may include a non-transitory storage medium or memory medium, such as a magnetic medium or an optical medium, for example, a disk or DVD / CD coupled to the computer system 2900 via the I / O interface 2930. The non-transitory computer-accessible storage medium may also include any volatile or non-volatile medium, such as RAM (e.g., SDRAM, double data rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc., which may be included in some embodiments of the computer system 2900 as system memory 2920 or another type of memory. In addition, the computer-accessible medium may include a transmission medium or signal (such as an electrical signal, an electromagnetic signal, or a digital signal) conveyed via a communication medium (such as a network and / or a wireless link), such as may be implemented via the network interface 2940.

[0184] Figure 30 is a block diagram of an illustrative operating environment for training and hosting machine learning models according to some embodiments. The operating environment includes an end-user device (e.g., one or more computing devices, one or more edge devices), a model training system, a model hosting system, a training data store 3060, a training metrics data store 3065, a container data store 3070, a training model data store 3075, and a model prediction data store 3080.

[0185] The machine learning services described herein may include one or more of these entities, such as the model hosting system 140 , the model training system 120 , and the like.

[0186] In some embodiments, a user interacts with the model training system 120 via a user device to provide data that causes the model training system 120 to train one or more machine learning models, e.g., as described elsewhere herein. In general, a machine learning model can be thought of as one or more equations that are "trained" using a collection of data. In some embodiments, the model training system 120 provides ML functionality as a network service, and thus messaging between the user device and the model training system 120 (or provider network 100) and / or between components of the model training system 120 (or provider network 100) can use HTTP messages to transfer data in a machine-readable file format such as Extensible Markup Language (XML) or JavaScript Object Notation (JSON). In some embodiments, providing access to various functions as network services is not limited to communications exchanged via the World Wide Web, and more generally refers to services that are capable of communicating with other electronic devices via a computer network.

[0187] The user device may interact with the model training system 120 via the front end 3029 of the model training system 120. For example, the user device may provide a training request to the front end 3029, which includes a container image (or multiple container images, or an identifier of one or more locations where the container images are stored), an identifier of the input data (e.g., the address or location of the input data), one or more hyperparameter values ​​(e.g., a value indicating how the algorithm is to operate, how many algorithms to run in parallel, how many clusters to divide the data into, etc.), and / or information describing the computing machine on which the machine learning model is to be trained (e.g., a graphics processing unit (GPU) instance type, a central processing unit (CPU) instance type, an amount of memory to allocate, a type of virtual machine instance to be used for training, etc.).

[0188] In some embodiments, the container image may include one or more layers, each of which represents executable instructions. Some or all of the executable instructions together represent an algorithm that defines a machine learning model. Executable instructions (e.g., an algorithm) can be written in any programming language (e.g., Python, Ruby, C++, Java, etc.). In some embodiments, the algorithm is pre-generated and obtained by the user via the user device from an algorithm repository (e.g., a network-accessible marketplace, a data storage area provided by a machine learning training service, etc.). In some embodiments, the algorithm is completely generated by the user or partially generated by the user (e.g., user-provided code modifies or configures existing algorithm code).

[0189] In some embodiments, instead of providing a container image (or its identifier) ​​in a training request, a user device may provide an algorithm written in any programming language in a training request. The model training system 120 then packages the algorithm into a container (optionally with other code, such as a "base" ML algorithm supplemented by the user-provided code) that is ultimately loaded into the virtual machine instance 3022 for use in training the machine learning model, as described in more detail below. For example, a user may develop an algorithm / code using an application (e.g., an interactive web-based programming environment) via a user device and have the algorithm / code provided (possibly as part of (or referenced in) a training request) to the model training system 120, where this algorithm / code may be containerized separately or used with, for example, an existing container with a machine learning framework.

[0190] In some embodiments, instead of providing a container image in the training request, the user device provides an identifier of the container image (e.g., an address or an indication of the storage location of the container image) in the training request. For example, the container image may be stored in the container data storage area 3070, and this container image may have been previously created / uploaded by the user. The model training system 120 may retrieve the container image from the indicated location and create a container using the retrieved container image. The container is then loaded into the virtual machine instance 3022 for use in training the machine learning model, as described in more detail below.

[0191] In some embodiments, the model training system 120 can use information provided by the user device to train a machine learning model in one or more pre-established virtual machine instances 3022. Specifically, the model training system 120 includes a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein one or more physical computing devices host one or more virtual machine instances 3022. The model training system 120 can handle the acquisition and configuration of computing capacity (e.g., containers, instances, etc., which are described in more detail below) based on information describing the computing machine on which the machine learning model provided by the user device is to be trained. The model training system 120 can then use the computing capacity to train the machine learning model, as described in more detail below. The model training system 120 can automatically scale based on the capacity of training requests received from the user device via the front end 3029, thereby alleviating the burden on the user of having to worry about overutilization (e.g., acquiring too few computing resources and experiencing performance issues) or underutilization (e.g., acquiring more computing resources than necessary to train the machine learning model and therefore overpaying).

[0192] In some embodiments, virtual machine instances 3022 are used to perform tasks. For example, these tasks may include training a machine learning model. Figure 30 As shown, each virtual machine instance 3022 includes an operating system (OS) 3024, a language runtime 3026, and one or more ML training containers 3030. Generally, an ML training container 3030 is a logical unit created within the virtual machine instance using resources available on the instance and can be used to isolate the execution of tasks from other processes (e.g., task execution) occurring within the instance. In some embodiments, an ML training container 3030 is formed from one or more container images and a top container layer. Each container image may also include one or more image layers, each of which represents executable instructions. As described above, some or all of the executable instructions together represent the algorithm that defines the machine learning model. Changes made to the ML training container 3030 (e.g., creation of new files, modification of existing files, file deletion, etc.) are stored in the top container layer. If an ML training container 3030 is deleted, the top container layer is also deleted. However, one or more container images forming part of the deleted ML training container 3030 may remain unchanged. The ML training container 3030 may be implemented, for example, as a Linux container (LXC), a Docker container, or the like.

[0193] In some embodiments, ML training container 3030 may include a separate runtime 3034, code 30330, and dependencies 3032 required by code 30330. Runtime 3034 may be defined by one or more executable instructions that form at least a portion of a container image used to form ML training container 3030 (e.g., one or more executable instructions in a container image that defines an operating system and / or runtime for running in a container formed by the container image). Code 30330 includes one or more executable instructions that form at least a portion of a container image used to form ML training container 3030. For example, code 30330 may include executable instructions in a container image that represent an algorithm defining a machine learning model, which may reference (or utilize) code or libraries from dependencies 3032. Runtime 3034 is configured to execute code 30330 in response to an instruction to initiate machine learning model training. Execution of code 30330 results in the generation of model data, as described in more detail below.

[0194] In some embodiments, code 30330 includes executable instructions representing algorithms that define different machine learning models. For example, code 30330 includes a set of executable instructions representing a first algorithm that defines a first machine learning model; and a second set of executable instructions representing a second algorithm that defines a second machine learning model. In some embodiments, virtual machine instance 3022 executes code 30330 and trains all machine learning models. In some embodiments, virtual machine instance 3022 executes code 30330, thereby selecting one of the machine learning models for training. For example, virtual machine instance 3022 may identify a type of training data indicated by the training request and select a machine learning model corresponding to the identified type of training data for training (e.g., executing executable instructions representing an algorithm that defines the selected machine learning model).

[0195] In some embodiments, the runtime 3034 is the same as the runtime 3026 utilized by the virtual machine instance 3022. In some embodiments, the runtime 3034 is different from the runtime 3026 utilized by the virtual machine instance 3022.

[0196] In some embodiments, the model training system 120 uses one or more container images included in the training request (or container images retrieved from the container data store 3070 in response to the received training request) to create and initialize the ML training container 3030 in the virtual machine instance 3022. For example, the model training system 120 creates the ML training container 3030, which includes one or more container images and / or a top container layer.

[0197] Before starting the training process, in some embodiments, the model training system 120 retrieves the training data from the location indicated in the training request. For example, the location indicated in the training request can be a location in the training data store 3060. Therefore, the model training system 120 retrieves the training data from the location indicated in the training data store 3060. In some embodiments, the model training system 120 does not retrieve the training data before starting the training process. Instead, the model training system 120 streams the training data from the indicated location during the training process. For example, the model training system 120 can initially retrieve a portion of the training data and provide the retrieved portion to the virtual machine instance 3022 that is training the machine learning model. Once the virtual machine instance 3022 has requested and used the retrieved portion, or once the virtual machine instance 3022 is about to use all of the retrieved portion (e.g., the buffer storing the retrieved portion is almost empty), the model training system 120 can then retrieve a second portion of the training data and provide the second retrieved portion to the virtual machine instance 3022, and so on.

[0198] In some embodiments, to perform machine learning model training, virtual machine instance 3022 executes code 30330 stored in ML training container 3030. For example, code 30330 includes some or all of the executable instructions that form the container image of ML training container 3030 initialized therein. Thus, virtual machine instance 3022 executes some or all of the executable instructions (which form the container image of ML training container 3030 initialized therein) to train the machine learning model. Virtual machine instance 3022 executes some or all of the executable instructions based on the hyperparameter values ​​included in the training request. As an illustrative example, virtual machine instance 3022 trains the machine learning model by identifying values ​​for certain parameters (e.g., coefficients, weights, center of gravity, etc.). The identified values ​​depend on hyperparameters that define how training is performed. Thus, virtual machine instance 3022 can execute the executable instructions to initiate a machine learning model training process, where the training process is run using the hyperparameter values ​​included in the training request. Execution of the executable instructions may include the virtual machine instance 3022 applying the training data retrieved by the model training system 120 as input parameters to some or all of the executed instructions.

[0199] In some embodiments, execution of the executable instructions causes virtual machine instance 3022 (e.g., ML training container 3030) to generate model data. For example, ML training container 3030 generates model data and stores the model data in the file system of ML training container 3030. The model data includes characteristics of the machine learning model being trained, such as multiple layers in the machine learning model, hyperparameters of the machine learning model, coefficients of the machine learning model, weights of the machine learning model, etc. Specifically, the generated model data includes values ​​used to define characteristics of the machine learning model being trained. In some embodiments, execution of the executable instructions causes modification of ML training container 3030 such that the model data is written to the top container layer of ML training container 3030, and / or modification of one or more container images forming part of ML training container 3030 to include the model data.

[0200] The virtual machine instance 3022 (or the model training system 120 itself) extracts the generated model data from the ML training container 3030 and stores the generated model data in the trained model data store 3075 as an entry associated with the virtual machine instance 3022 and / or the machine learning model being trained. In some embodiments, the virtual machine instance 3022 generates a single file including the model data and stores the single file in the trained model data store 3075. In some embodiments, the virtual machine instance 3022 generates multiple files during the process of training the machine learning model, each file including model data. In some embodiments, each model data file includes the same or different model data information (e.g., one file identifies the structure of the algorithm, another file includes a list of coefficients, etc.). The virtual machine instance 3022 can package the multiple files into a single file once training is complete and store the single file in the trained model data store 3075. Alternatively, the virtual machine instance 3022 stores multiple files in the trained model data store 3075. While the training process is ongoing and / or after the training process is completed, the virtual machine instance 3022 stores one or more files in the trained model data storage area 3075.

[0201] In some embodiments, while the training process is ongoing, the virtual machine instance 3022 periodically stores model data files in the trained model data store 3075. Thus, one or more model data files may be stored in the trained model data store 3075 at different times during the training process. Each set of model data files corresponding to a particular time or each set of model data files present in the trained model data store 3075 as of a particular time may be a checkpoint representing a different version of the partially trained machine learning model during different stages of the training process. Thus, before training is completed, a user may submit a deployment and / or execution request via a user device in the manner described below to deploy and / or execute a version of the partially trained machine learning model (e.g., a trained machine learning model as of a certain stage of the training process). A version of the partially trained machine learning model may be based on some or all of the model data files stored in the trained model data store 3075.

[0202] In some embodiments, virtual machine instance 3022 executes code 30330 stored in multiple ML training containers 3030. For example, the algorithm included in the container image may be in a format that allows for parallelization of the training process. Therefore, model training system 120 may create multiple copies of the container image provided in the training request and cause virtual machine instance 3022 to load each container image copy into a separate ML training container 3030. Virtual machine instance 3022 may then execute code 30330 stored in ML training containers 3030 in parallel. Virtual machine instance 3022 may also provide configuration information to each ML training container 3030 (e.g., information indicating that N ML training containers 3030 are jointly training a machine learning model and that the particular ML training container 3030 receiving the configuration information is the Xth ML training container 3030 among the N), which may be included in the generated model data. In some embodiments, by parallelizing the training process, model training system 120 may significantly reduce training time.

[0203] In some embodiments, multiple virtual machine instances 3022 execute code 30330 stored in multiple ML training containers 3030. For example, the resources used to train a particular machine learning model may exceed the limitations of a single virtual machine instance 3022. However, the algorithm included in the container image may be in a format that allows for parallelization of the training process. Therefore, the model training system 120 may create multiple copies of the container image provided in the training request, initialize multiple virtual machine instances 3022, and cause each virtual machine instance 3022 to load a copy of the container image in one or more separate ML training containers 3030. The virtual machine instances 3022 may then each execute the code 30330 stored in the ML training container 3030 in parallel. The model training system 120 may also provide configuration information (e.g., information indicating that N ML training containers 3030 are jointly training a machine learning model and that the specific ML training container 3030 receiving the configuration information is the Xth ML training container 3030 among the N ML training containers, information indicating that M virtual machine instances 3022 are jointly training a machine learning model and that the specific ML training container 3030 receiving the configuration information is initialized in the Yth virtual machine instance 3022 among the M virtual machine instances, etc.) to each ML training container 3030 via the virtual machine instance 3022, which may be included in the generated model data. As described above, in some embodiments, by parallelizing the training process, the model training system 120 can significantly reduce training time.

[0204] In some embodiments, the model training system 120 includes multiple physical computing devices, and two or more of the physical computing devices host one or more virtual machine instances 3022 that execute code 30330. Thus, in addition to being on different virtual machine instances 3022 and / or ML training containers 3030, parallelization can occur on different physical computing devices.

[0205] In some embodiments, the model training system 120 includes an ML model evaluator 3028. The ML model evaluator 3028 can monitor the virtual machine instance 3022 while the machine learning model is being trained, thereby obtaining generated model data and processing the obtained model data to generate model metrics. For example, model metrics may include quality metrics such as an error rate of the machine learning model being trained, a statistical distribution of the machine learning model being trained, a latency of the machine learning model being trained, a confidence level in the machine learning model being trained (e.g., a confidence level in the accuracy of the machine learning model being trained), etc. The ML model evaluator 3028 may obtain model data and evaluation data for the machine learning model being trained from the training data store 3060. The evaluation data is separate from the data used to train the machine learning model and includes both input data and expected output (e.g., a known result), and thus the ML model evaluator 3028 may define a machine learning model using the model data and execute the machine learning model by providing the input data as input to the machine learning model. The ML model evaluator 3028 may then compare the output of the machine learning model with the expected output and, based on the comparison, determine one or more quality metrics for the machine learning model being trained (e.g., an error rate may be the difference or distance between the machine learning model output and the expected output).

[0206] In some embodiments, the ML model evaluator 3028 periodically generates model metrics during the training process and stores the model metrics in the training metrics data store 3065. While the machine learning model is being trained, the user can access and retrieve the model metrics from the training metrics data store 3065 via the user device. The user can then use the model metrics to determine whether to adjust the training process and / or stop the training process. For example, the model metrics may indicate that the machine learning model is performing poorly (e.g., has an error rate above a threshold, has a statistical distribution that is not an expected or desired distribution (e.g., not a binomial distribution, a Poisson distribution, a geometric distribution, a normal distribution, a Gaussian distribution, etc.), has an execution latency above a threshold, has a confidence level below a threshold) and / or is performing gradually worse (e.g., the quality metrics continue to deteriorate over time). In response, in some embodiments, the user can transmit a request to the model training system 120 via the user device to modify the machine learning model being trained (e.g., transmit a modification request). The request can include a new or modified container image, a new or modified algorithm, one or more new or modified hyperparameters, and / or new or modified information describing the computing machine on which the machine learning model is to be trained. The model training system 120 can modify the machine learning model accordingly. For example, the model training system 120 can cause the virtual machine instance 3022 to optionally delete the existing ML training container 3030, create and initialize a new ML training container 3030 using some or all of the information included in the request, and execute the code stored in the new ML training container 3030 to restart the machine learning model training process. As another example, the model training system 120 can cause the virtual machine instance 3022 to modify the execution of the code stored in the existing ML training container 3030 based on the data provided in the modification request. In some embodiments, the user can transmit a request to the model training system 120 via a user device to stop the machine learning model training process. The model training system 120 can then instruct the virtual machine instance 3022 to delete the ML training container 3030 and / or delete any model data stored in the training model data storage area 3075.

[0207] As described below, in some embodiments, the model data stored in the training model data storage area 3075 is used by the model hosting system 140 to deploy the machine learning model. Alternatively or additionally, a user device or another computing device (not shown) can retrieve the model data from the training model data storage area 3075 to implement the learning algorithm in an external device. As an illustrative example, a robotic device may include a sensor for capturing input data. The user device can retrieve the model data from the training model data storage area 3075 and store the model data in the robotic device. The model data defines the machine learning model. Therefore, the robotic device can provide the captured input data as input to the machine learning model, thereby generating an output. The robotic device can then perform an action based on the generated output (e.g., move forward, raise an arm, generate a sound, etc.).

[0208] Although virtual machine instance 3022 is Figure 30 3022, some embodiments of the present application separate virtual machine instances 3022 that are actively assigned to execute tasks from those virtual machine instances 3022 that are not actively assigned to execute tasks. For example, those virtual machine instances 3022 that are actively assigned to execute tasks are grouped into an "active pool," while those virtual machine instances 3022 that are not actively assigned to execute tasks are placed in a "warming pool." In some embodiments, those virtual machine instances 3022 in the warming pool can be pre-initialized with the operating system, language runtime, and / or other software required to enable rapid execution of tasks (e.g., rapid initialization of machine learning model training in one or more ML training containers 3030) in response to training requests.

[0209] In some embodiments, the model training system 120 includes a processing unit, a network interface, a computer-readable medium driver, and an input / output device interface, all of which can communicate with each other via a communication bus. The network interface can provide connectivity to one or more networks or computing systems. Therefore, the processing unit can receive information and instructions from other computing systems or services (e.g., user devices, model hosting systems 140, etc.). The processing unit can also communicate to and from the memory of the virtual machine instance 3022 and also provide output information for an optional display via the input / output device interface. The input / output device interface can also receive input from an optional input device. The memory can contain computer program instructions (which are grouped into modules in some embodiments), and the processing unit executes the computer program instructions to implement one or more aspects of the present disclosure.

[0210] In some embodiments, the model hosting system 140 includes a single physical computing device or multiple physical computing devices interconnected using one or more computing networks (not shown), wherein one or more physical computing devices host one or more virtual machine instances 3042. The model hosting system 140 can handle the acquisition and configuration of computing power (e.g., containers, instances, etc.) based on the demand for executing the trained machine learning model. The model hosting system 140 can then use the computing power to execute the machine learning model, as described in more detail below. The model hosting system 140 can automatically scale based on the capacity of execution requests received from user devices via the front end 3049 of the model hosting system 140, thereby alleviating the burden of users having to worry about overutilization (e.g., acquiring too few computing resources and experiencing performance issues) or underutilization (e.g., acquiring more computing resources than necessary to run the machine learning model and thus overpaying).

[0211] In some embodiments, virtual machine instances 3042 are used to perform tasks. For example, these tasks may include executing machine learning models. Figure 30 As shown, each virtual machine instance 3042 includes an operating system (OS) 3044, a language runtime 3046, and one or more ML scoring containers 3050. The ML scoring container 3050 is similar to the ML training container 3030 in that it is a logical unit created within the virtual machine instance using the resources available on the instance and can be used to isolate the execution of tasks from other processes (e.g., task execution) occurring within the instance. In some embodiments, the ML scoring container 3050 is formed from one or more container images and a top container layer. Each container image also includes one or more image layers, each of which represents executable instructions. As described above, some or all of the executable instructions together represent the algorithm that defines the machine learning model. Changes made to the ML scoring container 3050 (e.g., creation of new files, modification of existing files, file deletion, etc.) are stored in the top container layer. If the ML scoring container 3050 is deleted, the top container layer is also deleted. However, one or more container images forming part of the deleted ML scoring container 3050 may remain unchanged. The ML scoring container 3050 can be implemented, for example, as a Linux container.

[0212] In some embodiments, each ML scoring container 3050 includes a runtime 3054, code 3056, and dependencies 3052 (e.g., supporting software such as libraries) required by code 3056. Runtime 3054 may be defined by one or more executable instructions that form at least a portion of a container image used to form the ML scoring container 3050 (e.g., one or more executable instructions in a container image that defines an operating system and / or runtime for running in a container formed by the container image). Code 3056 includes one or more executable instructions that form at least a portion of a container image used to form the ML scoring container 3050. For example, code 3056 may include executable instructions in a container image that represent an algorithm that defines a machine learning model, which may reference dependencies 3052. Code 3056 may also include model data representing characteristics of the defined machine learning model, as described in more detail below. Runtime 3054 is configured to execute code 3056 in response to an instruction to begin executing the machine learning model. Execution of code 3056 results in the generation of output (e.g., a prediction result), as described in more detail below.

[0213] In some embodiments, the runtime 3054 is the same as the runtime 3046 utilized by the virtual machine instance 3042. In some embodiments, the runtime 3054 is different from the runtime 3046 utilized by the virtual machine instance 3042.

[0214] In some embodiments, the model hosting system 140 uses one or more container images included in the deployment request (or container images retrieved from the container data store 3070 in response to the received deployment request) to create and initialize the ML scoring container 3050 in the virtual machine instance 3042. For example, the model hosting system 140 creates the ML scoring container 3050, which includes one or more container images and / or a top container layer.

[0215] As described above, in some embodiments, the user device can submit a deployment request and / or an execution request to the model hosting system 140 via the front end 3049. The deployment request causes the model hosting system 140 to deploy the trained machine learning model to the virtual machine instance 3042. For example, the deployment request can include an identification of an endpoint (e.g., an endpoint name, such as an HTTP endpoint name) and an identification of one or more trained machine learning models (e.g., the location of one or more model data files stored in the training model data store 3075). Optionally, the deployment request also includes an identification of one or more container images stored in the container data store 3070.

[0216] Upon receiving a deployment request, the model hosting system 140 initializes one or more ML scoring containers 3050 in one or more hosted virtual machine instances 3042. In embodiments where the deployment request includes the identification of one or more container images, the model hosting system 140 forms the one or more ML scoring containers 3050 from the one or more identified container images. For example, the container image identified in the deployment request can be the same container image used to form the ML training container 3030 used to train the machine learning model corresponding to the deployment request. Thus, the code 3056 of the one or more ML scoring containers 3050 includes one or more executable instructions within the one or more container images that represent an algorithm defining the machine learning model. In embodiments where the deployment request does not include the identification of a container image, the model hosting system 140 forms the one or more ML scoring containers 3050 from one or more container images stored in the container data store 3070 that are suitable for executing the one or more identified trained machine learning models. For example, a suitable container image can be a container image that includes executable instructions that represent an algorithm defining the one or more identified trained machine learning models.

[0217] In some embodiments, the model hosting system 140 also forms one or more ML score containers 3050 by retrieving model data corresponding to one or more identified trained machine learning models. For example, the deployment request may identify the location of one or more model data files stored in the trained model data store 3075. In embodiments where a single model data file is identified in the deployment request, the model hosting system 140 retrieves the identified model data file from the trained model data store 3075 and inserts the model data file into a single ML score container 3050, which forms part of the code 3056. In some embodiments, the model data file is archived or compressed (e.g., formed from a package of separate files). Accordingly, the model hosting system 140 unarchives or decompresses the model data file to obtain multiple separate files and inserts the separate files into the ML score container 3050. In some embodiments, the model hosting system 140 stores the model data file in the same location as the model data file was stored in the ML training container 3030 from which the model data file was generated. For example, the model data file is initially stored in the top container layer of the ML training container 3030 at a certain offset, and the model hosting system 140 subsequently stores the model data file in the top container layer of the ML scoring container 3050 at the same offset.

[0218] In an embodiment where multiple model data files are identified in the deployment request, the model hosting system 140 retrieves the identified model data files from the trained model data store 3075. The model hosting system 140 can insert the model data files into the same ML score container 3050, into different ML score containers 3050 initialized in the same virtual machine instance 3042, or into different ML score containers 3050 initialized in different virtual machine instances 3042. As an illustrative example, the deployment request can identify multiple model data files corresponding to different trained machine learning models because the trained machine learning models are related (e.g., the output of one trained machine learning model is used as the input of another trained machine learning model). Therefore, a user may desire to deploy multiple machine learning models to ultimately receive a single output that depends on the outputs of multiple machine learning models.

[0219] In some embodiments, the model hosting system 140 associates one or more initialized ML score containers 3050 with the endpoint identified in the deployment request. For example, each of the one or more initialized ML score containers 3050 can be associated with a network address. The model hosting system 140 can map the network address to the identified endpoint, and the model hosting system 140 or another system (e.g., a routing system, not shown) can store the mapping. Thus, the user device can reference one or more trained machine learning models stored in the one or more ML score containers 3050 using the endpoint. This allows the network address of the ML score container 3050 to change without causing the user operating the user device to change how the user references the trained machine learning model.

[0220] Once the one or more ML scoring containers 3050 are initialized, the one or more ML scoring containers 3050 are ready to execute one or more trained machine learning models. In some embodiments, the user device transmits an execution request to the model hosting system 140 via the front end 3049, where the execution request identifies an endpoint and includes input to the machine learning model (e.g., a set of input data). The model hosting system 140 or another system (e.g., a routing system, not shown) can obtain the execution request, identify the one or more ML scoring containers 3050 corresponding to the identified endpoint, and route the input to the one or more identified ML scoring containers 3050.

[0221] In some embodiments, virtual machine instance 3042 executes code 3056 stored in the identified ML scoring container 3050 in response to model hosting system 140 receiving the execution request. Specifically, execution of code 3056 causes executable instructions in code 3056 corresponding to the algorithm to read the model data file stored in ML scoring container 3050, use the input included in the execution request as input parameters, and generate corresponding output. As an illustrative example, the algorithm may include coefficients, weights, layers, cluster centroids, etc. The executable instructions in code 3056 corresponding to the algorithm may read the model data file to determine values ​​for the coefficients, weights, layers, cluster centroids, etc. The executable instructions may include input parameters, and the inputs included in the execution request may be provided as input parameters by virtual machine instance 3042. Given the machine learning model characteristics and the input parameters, execution of the executable instructions by virtual machine instance 3042 may be completed, thereby generating output.

[0222] In some embodiments, the virtual machine instance 3042 stores the output in the model prediction data store 3080. Alternatively or additionally, the virtual machine instance 3042 transmits the output via the front end 3049 to the user device that submitted the execution result.

[0223] In some embodiments, the execution request corresponds to a set of related trained machine learning models. Thus, the ML scoring container 3050 can transmit the output to a second ML scoring container 3050 initialized in the same virtual machine instance 3042 or in a different virtual machine instance 3042. The virtual machine instance 3042 that initialized the second ML scoring container 3050 can then execute the second code 3056 stored in the second ML scoring container 3050, thereby providing the received output as an input parameter to the executable instructions in the second code 3056. The second ML scoring container 3050 also includes a model data file stored therein, which is read by the executable instructions in the second code 3056 to determine the values ​​used to define the characteristics of the machine learning model. Execution of the second code 3056 produces a second output. The virtual machine instance 3042 that initialized the second ML scoring container 3050 can then transmit the second output to the model prediction data store 3080 and / or the user device via the front end 3049 (e.g., if no more trained machine learning models are needed to generate output) or transmit the second output to a third ML scoring container 3050 initialized in the same or different virtual machine instance 3042 (e.g., if output from one or more additional trained machine learning models is needed), and the above-mentioned process can be repeated with respect to the third ML scoring container 3050.

[0224] Although virtual machine instance 3042 is Figure 303042, some embodiments of the present application separate virtual machine instances 3042 that are actively assigned to execute tasks from those virtual machine instances 3042 that are not actively assigned to execute tasks. For example, those virtual machine instances 3042 that are actively assigned to execute tasks are grouped into an "active pool," while those virtual machine instances 3042 that are not actively assigned to execute tasks are placed in a "warming pool." In some embodiments, those virtual machine instances 3042 in the warming pool can be pre-initialized with the operating system, language runtime, and / or other software required to enable fast execution of tasks (e.g., fast initialization of one or more ML scoring containers 3050, fast execution of code 3056 in one or more ML scoring containers, etc.) in response to deployment and / or execution requests.

[0225] In some embodiments, the model hosting system 140 includes a processing unit, a network interface, a computer-readable medium drive, and an input / output device interface, all of which can communicate with each other via a communication bus. The network interface can provide connectivity to one or more networks or computing systems. Therefore, the processing unit can receive information and instructions from other computing systems or services (e.g., user devices, model training systems 120, etc.). The processing unit can also communicate to and from the memory of the virtual machine instance 3042 and also provide output information for an optional display via the input / output device interface. The input / output device interface can also receive input from an optional input device. The memory can contain computer program instructions (which are grouped into modules in some embodiments), and the processing unit executes the computer program instructions to implement one or more aspects of the present disclosure.

[0226] In some embodiments, the operating environment supports multiple different types of machine learning models, such as multi-armed bandit models, reinforcement learning models, ensemble machine learning models, deep learning models, and the like.

[0227] Figure 30 The model training system 120 and model hosting system 140 depicted in FIG are not intended to be limiting. For example, the model training system 120 and / or the model hosting system 140 may also be used in a system having a larger Figure 30 The present invention operates within a computing environment having fewer or more devices than those shown. Figure 30The depiction of the model training system 120 and / or model hosting system 140 in the accompanying drawings may be considered illustrative and not limiting of the present disclosure. For example, the model training system 120 and / or model hosting system 140 or its various components may implement various network service components, hosted or "cloud" computing environments, and / or peer-to-peer network configurations to implement at least a portion of the processes described herein. In some embodiments, the model training system 120 and / or model hosting system 140 is implemented directly in hardware or software executed by a hardware device and may, for example, include one or more physical or virtual servers implemented on physical computer hardware configured to execute computer-executable instructions for performing the various features described herein. The one or more servers may be geographically dispersed or geographically co-located, such as in one or more points of presence (POPs) or regional data centers.

[0228] Front-end 3029 processes all training requests received from user devices and provisions virtual machine instances 3022. In some embodiments, front-end 3029 serves as a front door to all other services provided by model training system 120. Front-end 3029 processes requests and ensures that the requests are properly authorized. For example, front-end 3029 can determine whether the user associated with the training request is authorized to initiate the training process.

[0229] Similarly, the front end 3049 processes all deployment and execution requests received from user devices and provisions virtual machine instances 3042. In some embodiments, the front end 3049 serves as a front door to all other services provided by the model hosting system 140. The front end 3049 processes requests and ensures that the requests are properly authorized. For example, the front end 3049 can determine whether the user associated with the deployment request or execution request is authorized to access the indicated model data and / or execute the indicated machine learning model.

[0230] The training data storage area 3060 stores training data and / or evaluation data. The training data can be data used to train the machine learning model and the evaluation data can be data used to evaluate the performance of the machine learning model. In some embodiments, the training data and the evaluation data have common data. In some embodiments, the training data and the evaluation data do not have common data. In some embodiments, the training data includes input data and expected outputs. Although the training data storage area 3060 is depicted as being located outside the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the training data storage area 3060 is located inside at least one of the model training system 120 or the model hosting system 140.

[0231] In some embodiments, training metrics data store 3065 stores model metrics. Although training metrics data store 3065 is depicted as being external to model training system 120 and model hosting system 140, this is not intended to be limiting. For example, in some embodiments (not shown), training metrics data store 3065 is internal to at least one of model training system 120 or model hosting system 140.

[0232] The container data store 3070 stores container images, such as those used to form the ML training container 3030 and / or the ML scoring container 3050, which can be retrieved by various virtual machine instances 3022 and / or 3042. Although the container data store 3070 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments (not shown), the container data store 3070 is internal to at least one of the model training system 120 and the model hosting system 140.

[0233] The trained model data store 3075 stores model data files. In some embodiments, some of the model data files consist of a single file, while other model data files are bundles of multiple separate files. Although the trained model data store 3075 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments (not shown), the trained model data store 3075 is internal to at least one of the model training system 120 or the model hosting system 140.

[0234] In some embodiments, the model prediction data store 3080 stores output (e.g., execution results) generated by the ML scoring container 3050. Although the model prediction data store 3080 is depicted as being external to the model training system 120 and the model hosting system 140, this is not intended to be limiting. For example, in some embodiments not shown, the model prediction data store 3080 is internal to at least one of the model training system 120 and the model hosting system 140.

[0235] Although the model training system 120, the model hosting system 140, the training data store 3060, the training indicator data store 3065, the container data store 3070, the training model data store 3075, and the model prediction data store 3080 are shown as separate components, this is not intended to be limiting. In some embodiments, any or all of these components can be combined to perform the functions described herein. For example, any or all of these components can be implemented by a single computing device, or by a plurality of different computing devices such as computer servers that are logically or physically grouped together to serve as a server system. Any or all of these components can communicate via a shared internal network, and the collective system (e.g., also referred to herein as a machine learning service) can communicate with one or more of the user devices via one or more networks 106.

[0236] Various exemplary user equipment are shown in Figure 30 , which includes desktop computers, laptop computers, and mobile phones, each of which is provided by way of illustration. Generally, a user device can be any computing device, such as a desktop computer, laptop computer or tablet computer, a personal computer, a wearable computer, a server, a personal digital assistant (PDA), a hybrid PDA / mobile phone, a mobile phone, an e-book reader, a set-top box, a voice command device, a camera, a digital media player, etc. In some embodiments, the model training system 120 and / or the model hosting system 140 provides one or more user interfaces, command line interfaces (CLIs), application programming interfaces (APIs), and / or other programming interfaces to the user device, which are used to submit training requests, deployment requests, and / or execution requests. In some embodiments, the user device can execute a standalone application that interacts with the model training system 120 and / or the model hosting system 140 to submit training requests, deployment requests, and / or execution requests.

[0237] In some embodiments, network 106 includes any wired network, wireless network or a combination thereof. For example, network 106 can be a personal area network, a local area network, a wide area network, an air broadcast network (for example, for radio or television), a cable network, a satellite network, a cellular telephone network or a combination thereof. As another example, network 106 can be a publicly accessible network of a link network that may be operated by various different parties, such as the Internet. In some embodiments, network 106 can be a private or semi-private network, such as a company or university intranet. Network 106 may include one or more wireless networks, such as a global system for mobile communications (GSM) network, a code division multiple access (CDMA) network, a long term evolution (LTE) network or any other type of wireless network. Network 106 can use protocols and components for communicating via any of the Internet or other aforementioned types of networks. For example, the protocols used by network 106 may include HTTP, HTTP security (HTTPS), message queue telemetry transport (MQTT), constrained application protocol (CoAP) etc. The protocols and components for communicating via the Internet or any of the other aforementioned types of communication networks are well known to those skilled in the art and are therefore not described in further detail herein.

[0238] The various embodiments discussed or proposed herein can be implemented in a wide variety of operating environments, which in some cases may include one or more user computers, computing devices, or processing devices that can be used to operate any of a number of applications. User or client devices may include any of a number of general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting a number of networking and messaging protocols. Such systems may also include a number of workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0239] Most embodiments utilize at least one network with which those skilled in the art will be familiar to support communications using any of a number of widely available protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Common Internet File System (CIFS), Extensible Messaging and Presence Protocol (XMPP), AppleTalk, etc. The one or more networks may include, for example, a local area network (LAN), a wide area network (WAN), a virtual private network (VPN), the Internet, an intranet, an extranet, a public switched telephone network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0240] In embodiments utilizing a web server, the web server may run any of a variety of server or middle-tier applications, including an HTTP server, a File Transfer Protocol (FTP) server, a Common Gateway Interface (CGI) server, a data server, a Java server, a flow application server, etc. The one or more servers may also be capable of responding to requests from user devices (such as by executing a program that may be implemented in any programming language such as The server(s) may further comprise one or more web applications (e.g., one or more scripts or programs written in a C, C#, or C++ language) or any scripting language (e.g., Perl, Python, PHP, or TCL), and combinations thereof) to execute the programs or scripts. The server(s) may further comprise a database server, including but not limited to commercially available database servers from Oracle(R), Microsoft(R), Sybase(R), IBM(R), etc. The database server may be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0241] The environment disclosed herein may include a variety of data storage areas and other memories and storage media as discussed above. These may reside in a variety of locations, such as on a storage medium local to (and / or residing in) one or more computers, or on a storage medium across a network away from any one or all of the computers. In a specific set of embodiments, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and / or remotely as appropriate. In the case where the system includes a computerized device, each such device may include hardware elements that may be electrically coupled via a bus, including, for example, at least one central processing unit ("CPU"), at least one input device (e.g., a mouse, keyboard, controller, touch screen, or keypad), and / or at least one output device (e.g., a display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0242] Such devices may also include a computer-readable storage medium reader, a communication device (e.g., a modem, a network card (wireless or wired), an infrared communication device, etc.), and a working memory, as described above. The computer-readable storage medium reader may be connected to or configured to receive a computer-readable storage medium, which represents a remote, local, fixed, and / or removable storage device and a storage medium for temporarily and / or more permanently containing, storing, transmitting, and retrieving computer-readable information. The system and various devices will typically also include a plurality of software applications, modules, services, or other elements located within at least one working memory device, including an operating system and applications, such as a client application or a web browser. It should be understood that alternative embodiments may have a large number of variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software, such as applets), or both. In addition, connections to other computing devices such as network input / output devices may be employed.

[0243] The storage medium and computer-readable medium used to contain the code or part of the code may include any suitable media known or used in the art, including storage media and communication media, such as but not limited to volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing and / or transmitting information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk memory (CD-ROM), digital versatile disk (DVD) or other optical storage devices, magnetic cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by the system device. Based on the disclosure and teachings provided herein, those skilled in the art will understand other ways and / or methods for implementing various embodiments.

[0244] In the above description, various embodiments are described. For illustrative purposes, specific configurations and details are set forth to provide a thorough understanding of the embodiments. However, it will be apparent to those skilled in the art that the embodiments may be practiced without these specific details. Furthermore, well-known features may be omitted or simplified in order not to obscure the described embodiments.

[0245] Bracketed text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used herein to illustrate optional operations that add additional features to some embodiments. However, this notation should not be taken to mean that these are the only options or optional operations, and / or that in some embodiments, boxes with solid borders are not optional.

[0246] In various embodiments, reference numerals with suffix letters may be used to indicate one or more instances of the referenced entity, and when multiple instances are present, each instance need not be identical, but may instead share some general characteristics or follow a convention. Furthermore, unless expressly stated to the contrary, the use of a particular suffix is ​​not meant to imply the presence of a particular number of entities. Thus, in various embodiments, two entities using the same or different suffix letters may or may not have the same number of instances.

[0247] References to "one embodiment," "an embodiment," "an exemplary embodiment," etc., indicate that the described embodiment may include a particular feature, structure, or characteristic, but not every embodiment may include the particular feature, structure, or characteristic. Furthermore, such phrases are not necessarily referring to the same embodiment. Furthermore, when a particular feature, structure, or characteristic is described in conjunction with an embodiment, it should be understood that it is within the knowledge of those skilled in the art to implement such feature, structure, or characteristic in conjunction with other embodiments, whether or not explicitly described.

[0248] Furthermore, in the various embodiments described above, unless specifically stated otherwise, disjunctive language such as the phrase "at least one of A, B, or C" is intended to be understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Thus, disjunctive language is generally not intended and should not be understood to imply that a given embodiment requires that at least one of A, at least one of B, or at least one of C each be present.

[0249] Unless expressly provided otherwise, articles such as "a" or "an" should generally be understood to include one or more of the described items. Thus, phrases such as "a device configured to..." are intended to include one or more of said devices. Such one or more of said devices may also be collectively configured to perform the stated statements. For example, "a processor configured to perform statements A, B, and C" may include a first processor configured to perform statement A working in conjunction with a second processor configured to perform statements B and C.

[0250] At least some embodiments of the disclosed technology can be described in terms of:

[0251] 1. A computer-implemented method comprising:

[0252] receiving, via one or more graphical user interfaces, one or more requests to train an anomaly detection machine learning model using feedback-based training, the anomaly detection machine learning model for finding anomalies in images, the requests indicating one or more of a type of analysis to perform, a model selection indication, and a configuration of a training dataset;

[0253] training the anomaly detection machine learning model based on the one or more requests using unlabeled training data based on the requests;

[0254] Perform feedback-based training on the trained anomaly detection machine learning model by at least:

[0255] applying the anomaly detection machine learning model to a set of unlabeled test data distinct from the unlabeled training data to generate a prediction of whether the unlabeled test data is anomalous,

[0256] presenting a graphical user interface showing a result of the prediction, the result of the prediction including at least an indication of which images of the test dataset were determined to be anomalous, an anomaly score for each item, and the image itself;

[0257] receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction,

[0258] updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and

[0259] Retrain the trained anomaly detection machine learning model using the updated training data.

[0260] 2. The computer-implemented method of clause 1, wherein the labeling of the tagged content is provided via a graphical user interface, and the label includes at least one of an indication of an abnormal state, a region of interest, and an orientation.

[0261] 3. The computer-implemented method of any of clauses 1-2, further comprising:

[0262] Using the retrained anomaly detection machine learning model to perform inference on unlabeled data; and

[0263] Results of the inference are provided in a graphical user interface, the graphical user interface including a sorting mechanism to sort the data by at least abnormal and normal.

[0264] 4. A computer-implemented method comprising:

[0265] receiving one or more requests to train an anomaly detection machine learning model using feedback-based training, the requests indicating one or more of a type of analysis to perform, a model selection indication, and a configuration of a training dataset;

[0266] using the training data to train the anomaly detection machine learning model based on the one or more requests; and

[0267] Perform feedback-based training on the trained anomaly detection machine learning model by at least:

[0268] applying the trained anomaly detection machine learning model to a set of test data different from the training data to generate a prediction of whether the test data is anomalous,

[0269] presenting a graphical user interface showing a result of the prediction, the result of the prediction including at least an indication of which items of the test dataset were determined to be anomalies, an anomaly score for each item, and the item itself;

[0270] receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction,

[0271] updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and

[0272] Retrain the trained anomaly detection machine learning model using the updated training data.

[0273] 5. The computer-implemented method of clause 4, wherein the training data comprises unlabeled anomalous and normal content.

[0274] 6. The computer-implemented method of clause 4, wherein the training data comprises labeled anomalous and normal content.

[0275] 7. A computer-implemented method as described in claim 6, wherein annotations of the marked content are provided via a graphical user interface, and the annotations include at least one of an indication of an abnormal state, an object, a region of interest, an alignment, an object shape, an object geometry, and an object position.

[0276] 8. The computer-implemented method of any of clauses 4-7, further comprising:

[0277] Using the retrained anomaly detection machine learning model to perform inference on unlabeled data; and

[0278] Results of the inference are provided in a graphical user interface, the graphical user interface including a sorting mechanism to sort the data by at least abnormal and normal.

[0279] 9. The computer-implemented method of any of clauses 4-7, further comprising:

[0280] Using the retrained anomaly detection machine learning model to perform inference on unlabeled data; and

[0281] Notify users when anomalies are identified based on a retrained anomaly detection machine learning model.

[0282] 10. The computer-implemented method of any of clauses 4-7, further comprising:

[0283] Use the retrained anomaly detection machine learning model to perform inference on unlabeled data;

[0284] receiving feedback regarding results of said inferences; and

[0285] The retrained anomaly detection machine learning model is retrained based on feedback received regarding the result of the inferring.

[0286] 11. The computer-implemented method of any of clauses 4-10, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an improper alignment, an indication of an appropriate exception result, and an indication of an improper exception result.

[0287] 12. The computer-implemented method of any of clauses 4-7, further comprising:

[0288] Using the retrained anomaly detection machine learning model to perform inference on unlabeled data; and

[0289] A mask is generated to overlay on images determined to have anomalies during inference.

[0290] 13. The computer-implemented method of any of clauses 4-7, further comprising:

[0291] Inference is performed on the unlabeled data using the retrained anomaly detection machine learning model, wherein an indication of an anomaly and a score for the anomaly are provided on the unlabeled data when the inference is performed.

[0292] 14. A computer-implemented method as described in any of clauses 4-13, wherein the retrained anomaly detection machine learning model is used to detect anomalies relative to one or more of object, texture, placement and orientation.

[0293] 15. A system comprising:

[0294] one or more first electronic devices for generating data to be consumed by the anomaly detection service; and

[0295] One or more second electronic devices, the one or more second electronic devices being configured to implement an anomaly detection service in a multi-tenant provider network, the anomaly detection service comprising instructions that, when executed, cause the anomaly detection service to:

[0296] receiving one or more requests to train an anomaly detection machine learning model using feedback-based training, the requests indicating one or more of a type of analysis to perform, a model selection indication, and a configuration of a training dataset;

[0297] using the training data to train the anomaly detection machine learning model based on the one or more requests; and

[0298] Perform feedback-based training on the trained anomaly detection machine learning model by at least:

[0299] applying the trained anomaly detection machine learning model to a set of test data different from the training data to generate a prediction of whether the test data is anomalous,

[0300] presenting a graphical user interface showing a result of the prediction, the result of the prediction including at least an indication of which items of the test dataset were determined to be anomalies, an anomaly score for each item, and the item itself;

[0301] receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction,

[0302] updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and

[0303] retraining the trained anomaly detection machine learning model using the updated training data; and

[0304] The retrained anomaly detection machine learning model is used to detect anomalies from the one or more first electronic devices.

[0305] 16. The system of clause 15, wherein the training data comprises labeled anomalous and normal content.

[0306] 17. The system of clause 15, wherein the training data comprises unlabeled anomalous and normal content.

[0307] 18. The system of any of clauses 15-17, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an improper alignment, an indication of an appropriate exception result, and an indication of an improper exception result.

[0308] 19. The system of any of clauses 15-18, wherein the anomaly detection service is further configured to:

[0309] Use the retrained anomaly detection machine learning model to perform inference on the unlabeled data; and

[0310] A mask is generated to overlay on images determined to have anomalies during inference.

[0311] 20. The system of any of clauses 15-19, wherein the retrained anomaly detection machine learning model is used to detect anomalies relative to one or more of object, texture, placement, and orientation.

[0312] 21. A computer-implemented method comprising:

[0313] receiving a request to perform feedback-based retraining, the request including one or more of an identifier of one or more models to be retrained, an identifier of a dataset to be used for retraining, an identifier of a dataset to be used for testing, an indication of a threshold for anomalies, an indication of how to display items to be verified, and an indication of a location to store historical information;

[0314] training multiple anomaly detection machine learning models using the at least partially annotated training dataset;

[0315] selecting an anomaly detection machine learning model from the trained plurality of scoring machine learning models based at least in part on the test metric;

[0316] applying the selected anomaly detection machine learning model to an unlabeled dataset to generate at least one of a score, a confidence of the score, and a ranking of the score by dataset item of the unlabeled dataset;

[0317] providing the results of applying the selected anomaly detection machine learning model to the unlabeled dataset to request feedback;

[0318] receiving requested feedback;

[0319] When the received requested feedback indicates a validation result, adding data from the unlabeled dataset to the training dataset;

[0320] retraining the anomaly detection scoring machine learning model using the training data and the added data from the unlabeled dataset; and

[0321] Deploy the retrained anomaly detection scoring machine learning model to perform inference on unlabeled images.

[0322] 22. The computer-implemented method of clause 21, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an appropriate exception result, and an indication of an inappropriate exception result.

[0323] 23. The computer-implemented method of any of clauses 21-22, wherein providing the results of applying the selected scoring machine learning model to the unlabeled dataset to request feedback comprises:

[0324] The dataset items are displayed in an order based at least in part on the rankings of the scores of the dataset items.

[0325] 24. A computer-implemented method comprising:

[0326] receiving a request to perform feedback-based retraining, the request including one or more of an identifier of one or more models to be retrained, an identifier of a dataset to be used for retraining, an identifier of a dataset to be used for testing, an indication of a threshold for anomalies, an indication of how to display items to be verified, and an indication of a location to store historical information;

[0327] selecting a scoring machine learning model from the trained plurality of scoring machine learning models based at least in part on the test metric;

[0328] applying the selected scoring machine learning model to an unlabeled dataset to generate at least one of a score and a confidence level for the score per dataset item of the unlabeled dataset;

[0329] providing the results of applying the selected scoring machine learning model to the unlabeled dataset to request feedback in the form of a graphical user interface;

[0330] receiving the requested feedback via the graphical user interface;

[0331] When the received requested feedback indicates a validation result, adding data from the unlabeled dataset to the training dataset; and

[0332] The selected scoring machine learning model is retrained using the training data with the added data from the unlabeled dataset.

[0333] 25. The computer-implemented method of clause 24, wherein the training data comprises an extended class of data that cannot vary between items in the dataset.

[0334] 26. The computer-implemented method of any of clauses 24-25, wherein the test metric is one or more of accuracy, precision, recall, receiver operating characteristic curve, and area under the curve.

[0335] 27. A computer-implemented method as described in any of clauses 24-26, wherein the scoring machine learning model is an anomaly detection model.

[0336] 28. The computer-implemented method of any of clauses 24-27, wherein the ranking of the scores is based at least in part on one or more of historical scores, historical feature vectors, historical confidences of the historical scores, and historical rankings.

[0337] 29. The computer-implemented method of any of clauses 24-28, wherein providing the results of applying the selected scoring machine learning model to the unlabeled dataset to request feedback comprises:

[0338] The dataset items are displayed in an order of the scores based at least in part on the scores of the dataset items.

[0339] 30. The computer-implemented method of clause 29, wherein data set items having less correct results are displayed first.

[0340] 31. The computer-implemented method of any of clauses 24-30, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an appropriate exception result, and an indication of an inappropriate exception result.

[0341] 32. The computer-implemented method of any of clauses 24-31, further comprising:

[0342] Use the retrained selected scoring machine learning model to perform inference on the unlabeled data;

[0343] providing the inferred result to request feedback;

[0344] receiving requested feedback;

[0345] When the received requested feedback indicates a validation result, adding data from the unlabeled data to the training dataset; and

[0346] The selected scoring machine learning model is retrained using the training data with the added data from the unlabeled data subject to inference.

[0347] 33. The computer-implemented method of any of clauses 24-32, wherein the training dataset comprises a plurality of images.

[0348] 34. A computer-implemented method as described in any of clauses 24-33, wherein in addition to retraining the selected scoring machine learning model using the training data with the added data from the unlabeled dataset, other scoring machine learning models among the multiple scoring machine learning models are retrained using the training data with the added data from the unlabeled dataset.

[0349] 35. A system comprising:

[0350] One or more first electronic devices, the one or more first electronic devices being used to implement a data storage service in a multi-tenant provider network; and

[0351] One or more second electronic devices, the one or more second electronic devices being configured to implement a machine learning model training service in a multi-tenant provider network, the machine learning model training service comprising instructions that, when executed, cause the machine learning model training service to:

[0352] receiving a request to perform feedback-based retraining, the request including one or more of an identifier of one or more models to be retrained, an identifier of a dataset to be used for retraining, an identifier of a dataset to be used for testing, an indication of a threshold for anomalies, an indication of how to display items to be verified, and an indication of a location to store historical information;

[0353] selecting a scoring machine learning model from the trained plurality of scoring machine learning models based at least in part on the test metric;

[0354] Applying the selected scoring machine learning model to an unlabeled dataset to generate at least one of a score and a confidence level for the score per dataset item of the unlabeled dataset

[0355] providing the results of applying the selected scoring machine learning model to the unlabeled dataset to request feedback in the form of a graphical user interface;

[0356] receiving the requested feedback via the graphical user interface;

[0357] When the received requested feedback indicates a validation result, adding data from the unlabeled dataset to the training dataset; and

[0358] The selected scoring machine learning model is retrained using the training data with the added data from the unlabeled dataset.

[0359] 36. The system of clause 35, wherein the training data comprises an extended class of data that cannot vary between items in the dataset.

[0360] 37. The system of any of clauses 35-36, wherein the test metric is one or more of accuracy, precision, recall, receiver operating characteristic curve, and area under the curve.

[0361] 38. A system as described in any of clauses 35-37, wherein the scoring machine learning model is an anomaly detection model.

[0362] 39. A system as described in any of clauses 35-38, wherein providing the results of applying the selected scoring machine learning model to the unlabeled dataset to request feedback includes: displaying the dataset items in an order of the scores based at least in part on the scores of the dataset items.

[0363] 40. The system of any of clauses 35-39, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an appropriate exception result, and an indication of an inappropriate exception result.

[0364] 41. A computer-implemented method comprising:

[0365] receiving a request to create a training dataset from at least one image, the request comprising an indication of the at least one image and at least one indication of an operation to be performed on the at least one image to generate a plurality of images from the at least one image;

[0366] Creating a training dataset of images by extracting one or more patches from the first at least one image according to the request by performing one or more of the following:

[0367] extracting same-sized, non-overlapping blocks from said first at least one image,

[0368] extracting different dimensions from said first at least one image,

[0369] extracting any identified regions of interest as blocks from the first at least one image, and

[0370] performing object detection on the first at least one image and extracting any detected objects of interest as blocks; and

[0371] receiving one or more requests to train an anomaly detection machine learning model using the created training dataset; and

[0372] Use the created training data to train an anomaly detection machine learning model based on one or more requests.

[0373] 42. The computer-implemented method of clause 41, wherein the region of interest is provided via a graphical user interface on an image from the same source as the at least one image.

[0374] 43. The computer-implemented method of any of clauses 41-42, wherein the image from the same source is a wide-angle image.

[0375] 44. A computer-implemented method comprising:

[0376] receiving a request to create a training dataset from at least one image, the request comprising an indication of the at least one image and at least one indication of an operation to be performed on the at least one image to generate a plurality of images from the at least one image;

[0377] creating a training data set by extracting one or more patches from the first at least one image according to the request; and

[0378] receiving one or more requests to train an anomaly detection machine learning model using the created training dataset; and

[0379] Use the created training data to train an anomaly detection machine learning model based on one or more requests.

[0380] 45. The computer-implemented method of clause 44, further comprising:

[0381] Object detection is performed on the at least one image to detect the object indicated in the request, the object detection being performed by a machine learning model, wherein the extracted one or more blocks include the detected object.

[0382] 46. ​​The computer-implemented method of any of clauses 44-45, wherein the region of interest to be extracted is provided via a graphical user interface on an image from the same source as the at least one image.

[0383] 47. The computer-implemented method of any of clauses 44-46, wherein the image from the same source is a wide-angle image.

[0384] 48. The computer-implemented method of any of clauses 44-47, further comprising:

[0385] Inference is performed on an unlabeled image using the trained anomaly detection machine learning model, wherein the unlabeled image is subjected to extraction of the same one or more patches as the at least first at least one image.

[0386] 49. The computer-implemented method of any of clauses 44-48, further comprising:

[0387] Creating a test data set by extracting one or more patches from a second at least one image different from the first at least one image by performing one or more of the following:

[0388] extracting same-sized, non-overlapping blocks from said second at least one image,

[0389] extracting different dimensions from the second at least one image,

[0390] extracting any identified regions of interest as blocks from the second at least one image, and

[0391] Object detection is performed on the second at least one image and any detected objects of interest are extracted as blocks.

[0392] 50. A computer-implemented method as described in clause 49, wherein the request is set to perform which of the following: extracting same-sized, non-overlapping blocks from the second at least one image, extracting different sizes from the second at least one image, extracting any identified areas of interest from the second at least one image as blocks, and performing object detection on the second at least one image and extracting any detected objects of interest as blocks.

[0393] 51. The computer-implemented method of any of clauses 44-50, further comprising:

[0394] The first at least one image is pre-processed by performing one or more of at least one geometric transformation, at least one photometric transformation, and normalization.

[0395] 52. The computer-implemented method of any one of clauses 44-51, further comprising

[0396] The extracted blocks are post-processed by performing one or more of pre-processing the first at least one image by performing one or more of at least one geometric transform, at least one photometric transform, and normalization.

[0397] 53. The computer-implemented method of any of clauses 44-52, wherein the normalization is to conform each patch to a training image of a different size.

[0398] 54. The computer-implemented method of any of clauses 44-53, further comprising:

[0399] Perform feedback-based training on the trained anomaly detection machine learning model by at least:

[0400] applying the trained anomaly detection machine learning model to a set of test data different from the training data set to generate a prediction of whether the unlabeled test data is anomalous,

[0401] receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction,

[0402] updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and

[0403] Retrain the trained anomaly detection machine learning model using the updated training data.

[0404] 55. The computer-implemented method of any of clauses 44-54, wherein the first at least one image is an image generated from one of a still camera, a video camera, and an x-ray machine.

[0405] 56. A system comprising:

[0406] One or more first electronic devices, the one or more first electronic devices being used to implement a data storage service in a multi-tenant provider network; and

[0407] One or more second electronic devices, the one or more second electronic devices being configured to implement a data set generation service in a multi-tenant provider network, the data set generation service comprising instructions that, when executed, cause the data set generation service to:

[0408] receiving a request to create a training dataset from at least one image, the request comprising an indication of the at least one image and at least one indication of an operation to be performed on the at least one image to generate a plurality of images from the at least one image;

[0409] creating a training data set by extracting one or more blocks from the first at least one image according to the request and storing the training data set in the data storage service; and

[0410] receiving one or more requests to train an anomaly detection machine learning model using the created training dataset; and

[0411] Use the created training data to train an anomaly detection machine learning model based on one or more requests.

[0412] 57. The system of clause 56, wherein the dataset generation service is a subset of anomaly detection service.

[0413] 58. The system of any of clauses 56-57, wherein the region of interest to be extracted is provided via a graphical user interface on an image from the same source as the at least one image.

[0414] 59. The system of any of clauses 56-58, wherein the image from the same source is a wide-angle image.

[0415] 60. A system as described in any of clauses 56-59, wherein the dataset generation service is further used to post-process the extracted blocks by performing one or more of pre-processing the first at least one image by performing one or more of at least one geometric transformation, at least one photometric transformation and normalization.

[0416] Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader scope of the disclosure as set forth in the claims.

Claims

1. A computer-implemented method comprising: receiving one or more requests to train an anomaly detection machine learning model using feedback-based training, the anomaly detection machine learning model for finding anomalies in images, the requests indicating one or more of a type of analysis to perform, a model selection indication, and a configuration of a training dataset, the training dataset comprising training data, the training data comprising images having defects; using the training data to train the anomaly detection machine learning model based on the one or more requests; as well as Perform feedback-based training on the trained anomaly detection machine learning model by at least: applying the trained anomaly detection machine learning model to a set of test data different from the training data to generate a prediction of whether the test data is anomalous, the test data comprising images, presenting a graphical user interface showing a result of the prediction, the result of the prediction including at least an indication of which items of the test dataset were determined to be anomalies, an anomaly score for each item, and the item itself; receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction, updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and Retrain the trained anomaly detection machine learning model using the updated training data.

2. The computer-implemented method of claim 1, wherein the training data comprises unlabeled anomalous and normal content.

3. The computer-implemented method of claim 1, wherein the training data includes labeled abnormal and normal content.

4. The computer-implemented method of claim 3, wherein the annotation of the marked content is provided via a graphical user interface, and the annotation includes at least one of an indication of an abnormal state, an object, a region of interest, an alignment, an object shape, an object geometry, and an object position.

5. The computer-implemented method of any one of claims 1 to 4, further comprising: Use the retrained anomaly detection machine learning model to perform inference on unlabeled data; as well as Results of the inference are provided in a graphical user interface, the graphical user interface including a sorting mechanism to sort the data by at least abnormal and normal.

6. The computer-implemented method of any one of claims 1 to 4, further comprising: Use the retrained anomaly detection machine learning model to perform inference on unlabeled data; as well as Notify users when anomalies are identified based on a retrained anomaly detection machine learning model.

7. The computer-implemented method of any one of claims 1 to 4, further comprising: Use the retrained anomaly detection machine learning model to perform inference on unlabeled data; receiving feedback regarding results of said inference; as well as The retrained anomaly detection machine learning model is retrained based on feedback received regarding the result of the inferring.

8. The computer-implemented method of any of claims 1-4, wherein the received feedback comprises one or more of an indication of a missed exception, an indication of an improper alignment, an indication of an appropriate exception result, and an indication of an improper exception result.

9. The computer-implemented method of any one of claims 1 to 4, further comprising: Use the retrained anomaly detection machine learning model to perform inference on unlabeled data; as well as A mask is generated to overlay on images determined to have anomalies during inference.

10. The computer-implemented method of any one of claims 1 to 4, further comprising: Inference is performed on the unlabeled data using the retrained anomaly detection machine learning model, wherein an indication of an anomaly and a score for the anomaly are provided on the unlabeled data when the inference is performed.

11. The computer-implemented method of any one of claims 1-4, wherein the retrained anomaly detection machine learning model is used to detect anomalies with respect to one or more of object, texture, placement, and orientation.

12. A system comprising: one or more first electronic devices for generating data to be consumed by the anomaly detection service; as well as One or more second electronic devices, the one or more second electronic devices being configured to implement an anomaly detection service in a multi-tenant provider network, the anomaly detection service comprising instructions that, when executed, cause the anomaly detection service to: receiving one or more requests to train an anomaly detection machine learning model using feedback-based training, the anomaly detection machine learning model for finding anomalies in images, the requests indicating one or more of a type of analysis to perform, a model selection indication, and a configuration of a training dataset, the training dataset comprising training data, the training data comprising images having defects; using the training data to train the anomaly detection machine learning model based on the one or more requests; and Perform feedback-based training on the trained anomaly detection machine learning model by at least: applying the trained anomaly detection machine learning model to a set of test data different from the training data to generate a prediction of whether the test data is anomalous, the test data comprising images, presenting a graphical user interface showing a result of the prediction, the result of the prediction including at least an indication of which items of the test dataset were determined to be anomalies, an anomaly score for each item, and the item itself; receiving feedback regarding the prediction, wherein the feedback includes an indication of the accuracy of the prediction, updating the training data to include labels of correctly predicted data for unlabeled test data based on the received feedback, and retraining the trained anomaly detection machine learning model using the updated training data; and The retrained anomaly detection machine learning model is used to detect anomalies from the one or more first electronic devices.

13. The system of claim 12, wherein the received feedback includes one or more of an indication of a missed exception, an indication of an improper alignment, an indication of an appropriate exception result, and an indication of an inappropriate exception result.

14. The system of any one of claims 12-13, wherein the anomaly detection service is further configured to: Use the retrained anomaly detection machine learning model to perform inference on the unlabeled data; and A mask is generated to overlay on images determined to have anomalies during inference.

15. The system of any one of claims 12-13, wherein the retrained anomaly detection machine learning model is used to detect anomalies with respect to one or more of object, texture, placement, and orientation.

Images