Computer Systems, Methods, and Devices for Active Learning

Abstract

Systems, methods, and devices for active learning are provided. The system includes a data storage device, a dataset analysis tool, a synthetic data generation module, an anomaly module, an image search engine module, an explainable AI module, an automated training platform, and optionally, a federated learning module. The system may be configured to operate on a general-purpose or dedicated computer and may further include a processor, memory, and a network interface. The system analyzes the provided dataset and generates synthetic data for augmenting the data within the provided dataset through the interaction of its components. The provided data and the generated data are used to train a machine learning model. The system may be operated iteratively and sequentially to continuously improve the machine learning model trained by the system by applying explainable artificial intelligence techniques with little or no human intervention.

Description

Technical Field

[0001] The following generally relates to machine learning and artificial intelligence, and more particularly to systems and methods for active learning.

Background Art

[0002] Introduction Current active learning systems and methods are limited in functionality and may provide relatively poor or unreliable performance. Such limitations may be exacerbated by inherent limitations in the samples and training datasets.

[0003] Therefore, there is a need for improved systems, methods, and devices for active learning that overcome at least some of the drawbacks of existing systems and methods.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Systems, methods, and devices are provided for implementing an active learning platform for training and deploying machine learning-based models.

Means for Solving the Problems

[0005] The platform is provided with an initial training dataset for training a machine learning model. The processor of the system is configured to execute a dataset analysis tool to perform preliminary analysis on the dataset. The dataset analysis may be by class. This analysis may include data sample size and sparsity calculations.

[0006] When the dataset analysis tool determines that the dataset is insufficient, the dataset analysis tool is configured to identify the insufficient areas within the dataset. The insufficient areas within the dataset can be described to the synthetic data generation component by the dataset analysis tool. The synthetic data generation component generates appropriate synthetic data to augment the dataset based on the information provided by the dataset analysis tool. The dataset analysis tool may be re-executed to re-evaluate the sufficiency of the augmented dataset.

[0007] If the augmented dataset is considered sufficient, the processor applies the augmented dataset to train a machine learning model and generates a trained machine learning model.

[0008] The processor is configured to evaluate the performance of the trained machine learning model by providing a validation dataset to the model. When certain samples of the validation dataset result in poor performance or incorrect output when provided to the trained machine learning model, these certain samples, or a portion of these certain samples, may be provided to a search engine module. The search engine module may be an image search engine. The search engine module scans a database to find data (e.g., images) similar to these certain samples. If such data is not available within the database, the system can call a synthetic data generation module to generate synthetic data similar to these certain samples. Then, the generated synthetic data is used to re-train the model.

[0009] This system may repeat this process until acceptable model performance is achieved. In some cases, acceptable model performance may be determined by a user, for example, by evaluating performance metrics rendered by the system to a graphical user interface. In some cases, acceptable model performance may be automatically determined by the system by referring to one or more performance metric threshold metrics. Once acceptable model performance is achieved, the model may be deployed or provided for further collaborative training and then deployed.

[0010] A computer-implemented method for training a machine learning model and performing active learning for deployment is provided. The method includes storing in a data storage device an image database including a first training data set of training samples; determining a clustering or distribution of the first training data set using a data set analysis tool executed by at least one processor communicating with the data storage device; in response to the determination of the clustering or distribution, generating a first set of one or more synthetic images using a subset of training samples from the first training data set as input by a synthetic image generation module executed by the at least one processor, and generating a second training data set including the first set of one or more synthetic images; training a machine learning model using the second training data set to obtain a trained machine learning model; evaluating the performance of the trained machine learning model using a model performance evaluation tool executed by the at least one processor, the evaluation including identifying one or more images in the second training data set that adversely affect model performance; sending a request for one or more training samples to an image search engine executed by the at least one processor, the requested training samples being defined using image data from the identified one or more images that adversely affect model performance; executing the image search engine to search the image database for the requested one or more training samples; if the image search engine returns the requested one or more training samples, generating a third training data set including the requested one or more training samples, and training the machine learning model using the third training data set to obtain the trained machine learning model;If the image search engine does not return one or more training samples requested: A step of using, as an input, image data from one or more images in a first training data set that contributed to unacceptable model performance by a synthetic image generation module to generate a second set of one or more synthetic images and generating a fourth training data set including the second set of one or more synthetic images; and a step of training a machine learning model using the fourth training data set to obtain a trained machine learning model.

[0011] Determining the clustering or distribution of the first training data set may include calculating the sample size and sparsity of data points in the first training data set.

[0012] Determining the clustering or distribution of the first training data set may include performing an analysis operation for placing training samples into clusters, where the training samples within each cluster include similarity, and quantifying the variation of the training samples within each cluster.

[0013] Determining the clustering or distribution of the first training data set may include the data set analysis tool discriminating that the ability to generalize across training samples is insufficient.

[0014] The synthetic image generation module may include a neural network and may be configured to receive one or more training samples of a certain image class and generate and output one or more synthetic images of the same image class.

[0015] Evaluating the performance of the trained machine learning model may include discriminating that the classification accuracy of the trained machine learning model does not meet a predetermined accuracy threshold.

[0016] The image database may include a plurality of indexed images, and the image search engine may be configured to analyze the input image and the plurality of indexed images and return references to images within the plurality of indexed images that are similar to the input image.

[0017] The image search engine may be configured to generate a feature embedded vector corresponding to the input image and use the feature embedded vector to search the plurality of indexed images.

[0018] The model performance evaluation tool may be configured to detect one or more regions within one or more images in a second training dataset that have an adverse effect on model performance, and the input image to the image search engine may be a cropped image that includes the detected one or more regions.

[0019] The machine learning model may be configured to perform at least one computer vision task including any one or more of object detection, object tracking, image classification, semantic segmentation, and instance segmentation.

[0020] Evaluating the performance of a trained machine learning model may include generating, by a model performance evaluation tool, an output including any one or more of a prediction, a confidence level, a heatmap, and other information describing how the input affects the activation information of the internal and various layers of the trained machine learning model.

[0021] Evaluating the performance of a trained machine learning model may include measuring, by a model performance evaluation tool, the model output accuracy of the trained machine learning model against a known sample set, and the measuring may include measuring the performance of the trained machine learning model against the known samples and comparing the performance with one or more predetermined thresholds of precision, recall, and IoU (intersection over union).

[0022] The method may further include using at least one federated learning module executed by the at least one processor to execute a federated learning process using the trained model and additional training data to further train the trained machine learning model to obtain a federated trained machine learning model, and the method can further include evaluating the performance of the federated trained machine learning model using a model performance evaluation tool.

[0023] The additional training data may be from at least two physical sites performing computer vision-based visual inspections.

[0024] A computer system for performing active learning to train and deploy a machine learning model is also provided. The system includes a data storage device for storing an image database including a first training dataset of training samples; and at least one processor in communication with the data storage device. The at least one processor: uses a dataset analysis tool to determine the clustering or distribution of the first training dataset; in response to the determination of the clustering or distribution, uses a subset of training samples from the first training dataset as input by a synthetic image generation module to generate a first set of one or more synthetic images and generate a second training dataset including the first set of one or more synthetic images; uses an automated training module to train a machine learning model using the second training dataset to obtain a trained machine learning model; uses a model performance evaluation tool to evaluate the performance of the trained machine learning model, the evaluation including identifying one or more images in the second training dataset that adversely affect the model performance; sends a request for one or more training samples to an image search engine, the requested training samples being defined using image data from the identified one or more images that adversely affect the model performance; executes the image search engine to search the image database for the requested one or more training samples; if the image search engine returns the requested one or more training samples: generates a third training dataset including the requested one or more training samples and trains the machine learning model using the third training dataset to obtain the trained machine learning model;If the image search engine does not return one or more training samples requested: The synthetic image generation module uses, as input, image data from one or more images in a first training data set that contributed to unacceptable model performance, to generate a second set of one or more synthetic images, and to generate a fourth training data set that includes the second set of one or more synthetic images; and training a machine learning model using the fourth training data set to obtain a trained machine learning model.

[0025] Determining the clustering or distribution of the first training data set may include performing an analysis operation to place training samples into clusters, where the training samples within each cluster contain similarity, and quantifying the variation of the training samples within each cluster.

[0026] The synthetic image generation module may include a neural network and may be configured to receive one or more training samples of an image class and generate and output one or more synthetic images of the same image class.

[0027] Evaluating the performance of the trained machine learning model may include determining that the classification accuracy of the trained machine learning model does not meet a predetermined accuracy threshold.

[0028] The image database may include a plurality of indexed images, and the image search engine may be configured to analyze the input image and the plurality of indexed images and return references to images within the plurality of indexed images that are similar to the input image.

[0029] Provided is a non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to execute a method. The method includes: storing in a data storage device an image database including a first training data set of training samples; determining a clustering or distribution of the first training data set using a data set analysis tool executed by at least one processor communicating with the data storage device; in response to the determination of the clustering or distribution, generating a first set of one or more synthetic images using, as an input, a subset of training samples from the first training data set by a synthetic image generation module executed by the at least one processor, and generating a second training data set including the first set of one or more synthetic images; training a machine learning model using the second training data set to obtain a trained machine learning model; evaluating the performance of the trained machine learning model using a model performance evaluation tool executed by the at least one processor, the evaluation including identifying one or more images within the second training data set that have an adverse effect on model performance; transmitting a request for one or more training samples to an image search engine executed by the at least one processor, the requested training samples being defined using image data from the identified one or more images that have an adverse effect on model performance; executing the image search engine to search the image database for the requested one or more training samples, and if the image search engine returns the requested one or more training samples, generating a third training data set including the requested one or more training samples and training the machine learning model using the third training data set to obtain the trained machine learning model;If the image search engine does not return one or more requested training samples: generating, by a synthetic image generation module, a second set of one or more synthetic images using, as input, image data from one or more images in a first training data set that contributed to unacceptable model performance, and generating a fourth training data set that includes the second set of one or more synthetic images; and training a machine learning model using the fourth training data set to obtain a trained machine learning model.;

[0030] Other aspects and features will become apparent to those of ordinary skill in the art upon review of the following description of some exemplary embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] The drawings included herein are for purposes of illustrating various examples of the articles, methods, and apparatuses of this specification.

[0032]

Figure 1

[0033]

Figure 2

[0034]

Figure 3

[0035]

Figure 4

[0036]

Figure 5

Best Mode for Carrying Out the Invention

[0037] Various devices or processes are described below to provide examples of each claimed embodiment. The embodiments described below do not limit the claimed embodiments, and the claimed embodiments may cover processes or devices different from those described below. The claimed embodiments are not limited to a device or process having all the features of any one device or process described below, or to features common to a plurality or all of the devices described below.

[0038] One or more of the systems described herein may be implemented with a computer program executed on a programmable computer, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, but not limited to, a programmable computer may be a programmable logic unit, a mainframe computer, a server, and a personal computer, a cloud-based program or system, a laptop, a personal data assistant, a mobile phone, a smartphone, or a tablet device.

[0039] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language for communicating with a computer system. However, the program can be implemented in assembly language or machine language, if desired. In either case, the language can be a compiled or interpreted language. Each such computer program is preferably stored in a storage medium or device readable by a general or special purpose programmable computer, and when the storage medium or device is read by a computer, configures and operates the computer to perform the procedures described herein.

[0040] The description of embodiments having several components communicating with each other does not imply that all such components are required. On the contrary, various optional components are described to illustrate a wide variety of possible embodiments of the present invention.

[0041] Furthermore, process steps, method steps, algorithms, etc. may be described in sequential order (in the present disclosure and / or the claims), but such processes, methods, and algorithms may be configured to function in an alternative order. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of the processes described herein may be performed in virtually any order. Additionally, some steps may be performed simultaneously.

[0042] It will readily be apparent that when a single device or article is described herein, two or more devices / articles (regardless of whether they cooperate) may be used in place of the single device / article. Similarly, it will readily be apparent that when two or more devices or articles are described herein (regardless of whether they cooperate), a single device / article may be used in place of the two or more devices / articles.

[0043] The following generally relates to an active learning platform for training and deploying machine learning-based models.

[0044] Machine learning models can be deployed for use in a variety of tasks. For example, a machine learning model may be applied to speech recognition, text translation, image classification, object inspection, and other tasks.

[0045] Generally, such machine learning models can be generated by training an untrained machine learning model with a training dataset. In some use cases, the training dataset may be limited or insufficient in some way, such as having a limited number of training samples, and as a result, the performance of the trained model may be poor. It can be difficult to detect before deployment whether the model training dataset is such that the model can be trained to provide high performance.

[0046] Furthermore, in cases where the training dataset is insufficient, it can be difficult to determine the reason the training dataset is insufficient or how to improve the training dataset so that it is sufficient. Finally, if it is known how and why a dataset is insufficient, it can be difficult to easily and efficiently augment the dataset and correct its deficiencies.

[0047] Described herein are systems and related methods for an active learning platform. An initial training dataset for training a machine learning model may be provided to the platform. The platform may first perform a preliminary analysis on the dataset for each class. This analysis may include data sample size and sparsity calculations.

[0048] If such a dataset analysis determines that the dataset is insufficient, the insufficient areas may be identified and described to the synthetic data generation component, whereby appropriate synthetic data may be generated to augment the dataset. The dataset analysis tool may be re-executed to re-evaluate the sufficiency of the augmented dataset.

[0049] If the augmented dataset is deemed sufficient, the augmented dataset is applied to train a machine learning model, and a trained machine learning model can be generated.

[0050] The trained machine learning model can be evaluated for performance by providing a validation dataset to the model. If certain samples of the validation dataset result in poor performance or inaccurate output when provided to the trained machine learning model, these certain samples, or a portion of these certain samples, may be provided to a search engine module. The search engine module may scan a database to locate data similar to these certain samples. If such data is not available, the system can call a synthetic data generation module to generate synthetic data similar to these certain samples and use the generated synthetic data to retrain the model.

[0051] This process can be repeated until an acceptable model performance is achieved. The acceptable model performance may be determined by the user, for example, by evaluating performance metrics rendered in a graphical user interface by the system, or may be automatically determined by the system by referring to one or more performance metric threshold metrics. Once an acceptable model performance is achieved, the model may be deployed, or provided for further co-training and then deployed.

[0052] Such systems and related methods can improve the performance of machine learning models, particularly in instances where the training dataset is inherently limited or difficult to obtain. The system allows for retraining, visibility, and automation across the entire artificial intelligence pipeline.

[0053] Much of this disclosure is provided in the context of object defect detection and visual inspection (including manufacturing quality control and visual inspection), but the systems, methods, and devices provided herein may have additional uses and different applications beyond those described herein, whether in the context of object defect detection and visual inspection or not (e.g., other computer vision applications such as autonomous vehicles, medical image analysis, robotics using manipulation). The machine learning models described herein may, in other embodiments, be other forms of machine learning models configured to perform machine learning or computer vision tasks other than object detection, whether called a model or an object detection model. For example, the multi-model architecture described herein may include multiple neural networks configured to perform object detection or other image processing or computer vision tasks. The input data and output data may vary in those cases, but the elements of this disclosure such as multiple models and trigger conditions, and the data aggregation at the end of the processes disclosed herein, may operate similarly.

[0054] As used herein, the term "object detection" is generally intended to refer to computer vision techniques in which objects are detected or identified within a digital image. The term "object detection" as used in this disclosure includes, but is not intended to be limited to, specific computer vision techniques of "object detection" in which all instances of known object classes are localized and classified within a digital image. For example, the term "object detection" as used herein is intended to include image segmentation techniques in which the presence of objects within a digital image is marked using a pixel-wise mask for each object within the image. One particular example of image segmentation is instance segmentation in which objects within a digital image are detected and segmented via localization of specific objects and association of the pixels to which they belong. Instance segmentation involves identifying each object instance for all known objects within a digital image and involves assigning a label to each pixel of the digital image. Thus, references to "models", "object detection models", "neural networks", "object detection neural networks", etc. are intended to include embodiments in which instance segmentation models or neural networks are used and embodiments in which "object detection" models or neural networks are used.

[0055] As described herein, in one embodiment, the present disclosure provides a multi-model architecture that includes a plurality of neural networks configured to receive input data and generate at least one output. The neural network can be a feed-forward neural network. The neural network may have a plurality of processing nodes. The processing nodes can include a multi-variable input layer having a plurality of input nodes, at least one hidden layer of nodes, and an output layer having at least one output node. During operation of the neural network, each of the nodes in the hidden layer applies an activation / transfer function and weights to any input arriving at that node (from the input layer or from another layer of the hidden layer). The node can provide an output to other nodes (in a subsequent hidden layer or in the output layer). The neural network can be configured to perform a regression analysis that provides a continuous output, or a classification analysis for classifying data. The neural network can be trained using supervised or unsupervised learning techniques as described below. According to one supervised learning technique, a training data set is provided at the input layer in relation to a set of known output values at the output layer. During the training phase, the neural network can process the training data set. The neural network is intended to learn how to provide an output for new input data by generalizing the information learned in the training phase from the training data. Training can be performed by backpropagating the error to determine the weights of the nodes in the hidden layer to minimize the error. Once trained, or optionally during training, test or validation data can be provided to the neural network and an output can be provided. The neural network can thus cross-correlate the inputs provided to the input layer and provide at least one output at the output layer. In each embodiment, the output provided by the neural network is preferably close to the desired output for a given input such that the neural network satisfactorily processes the input data.

[0056] Referring now to FIG. 1, an active learning platform system 10 according to an embodiment is shown. System 10 includes, via network 20, a data storage device 12, an active learning server platform 14, an operator device 16, and an edge cloud mobile device 18.

[0057] Devices 12, 14, 16, 18 may be server computers, node computing devices (e.g., JETSON computing devices, etc.), embedded devices, desktop computers, notebook computers, tablets, PDAs, smartphones, or other computing devices. Devices 12, 14, 16, 18 can include a connection to network 20, such as a wired or wireless connection to the Internet. In some cases, network 20 can include other types of computers or telecommunications networks. Devices 12, 14, 16, 18 can include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. The memory can include random access memory (RAM) or a similar type of memory. Also, the memory can store one or more applications for execution by the processor. The applications can correspond to software modules comprising computer-executable instructions for performing the functions described below. The secondary storage device can include a hard disk drive, a floppy disk drive, a CD drive, a DVD drive, a Blu-ray drive, or other types of non-volatile data storage. The processor can execute an application, computer-readable instructions, or a program. The application, computer-readable instructions, or program can be stored in the memory or secondary storage, or can be received from the Internet or other network 20.

[0058] The input device may include any device for inputting information to devices 12, 14, 16, 18. For example, the input device may be a keyboard, keypad, cursor control device, touch screen, camera, or microphone. The display device may include any type of device for presenting visual information. For example, the display device may be a computer monitor, flat screen display, projector, or display panel. The output device may include any type of device for presenting a hard copy of information, such as a printer. The output device may also include other types of output devices, such as speakers. In some cases, devices 12, 14, 16, 18 may include one or more of a processor, application, software module, secondary storage device, network connection, input device, output device, and display device.

[0059] Devices 12, 14, 16, 18 are described as having various components, but those skilled in the art will understand that devices 12, 14, 16, 18 may, in some cases, include fewer, additional, or different components. Further, although aspects of the implementation of devices 12, 14, 16, 18 are described as being stored in memory, these aspects may also be stored in or read from other types of computer program products or computer-readable media, such as secondary storage devices including hard disks, floppy disks, CDs, or DVDs; carrier waves from the Internet or other networks; or other forms of RAM or ROM. Those skilled in the art will understand that the computer-readable media may include instructions for controlling devices 12, 14, 16, 18 and / or the processor to perform a particular method.

[0060] Devices 12, 14, 16, and 18 can be described as performing certain actions. It will be understood that any one or more of these devices can perform actions automatically or in response to an interaction by the user of the device. That is, the user of the device can operate one or more input devices (e.g., a touch screen, a mouse, or a button) to cause the device to perform the described actions. In many cases, this aspect will not be described below but will be understood.

[0061] As an example, it is described below that devices 12, 14, 16, and 18 can send information to one or more other devices 12, 14, 16, and 18. For example, a user using the operator device 16 can operate one or more inputs (e.g., a mouse and a keyboard) to interact with a user interface displayed on the display of the device 16. Generally, the device may receive the user interface from the network 20 (e.g., in the form of a web page). Alternatively, or in addition, the user interface can be stored locally on the device (e.g., in the cache of a web page or a mobile application).

[0062] Devices 12, 14, 16, and 18 can be configured to receive multiple information from one or more of the multiple devices 12, 14, 16, and 18.

[0063] In response to receiving information, each of the devices 12, 14, 16, 18 may store the information in a storage database. The storage may correspond to the secondary storage devices of one or more other devices 12, 14, 16, 18. Generally, the storage database may be any suitable storage device such as a hard disk drive, solid state drive, memory card, or disk (e.g., CD, DVD, or Blu-ray, etc.). Also, the storage database may be locally connected to the devices 12, 14, 16, 18. In some cases, the storage database may be located remotely from the devices 12, 14, 16, 18 and may be accessible to the devices 12, 14, 16, 18 via, for example, a network. In some cases, the storage database may include one or more storage devices located at a network-connected cloud storage provider.

[0064] The data storage device 12 may be a cloud-accessible data storage device that can store data for retrieval by other components of the system 10 through the network 20.

[0065] The active learning server platform 14 can communicate with the data storage device 12 to obtain training data for training a machine learning model. The platform 14 can communicate with the data storage device 12 to obtain a pre-trained machine learning model for further training. Once the active learning platform trains a machine learning model, the active learning server platform 14 may send the model to the edge cloud mobile device 18 through the network 20 for model deployment.

[0066] The operator device 16 includes a computer terminal that can be used by a human operator to control the system 10. The operator device 16 can be coupled to the active learning server platform 14 such that the operator device 16 can adjust parameters and control the operation of the active learning server platform 14. The operator device 16 includes user interface components (or modules) (e.g., a human-machine interface).

[0067] The user interface components of the operator device 16 can also render one or more user interface elements for receiving input from the operator or for displaying the output of the system 10 (e.g., model performance metrics). For example, the user interface components may provide yes / no or similar binary options for receiving user input data indicating option selections.

[0068] The edge cloud mobile device 18 is configured to deploy a trained machine learning module trained by the active learning server platform 14. The edge cloud mobile device 18 includes a general-purpose or dedicated computing device that can execute the trained machine learning model. The edge cloud mobile device 18 can communicate with the active learning server platform 14 to receive the trained machine learning model from the active learning server platform 14. The device 18 can send model performance data indicating the performance of the deployed model to the platform 14. In some cases, the received model performance data can be used by the platform 14 when retraining the model using the active learning platform.

[0069] Referring now to FIG. 2, FIG. 2 shows a simplified block diagram of the components of a computing device 1000, such as a mobile device or a portable electronic device, according to an embodiment. The software modules described in the disclosure herein may be configured to be executed on a computing device such as device 1000 of FIG. 2. Device 1000 includes a plurality of components such as a processor 1020 that controls the operation of device 1000. Communication functions, including data communication, voice communication, or both, may be performed through communication subsystem 1040. Data received by device 1000 may be decompressed and decoded by decoder 1060. Communication subsystem 1040 may receive messages from wireless network 1500 and transmit messages to wireless network 220.

[0070] Wireless network 1500 may be any type of wireless network including, but not limited to, a data-centric wireless network, a voice-centric wireless network, and a dual-mode network that supports both voice communication and data communication.

[0071] Device 1000 may be a battery-powered device and, as shown, includes a battery interface 1420 for receiving one or more rechargeable batteries 1440.

[0072] Processor 1020 also interacts with additional subsystems such as random access memory (RAM) 1080, flash memory 1100, display 1120 (e.g., having a touch sensor overlay 1140 connected to an electronic controller 1160 that together form a touch sensor display 1180), actuator assembly 1200, one or more optional force sensors 1220, auxiliary input / output (I / O) subsystem 1240, data port 1260, speaker 1280, microphone 1300, short-range communication system 1320, and other device subsystems 1340.

[0073] In some embodiments, user interaction with the graphical user interface may be performed through the touch sensor overlay 1140. The processor 1020 may interact with the touch sensor overlay 1140 via the electronic controller 1160. Information such as text, characters, symbols, images, icons, and other items that can be displayed or rendered on the portable electronic device and generated by the processor 102 may be displayed on the touch sensor display 118.

[0074] The processor 1020 can also interact with the accelerometer 1360 as shown in FIG. 2. The accelerometer 1360 can be used to detect the direction of gravity or the reaction force induced by gravity.

[0075] To identify a subscriber for network access according to this embodiment, the device 1000 may use a subscriber identification information module or a removable user identification information module (SIM / RUIM) card 1380 inserted into the SIM / RUIM interface 1400 for communication with a network (such as the wireless network 1500). Alternatively, the user identification information may be programmed in the flash memory 1100 or implemented using other techniques.

[0076] The device 1000 also includes an operating system 1460 and software components 1480 that are executed by the processor 1020 and can be stored in a persistent data storage device such as the flash memory 1100. Additional applications may be loaded onto the device 1000 through the wireless network 1500, the auxiliary I / O subsystem 1240, the data port 1260, the short-range communication subsystem 1320, or any other suitable device subsystem 1340.

[0077] For example, during use, received signals such as text messages, email messages, web page downloads, or other data can be processed by the communication subsystem 1040 and input to the processor 1020. The processor 1020 then processes the received signals for output to the display 1120 or alternatively to the auxiliary I / O subsystem 1240. A subscriber can also create data items such as email messages that can be transmitted over the wireless network 1500, for example, through the communication subsystem 1040.

[0078] For voice communication, the overall operation of the portable electronic device 1000 can be similar. The speaker 1280 may output audible information converted from an electrical signal, and the microphone 1300 may convert audible information into an electrical signal for processing.

[0079] Referring now to FIG. 3, a block diagram of a computing system 300 for performing active learning according to an embodiment is shown. The system 300 includes a data store 302, a dataset analysis tool 304, synthetic data generation 306, an anomaly model 308, an image search engine 310, an automated training platform 312, an explainable AI module 314, an federated learning module 316, a deployed model 318, an edge cloud mobile 320, and an active learning platform 322.

[0080] Components 302, 304, 306, 308, 310, 312, 314, and 316 include software modules that can be executed on any general-purpose computing device. In some examples, all of these components can be executed together by a single general-purpose computing device. In other examples, each software module may be executed on a separate general-purpose computing device, and each general-purpose computing device is network-connected such that components can pass data to other components.

[0081] The active learning platform 322 includes modules 302, 304, 306, 308, 310, 312, and 314. The active learning platform 322 may further include a management module (not shown) that can manage the operations and interactions of the modules including the active learning platform 322. The management module may be coupled to an analyst terminal such that a human analyst can adjust the operating parameters of the active learning platform 322 and control the operation of the active learning platform 322. For example, the analyst may specify a subset of the data present in the data store 302 for further processing, start and stop model training, or adjust model hyperparameters. The management module may be implemented on the operator device 16 of the system 10.

[0082] The data store 302 may include a cloud-hosted network-accessible non-transitory computer-readable memory.

[0083] The data store 302 may include a hard drive, NAND flash, or other non-transitory computer-readable memory.

[0084] The data store 302 can store any form of data including, but not limited to, source code, image data, text data, and video data.

[0085] In some examples, the data store 302 may be privately hosted. For example, an entity may privately own a server device that can be connected to a network such as the Internet, whereby the data store 302 may be accessible to any authorized individual or entity having access to the network.

[0086] In other examples, data storage 302 can include a cloud storage instance provided by a commercial cloud service provider such as Microsoft Azure, Amazon Web Services, or Google Cloud.

[0087] The data storage 302 can store a data set that includes a plurality of data samples. The data samples can include images, videos, text, or other data formats. The data samples can include training data samples, validation data samples, and input data samples. The data samples may include multiple data classes, and the classes can be defined, for example, as image types, image subjects, or specific features within an image in an image classification use case. Each data sample may be further associated with metadata such as a class label or other metadata.

[0088] The data set analysis tool 304 includes a software module configured to analyze a data set of images.

[0089] The tool 304 can cluster the provided image data samples into broad categories or narrow categories, calculate statistics regarding the generated data clusters, and, depending on the calculated statistics, direct further actions on the provided image data.

[0090] The statistics calculated by tool 304 may be used to indicate whether the dataset is a sufficient dataset for training a machine learning model (e.g., the platform can learn sufficiently from the dataset) or an insufficient or inappropriate dataset. Sufficiency can be measured, for example, by sectioning and / or sparsity. If the dataset is not sufficient, the dataset needs to be enhanced / improved so that the model learned through the training process on that dataset provides acceptable model performance. For example, if the model is an image classification model, the model may be considered to provide acceptable performance when the accuracy of image classification exceeds a predetermined accuracy threshold.

[0091] After clustering, tool 304 can be configured to determine whether any class of the dataset lacks sparsity or whether any data class lacks data samples.

[0092] Based on the output of the analysis, tool 304 may identify one or more classes of interest. In such an example, tool 304 can output data that describes one or more classes of the required data. Such output may include class labels or descriptions (e.g., text of the class or metadata format description), or images (or multiple images) from the class.

[0093] The dataset analysis tool 304 is coupled to the data store 302 such that the dataset analysis tool 304 can access and process the data stored in the data store 302.

[0094] The dataset analysis tool 304 can receive a plurality of images from the data store 302.

[0095] The dataset analysis tool 304 may perform an analysis operation to place images into clusters, and the images within each cluster contain similarities.

[0096] The dataset analysis tool 304 may use various methods for clustering images, such as, but not limited to, k-means, SVM, random forest clustering, and UMAP architecture.

[0097] Once the dataset analysis tool 304 clusters the images into categories, the dataset analysis tool 304 can proceed to calculate statistics for each cluster.

[0098] For example, the dataset analysis tool 304 can determine whether each cluster contains a sufficient number of images so that a machine learning-based inspection tool trained using each cluster can accurately identify the images of each cluster. In addition to the number of images in each cluster, the dataset analysis tool 304 may quantify the variation of the images within the cluster. For example, a cluster may contain five images that are all approximately the same (e.g., five different car images of the same type, model, and color). Such a cluster can be considered to have relatively low variation within the cluster.

[0099] In another example, a cluster may contain five images that are each very different (e.g., five car images each having a different type, model, and color). Such a cluster can be considered to have relatively high variation.

[0100] To quantify the variation of the images, various image analysis tools and algorithms can be applied.

[0101] The above description with reference to the dataset analysis tool 304 has focused on image data, but in other embodiments, the dataset analysis tool 304 may be configured to process other forms of data, such as video data.

[0102] The synthetic data generation module 306 comprises a software module configured to generate synthetic data.

[0103] In some examples, the generated synthetic data may include image data. In such examples, the synthetic data generation module may be provided with a training image dataset, and the synthetic data generation module 306 may generate image data similar to the training image dataset. For example, if the training image dataset includes images of a certain image class, the synthetic data generation module 306 may generate and output images of the same image class.

[0104] The synthetic data generation module 306 may include a trained machine learning model. The machine learning model may be a neural network.

[0105] In some examples, the synthetic data generation module 306 may comprise at least one generative adversarial network (GAN). The GAN may be trainable using the provided image dataset.

[0106] The synthetic data generation module 306 is coupled to the data storage 302 such that the synthetic data generation module 306 can access and process the data stored in the data storage 302.

[0107] Multiple data, such as multiple image files, may be provided to the synthetic data generation module 306 for training the synthetic data generation module 306.

[0108] Once trained, the synthetic data generation module 306 can be instructed to output a plurality of synthetic images similar to the provided input training images.

[0109] The synthetic data generation module 306 can be configured to output the synthetic images to the data storage 302 for data storage of these synthetic images.

[0110] The anomaly model 308 includes a software module configured to receive a plurality of training set images as training inputs. The training set images can preferably include images of image subjects in a manually verified acceptable state.

[0111] Once the anomaly model 308 is trained, the anomaly model 308 is configured to receive an input image and perform an analysis on the input image to determine whether the input image contains an image anomaly. In some examples, the anomaly can include features that are not generally present in the training set images.

[0112] The anomaly model 308 can employ any data anomaly model or data anomaly detection algorithm known in the art to detect anomalies within the input image.

[0113] If the anomaly model 308 detects an anomaly in the image, the anomaly model 308 can then identify a general area within the image where the anomaly exists. The anomaly model 308 can then crop this area to generate a cropped anomaly image for further processing or use, or draw a bounding box around the area where the anomaly exists and record the position information of the bounding box for further processing or use.

[0114] In some examples, the anomaly model 308 can be further configured to classify objects (e.g., defects) located within the image.

[0115] In some examples, the anomaly model 308 can include a generative model sub-component, a comparison module sub-component, and an anomaly classifier module sub-component.

[0116] In some examples, the generative model sub-component can include a neural network. In some examples, the neural network can be an adversarial generative network (GAN).

[0117] The generative model may be trained using training sample data considered to include "good" samples. For example, if the system 300 is configured to process image data, the sample data can include "good" images, where the images include only desirable features or a high percentage of desirable features. Once trained, the generative model may be provided with an image of the same class as the training sample images as input and can output an image with "good" variations of the input image.

[0118] In such examples, once the anomaly model 308 is trained with such an image dataset, the anomaly model 308 is ready to receive an input. An input image is provided to the anomaly model 308. The input image may be provided to the generative model of the anomaly model 308. The generative model of the anomaly model may generate a golden image output corresponding to the input image. Here, the golden image output is a "good" or "clean" variation of the input image.

[0119] The input image and the golden image output can then be provided to the comparison module as input. Next, the comparison module can output image comparison data that describes the difference between the input image and the golden image output. For example, if the input image contains a defect that does not exist in the golden image output, the image comparison data may indicate the presence of this defect. Note that the general purpose of the golden image is to provide a "good" or "clean" version of the input image. Thereby, any defects or anomalies (or other visual artifacts) present in the input image do not exist in the golden image (i.e., are removed) and can be identified through comparison of the input image and the golden image.

[0120] In one embodiment, direct image comparison is performed by the comparison module using matrix subtraction or pixel-to-pixel grayscale subtraction. The input and output are of the same dimension (the same as the actual input image (e.g., 512*512 from an autoencoder scaled back to 300*1000)), and only one value per pixel is subtracted.

[0121] The image comparison output data includes the differences identified by the image comparison process. In one embodiment, the image comparison output data is a grayscale image (e.g., 300*1000) that includes the difference between the inspection image and the generated golden image. As described above, the differences generally correspond to differences that exist in the inspection image and do not exist in the golden image.

[0122] The input image, the golden image, and the image comparison data may be provided to the anomaly classifier module, and the anomaly classifier module may output anomaly classification data, which may also be output from the anomaly model 308.

[0123] The anomaly classification data may include class labels for each located anomaly. The anomaly classification data may also include position data that designates the position of each anomaly. The position data may include the pixel coordinates of the bounding box of each anomaly.

[0124] The anomaly classifier module may include a machine learning model such as a neural network (e.g., a CNN). In one embodiment, the classifier module is a hybrid CNN binary classifier. In another embodiment, the classifier module may be a fine-tuned CNN.

[0125] In some examples, the classifier module may include a hybrid CNN binary classifier. The hybrid CNN binary classifier may be a combination of a convolutional layer (from a pre-trained CNN) and a support vector machine ("SVM") classifier. The SVM classifier may be trained on a small set of project-specific images. If there are sufficient training samples, the hybrid CNN may be replaced with a fine-tuned CNN.

[0126] In a particular example, the anomaly model 308 may be applied to an optical object inspection system. A training set of object images of objects known to be free of defects may be provided to the anomaly model 308 for training.

[0127] Once trained, inspection images of objects in an unknown defect state can be provided as input to the anomaly model 308. The inspection images may then be passed as input to the trained generation model, and the trained generation model may output a golden variation of the inspection images. Here, the golden variation of the inspection images does not contain defects. The golden variation of the inspection images and the inspection images may be passed as input to the comparison module. Next, the comparison module can output image comparison data that detects the difference between the golden variation of the inspection images and the inspection images. The comparison module may provide the image comparison data to the anomaly classification module, and the anomaly classification module may then classify the detected difference between the golden variation of the inspection images and the inspection images. For example, the difference may be classified as a certain known type of object defect. The anomaly classifier module can then output anomaly classification data from the anomaly model 308.

[0128] The image search engine 310 includes a software module configured to receive an image as input, seek images similar to the input image, and search through a plurality of accessible images.

[0129] For example, the image search engine 310 may be coupled to the data store 302 and may access a plurality of indexed images stored in the data store 302. The indexed images may be stored in a database.

[0130] The image search engine 310 can analyze the provided input image and the indexed images and return a reference to an image considered similar to the input image.

[0131] In one embodiment, the image search engine 310 can generate a feature embedded vector corresponding to the provided image and use the feature embedded vector to find similar images in the database.

[0132] The image search engine 310 can use any image search algorithm known in the art to perform image searches.

[0133] The automated training platform 312 includes software modules configured to receive training set image data and train a machine learning model for deployment. The machine learning model trained by the automated training platform 312 may include an image segmentation model. The image segmentation model may be an instance segmentation model. The machine learning model trained by the automated training platform may be an object detection model. The machine learning model trained by the automated training platform may be a machine learning model configured to perform at least one computer vision task. The at least one computer vision task may include, for example, any one or more of object detection, object tracking, image classification, semantic segmentation, and instance segmentation.

[0134] The explainable AI module 314 includes software modules configured to communicate with the automated training platform 312.

[0135] For each image input into the model trained by the automated training platform, the explainable AI module 314 can generate an output that includes data providing insights into the interaction between each input image and the output of the model trained by the automated training platform.

[0136] In some examples, this output may include class activation maps corresponding to each input image provided to the trained machine learning model.

[0137] The explainable AI module 314 can be configured to provide the output to another software module for further processing.

[0138] The output of the explainable AI module 314 can include any one or more of a prediction, a confidence level, a heat map, and other information describing how the input affects the internal layers and activation information of the layers of the model 308. This output data can enable the optimization of the selected input data and guide hyperparameter tuning to further optimize the performance of the model 308.

[0139] The collaborative learning module 316 includes a software module configured to receive data from the active learning platform 322.

[0140] The collaborative learning module 316 can receive at least one partially trained machine learning model from the active learning platform 322.

[0141] The collaborative learning module 316 can provide additional training data to the partially trained model to further train the partially trained model and generate a collaboratively trained machine learning model.

[0142] In some examples, the partially trained model received by the collaborative learning module 316 can include a trained anomaly model 308 configured for image anomaly detection and / or a model trained by the automated training platform 312 configured for image segmentation.

[0143] In some examples, the collaboratively trained machine learning model may be returned to the active learning platform 322 to incorporate the collaboratively trained machine learning model into the active learning platform 322.

[0144] In some examples, system 300 may include multiple federated learning modules 316 that may be distributed across multiple physical sites.

[0145] The deployed model 318 includes a trained machine learning model, which can be executed on a general-purpose or dedicated electronic computer.

[0146] The deployed model 318 receives an image as input and can output data regarding the input image, such as objects located within the image (e.g., defects, anomalies, other artifacts), classifications of the objects located within the image, the locations of the objects within the image, and other relevant metadata, such as the date and time of the analysis or the hardware identifier of the hardware on which the deployed model 318 is deployed.

[0147] The deployed model 318 can be trained and / or generated by a federated learning module or an active learning platform 322.

[0148] In some examples, the deployed model 318 may include a trained anomaly model 308 configured for image anomaly detection and / or a model trained by an automated training platform 312 configured for image segmentation or image anomaly detection.

[0149] The edge cloud mobile device 320 includes a computing device or service on which the deployed model 318 is executed.

[0150]

[0151] ​In other examples, the edge cloud mobile device 320 may include a cloud computing device. The cloud computing device may include a cloud computing instance on a commercial cloud computing service such as Amazon Web Services, Microsoft Azure, Google Cloud, or other commercial cloud computing services.

[0152] In other examples, the cloud computing device can include a private cloud computing device owned and operated by the same entity that deploys the deployed model 318.

[0153] In other examples, the edge cloud mobile device 320 may include a mobile device such as a smartphone, tablet, laptop computer, or other relatively portable computing device.

[0154] Referring now to FIG. 4, a flowchart is shown that illustrates a method 400 of operating a computing system, such as the system 300 of FIG. 3, according to an embodiment. The method 400 describes the operation of an active learning platform system configured to process an image dataset, but in other examples, the method can be modified to process other forms of data such as video data, audio data, or text data.

[0155] The method 400 may start at 402, and the method is started. In one example, the system 300 can start operating when an operator transfers a set of images to the data store 302. That becomes the trigger 402 that starts the method 400.

[0156] At 404, a training dataset / sample is retrieved from an image database such as the data store 302.

[0157] Once a dataset / sample is retrieved from the image database, the method proceeds to 406 where a dataset analysis tool is executed.

[0158] The dataset analysis tool may be the dataset analysis tool 304 of the system 300. The dataset analysis tool may calculate the sample size and sparsity of the data points within the dataset / sample.

[0159] Based on these sample size and sparsity calculations, at query 407, it can be determined whether the dataset / sample has a clustering or distribution that is acceptable for further use within an active learning system for training a machine learning model. Such determination may be based on application requirements and may also depend on the ability of the methods and systems described herein to generalize across a variety of datasets and all unique samples provided to the methods and systems described herein. The diversity of the dataset can be calculated based on a combination of application requirements, the nature of how certain objects can vary, and data - clustering diversity.

[0160] In an example of method 400 where it is determined that the dataset / sample does not provide an acceptable clustering or distribution, the data within the dataset / sample, which is considered insufficient, can be provided to a synthetic data generation component such as the synthetic data generation module 306 of the system 300.

[0161] As described above with reference to system 300, the synthetic data generation module 306 may receive sample data, and the sample data may include a subset of the dataset retrieved at 404. This sample data can be used to train the synthetic data generation component.

[0162] At 410, after the synthetic data generation component is trained, the synthetic data generation component can output synthetic data that is similar in characteristics to the sample data provided to the synthetic generation component for training.

[0163] At 412, the synthetic data generated by the synthetic generation component at 410 can be stored, for example, as a training data set / sample in the data storage 302. The synthetic data may be integrated into the data set initially obtained at 404, such that the data set includes a combination of the original data set (or a portion thereof) and the synthetic data generated at 410.

[0164] Method 400 can now proceed to 414. In an example where it is determined at 407 that the data set is properly distributed and clustered, method 400 can proceed directly from 407 to 414.

[0165] At 414, a machine learning model is trained using the current data set.

[0166] In an example of method 400, the machine learning model can include the model of the automated training platform 312 of system 300. In other examples, the machine learning model can include another model.

[0167] After the completion of 414, the model is trained. Here, the model can now be evaluated for performance.

[0168] At 416, an explainable AI tool can be executed to evaluate the performance of the model trained at 414. For example, the explainable AI tool can include the explainable AI module 314 of the system 300. At 416, the explainable AI can generate an output characterizing the performance of the model trained at 414. The explainable AI can further provide information regarding in what respects the model fails to execute its designed purpose.

[0169] At 418, the performance of the model trained at 418 can be evaluated according to the output characterizing the performance of the model generated at 416. Based on this output, it can be determined whether the model has acceptable performance or unacceptable performance.

[0170] In some examples, the model performance can be evaluated by measuring the model output accuracy for a known sample set. This evaluation can be performed by measuring the model performance against known samples and comparing the performance for certain attributes to a predetermined threshold. The attributes include, but are not limited to, precision, recall, and IoU (intersection over union).

[0171] For example, the model trained at 414 can include an image classifier. There may be a validation data set. The validation data set can include a set of images of known classes. These images may be provided to the model trained at 414 for classification of the images. If the percentage of images exceeding a pre-set threshold is misclassified, at 418, the trained model can be considered to provide insufficient performance. In such an example, the method 400 can proceed to 420.

[0172] In some instances, human factors may be further considered when evaluating model performance. Human factors may be used to provide an understanding of how a model evolves when new information or data is added to the model, and to enable an operator to ensure that the model maintains the highest possible generalization ability across all provided samples.

[0173] At 420, misclassified images, or images otherwise deemed to cause poor model performance, may be passed to an explainable AI tool. The explainable AI tool may, in some instances, be able to detect one or more regions of the input image for which the model trained at 414 provided inadequate performance. These regions may be cropped or otherwise indicated at 420 for providing to an image search engine tool (e.g., image search engine 320). Alternatively, the uncropped image may also be passed to the image search engine tool. This operation may be performed for all inappropriately classified images, or other images that are otherwise difficult to process.

[0174] At 422, the image search engine tool can search an image database (e.g., data store 302) queried at 404 for images similar to the image regions or crops provided to the image search engine tool as input.

[0175] At 424, it is determined whether such an image can be located by the image search engine tool. If such an image is located, the method may proceed to 426. If such an image is not located, the method can return to 408.

[0176] At 426, the images located by the image search engine at 422 can be stored as training samples in the database. After storing these images, the method may return to 414, where the model may be retrained with the training data set including the newly located images. After training at 414, the method can proceed to 416 again and then 418 as described above.

[0177] At 424, if no samples are returned by the image search engine, the method can return to 408, and samples of interest (e.g., misclassified images) may be provided as input to the synthetic image generation module, and the method can proceed to 410 again to generate synthetic image data characteristics of the problematic image or image part provided as input.

[0178] Once such images are generated, they may be stored in the training data set at 412, and method 400 may proceed to 414 to retrain the model using the updated data set and then proceed to 416 as described above.

[0179] At 418, if it is determined that the model performance is sufficient, at 428 and 430, the model can be deployed for collaborative training.

[0180] At 428, a collaborative learning data set is obtained. For example, the model may be deployed to a remote site trained after 414. At the remote site, the model may be further trained using an off-site data set, whereby the model is refined using this additional off-site input. Once trained, the collaborative training data set can be output by this further trained model and returned to the automated training platform system.

[0181] The federated training dataset includes data that includes the effect of additional training of the model using an off-site dataset without including the additional dataset itself.

[0182] At 430, the model is further trained using the federated learning dataset obtained at 428. Changes to the model resulting from further training using the off-site dataset may be incorporated into the model for further deployment.

[0183] In some examples of method 400, multiple iterative steps of federated learning may be applied to the model.

[0184] The federated training architecture applied at 428 and 430 allows the model to be trained using two separate datasets provided by two different parties without sacrificing the training data security of each party. For example, the operator of method 400 can access the original dataset obtained at 404 but cannot access the content of the off-site dataset. Similarly, the provider of the off-site dataset can access the content of the off-site dataset but cannot access the content of the dataset obtained at 404. The model in question can be collaboratively trained by two (or more) separate parties without any party having to share its training dataset with any other party. This can be advantageous since the training dataset may contain commercially valuable data.

[0185] At 432, the trained federated model is deployed. The trained federated model can be deployed for execution on a cloud, mobile, edge computing, or other computing device. Once deployed, the model can receive inputs and generate outputs according to the structure of the model and the provided training dataset. The deployed model can correspond to the deployed model 318 of the system 300.

[0186] Advantageously, method 400 can be executed without manual human intervention. In some examples, if method 400 does not meet pre-set performance metrics and the explainable AI output is different from what is expected from a model that functions well, a skilled human operator may intervene. In some examples, a skilled human operator may inspect encrypted data from a component (e.g., the federated learning module 316) to analyze the fault.

[0187] Referring now to FIG. 5, a method 500, which is a variation of method 400, is shown. The above description with reference to method 400 may be applicable to method 500. The steps of method 500 may be similar to those of method 400, with each step number incremented by 100.

[0188] Method 500 differs from method 400 in that method 500 includes training, execution, and evaluation of an anomaly model at 514, 516, and 518, respectively. The anomaly model may be the anomaly model 308 as described above with reference to system 300.

[0189] In 516, instead of running an explainable AI tool on the trained model, a validation data set can be provided to the model trained in 514, and the performance of the model can be evaluated in 518. Data points where the model provides poor performance are identified and can be provided to an image search engine in 520 for searching for similar images from the data set. If such similar images are located [discovered], they may be incorporated into the training data set and the model may be retrained in 514. If such similar images are not located, method 500 can call a synthetic image generation component in 510 and provide an input to the synthetic image generation component that describes the missing image data points, such as exemplary images. The synthetic image generation component can output synthetic image data for storage in an image database and incorporation into the training data set. The model can then be retrained in 514 and method 500 can proceed as described above with reference to method 400.

[0190] In some examples of method 500, the abnormal models of method 500 may be further analyzed using explainable artificial intelligence techniques. Such methods may be at least similar to those described with reference to step 416 of method 400 above. Additionally, the abnormal models of method 500 can be further evaluated for performance by testing for anomalies against a validation training set using known parameters.

[0191] The foregoing description provides examples of one or more apparatuses, methods, or systems, but it will be understood that other apparatuses, methods, or systems may be within the scope of the claims as interpreted by those skilled in the art.

Claims

1. A computer-implemented method for training a machine learning model and performing active learning for deployment, comprising: storing, in a data storage device, an image database including a first training dataset of training samples; determining a clustering or distribution of the first training dataset using a dataset analysis tool executed by at least one processor communicating with the data storage device; in response to the determination of the clustering or distribution, generating a first set of one or more synthetic images using, as input, a subset of the training samples from the first training dataset by a synthetic image generation module executed by the at least one processor, and generating a second training dataset including the first set of one or more synthetic images; training the machine learning model using the second training dataset to obtain the trained machine learning model; evaluating the performance of the trained machine learning model using a model performance evaluation tool executed by the at least one processor, the evaluation including identifying one or more images in the second training dataset that have an adverse effect on model performance; sending a request for one or more training samples to an image search engine executed by the at least one processor, the requested training samples being defined using image data from the identified one or more images that have an adverse effect on model performance; executing the image search engine to search the image database for the requested one or more training samples; if the image search engine returns the requested one or more training samples: generating a third training dataset including the requested one or more training samples, training the machine learning model using the third training dataset to obtain the trained machine learning model; If the image search engine does not return one or more requested training samples: The synthetic image generation module uses, as input, image data from the one or more images in the first training data set that contributed to unacceptable model performance, to generate a second set of one or more synthetic images, and generates a fourth training data set that includes the second set of one or more synthetic images, including the step of training the machine learning model using the fourth training data set to obtain the trained machine learning model. Method.

2. The method according to claim 1, wherein determining the clustering or distribution of the first training data set includes calculating the sparsity and sample size of data points within the first training data set.

3. The method according to claim 1, wherein determining the clustering or distribution of the first training data set is a step of performing an analysis operation for placing the training samples into clusters, wherein the training samples within each cluster include similarity, and a step of quantifying the variation of the training samples within each cluster.

4. The method according to claim 1, wherein determining the clustering or distribution of the first training data set includes the data set analysis tool determining that the ability to generalize across the training samples is insufficient.

5. The method according to claim 1, wherein the synthetic image generation module includes a neural network and is configured to receive one or more training samples of an image class and generate and output one or more synthetic images of the same image class.

6. The method according to claim 1, wherein evaluating the performance of the trained machine learning model includes determining that the classification accuracy of the trained machine learning model does not meet a predetermined accuracy threshold.

7. The method according to claim 1, wherein the image database includes a plurality of indexed images, and the image search engine is configured to analyze the input image and the plurality of indexed images and return a reference to an image in the plurality of indexed images that is similar to the input image.

8. The method according to claim 7, wherein the image search engine is configured to generate a feature embedding vector corresponding to the input image and use the feature embedding vector to search the plurality of indexed images.

9. The method according to claim 7, wherein the model performance evaluation tool is configured to detect one or more regions in the one or more images in the second training dataset that have an adverse effect on model performance, and the input image to the image search engine is a cropped image including the detected one or more regions.

10. The method according to claim 1, wherein the machine learning model is configured to perform at least one computer vision task including any one or more of object detection, object tracking, image classification, semantic segmentation, and instance segmentation.

11. Evaluating the performance of the trained machine learning model includes generating, by the model performance evaluation tool, an output including any one or more of a prediction, a confidence level, a heatmap, and other information describing how the input affects the activation information and internal layers of the various layers of the trained machine learning model, according to the method of claim 1.

12. Evaluating the performance of the trained machine learning model includes measuring, by the model performance evaluation tool, the model output accuracy of the trained machine learning model for a known sample set, and the measuring includes measuring the performance of the trained machine learning model for the known sample and comparing the performance with any one or more predetermined thresholds of precision, recall, and IoU, according to the method of claim 1.

13. Using at least one federated learning module executed by the at least one processor, executing a federated learning process using the trained model and additional training data to further train the trained machine learning model to obtain a federated trained machine learning model, the method further comprising evaluating the performance of the federated trained machine learning model using the model performance evaluation tool, the method according to claim 1.

14. The method according to claim 13, wherein the additional training data is from at least two physical sites performing computer vision-based visual inspection.

15. A computer system for performing active learning to train and deploy a machine learning model, comprising: A data storage device for storing an image database containing a first training dataset of training samples; At least one processor communicating with the data storage device, The at least one processor: Determining the clustering or distribution of the first training dataset using a dataset analysis tool; In response to the determination of the clustering or distribution, using a subset of the training samples from the first training dataset as input by a synthetic image generation module to generate a first set of one or more synthetic images and generating a second training dataset containing the first set of one or more synthetic images; Training the machine learning model using the second training dataset using an automated training module to obtain the trained machine learning model; Evaluating the performance of the trained machine learning model using a model performance evaluation tool, the evaluation including identifying one or more images in the second training dataset that adversely affect model performance. Sending a request for one or more training samples to an image search engine, wherein the requested training samples are defined using image data from one or more identified images that negatively impact model performance; Executing the image search engine to search for the requested one or more training samples in the image database; If the image search engine returns the requested one or more training samples: Generating a third training data set including the requested one or more training samples, Training the machine learning model using the third training data set to obtain the trained machine learning model; If the image search engine does not return the requested one or more training samples: Using, as input, image data from the one or more images in the first training data set that contributed to unacceptable model performance, by the synthetic image generation module, to generate a second set of one or more synthetic images, and generating a fourth training data set including the second set of one or more synthetic images, Training the machine learning model using the fourth training data set to obtain the trained machine learning model configured to perform system.

16. Determining the clustering or distribution of the first training data set is a step of performing an analysis operation for placing the training samples into clusters, wherein the training samples within each cluster include similarity, and quantifying the variation of the training samples within each cluster, the system according to claim 15.

17. The synthetic image generation module has a neural network and is configured to receive one or more training samples of an image class and generate and output one or more synthetic images of the same image class, the system according to claim 15.

18. The system according to claim 15, wherein evaluating the performance of the trained machine learning model includes determining that the classification accuracy of the trained machine learning model does not meet a predetermined accuracy threshold.

19. The system according to claim 15, wherein the image database includes a plurality of indexed images, and the image search engine is configured to analyze the input image and the plurality of indexed images and return a reference to an image in the plurality of indexed images that is similar to the input image.

20. A non-transitory computer-readable medium storing computer-executable instructions that, when executed by at least one processor, cause the at least one processor to execute a method, the method comprising: storing, in a data storage device, an image database including a first training data set of training samples; determining a clustering or distribution of the first training data set using a data set analysis tool executed by at least one processor communicating with the data storage device; in response to the determination of the clustering or distribution, generating, by a synthetic image generation module executed by the at least one processor, a first set of one or more synthetic images using, as input, a subset of the training samples from the first training data set, and generating a second training data set including the first set of one or more synthetic images; training the machine learning model using the second training data set to obtain the trained machine learning model; evaluating the performance of the trained machine learning model using a model performance evaluation tool executed by the at least one processor, the evaluation including identifying one or more images in the second training data set that have an adverse effect on the model performance. Sending a request for one or more training samples to an image search engine executed by the at least one processor, wherein the requested training samples are defined using image data from one or more identified images that have an adverse effect on model performance; Executing the image search engine to search for the requested one or more training samples in the image database; If the image search engine returns the requested one or more training samples: Generating a third training data set including the requested one or more training samples; Training the machine learning model using the third training data set to obtain the trained machine learning model; If the image search engine does not return the requested one or more training samples: Using, as input, image data from the one or more images in the first training data set that contributed to unacceptable model performance, by the synthetic image generation module, to generate a second set of one or more synthetic images and generating a fourth training data set including the second set of one or more synthetic images; Training the machine learning model using the fourth training data set to obtain the trained machine learning model; A non-transitory computer-readable medium.

Images