Model enhancement framework for domain-assisted continuous learning in deep learning

Abstract

Techniques for generating a block expander model are described herein. An example method includes a system accessing a base model trained to identify a base category. The system may access an extender including a block extender, the extender category being different from the base category. The system may connect an expander with the base model to generate an enhanced model. The system may input training data to the enhancement model, the training data being provided to the base model and the expander, the training data including the custom category. The system may train an expander model to identify custom categories based at least in part on training data and signals received from a base machine learning model. The system may generate a trained expander based at least in part on the training, the expander being trained to identify an object associated with the expander category.

Description

[0001] Cross-references to related applications

[0002] This application claims priority and benefit to U.S. Provisional Application No. 63 / 529,297, filed July 27, 2023, and U.S. Non-Provisional Application No. 18 / 406,905, filed January 8, 2024, each of which is incorporated herein by reference in its entirety for all purposes. Technical Field

[0003] This application relates to the field of neural networks, and more particularly to a model enhancement framework for domain-assisted continuous learning in deep learning within said neural networks. Background Technology

[0004] Cloud service providers (CSPs) can offer a variety of cloud services to subscribers. These services are offered under different models, including Software as a Service (SaaS), Platform as a Service (PaaS), Infrastructure as a Service (IaaS), and others.

[0005] CSPs can offer example services based on machine learning (ML). CSPs can use various techniques to train machine learning models to support client requests. Transfer learning on large-scale pre-trained models is an example of a technique used to train machine learning models to learn new tasks in an ML context. One problem is that the resulting model often loses its ability to perform at the same level as on existing classes (pre-trained classes), a phenomenon often referred to as catastrophic forgetting. Enabling models to support both pre-trained classes and newly encountered classes without degrading their performance on previously encountered tasks is a research area known as continuous learning (CL). Summary of the Invention

[0006] The embodiments described in this paper describe a technique for using a unified framework to connect a base model to an extender model to generate augmented models capable of performing customized tasks. This unified framework can include inter-type and intra-type extenders, which are capable of extracting / isolieving important information from the base model for a given dataset and domain. These extenders can be connected to the base model to facilitate predictions in the relevant domain without requiring prior task data on which the base model was trained. For inter-type extenders, the extender allows the base model to retain its previous learning and therefore does not cause any performance degradation in the pre-trained classes, a standard bottleneck for CL methods. Augmented models containing inter-type extenders can include a base model capable of continuously performing the original task on which it was trained. Augmented models with intra-type extenders can also include intra-type extenders that can be trained to perform tasks different from those performed by the base model. Augmented models with intra-type extenders can be trained to use previous learning and perform new tasks.

[0007] This unified framework may also include a fusion layer, which learns to effectively combine knowledge from previous learning tasks to facilitate faster knowledge transfer or adaptation to new datasets.

[0008] Inter-class and inner-class expanders can encode information from previously trained datasets. Therefore, it is not necessary to explicitly store the previous task dataset for CL training jobs. Learnable blocks and fusion layers enable the addition of learnable blocks to any ML model architecture (unimodal or multimodal) and allow for fine-tuning for optimization and performance.

[0009] This unified framework supports indexing of inter-class and inner-class extenders in the data repository based on domain metadata. Therefore, inter-class and inner-class extenders from relevant domains can be retrieved from the data repository on demand and fused during training to facilitate faster, plug-and-play knowledge transfer for associated client tasks. This unified framework enables faster learning of new tasks and domain data while preserving the performance of previously trained base or custom classes.

[0010] Furthermore, the data repository can store mappings from domains to dataset-specific expanders from past iterations, providing finer-grained control over expanders to be used for new client tasks. Due to the portable nature of learnable blocks, the unified framework CL can utilize a single training phase with on-demand selection and fusion of machine learning blocks.

[0011] This document describes various embodiments, including computer-implemented methods, systems, stored programs, code, or non-transitory computer-readable media of instructions executable by one or more processors. Some embodiments may be implemented using a computer program product comprising a program / instructions that, when executed by a processor, cause the processor to perform any of the methods described in this disclosure.

[0012] The embodiments described herein relate to techniques for training inter-class augmentation models, wherein the techniques may include, for example, methods, computing systems, or one or more computer-readable media. Example methods may include a computing system that identifies custom categories.

[0013] The method may also include a computational system selecting a pre-trained base model trained to predict base classes. This selection may be based at least in part on base classes belonging to the same domain as the custom class.

[0014] The method also includes a computational system generating an untrained inter-class expander model, which is trained to predict custom classes from data points associated with custom classes. This untrained inter-class expander model may include an untrained fusion layer and an untrained inter-class expander.

[0015] The method may also include a computational system that connects a pre-trained base model in parallel with an untrained inter-class expander model to generate an inter-class augmentation model.

[0016] The method may also include a computational system inputting training images into an inter-class augmentation model. These training images may include data points associated with custom categories.

[0017] The method may also include a computational system based at least in part on training images, such that a first block of output from a pre-trained base model is associated with a first plurality of activations of a base class.

[0018] The method may also include a computational system that enables an untrained fusion layer to generate multiple weights for weighting a first plurality of activation values. These multiple weights can be configured to enable an untrained inter-class expander model to recognize custom categories from training images.

[0019] The method may also include a computational system that enables an untrained fusion layer to generate a weighted sum based at least in part on weighting a first plurality of activation values ​​using multiple weights.

[0020] The method may also include a computational system that enables a pre-trained base model to predict a base class from training images, at least in part, based on a first plurality of activation values.

[0021] The embodiments may also include techniques for training the in-class augmentation model, wherein the techniques may include, for example, methods, computing systems, or one or more computer-readable media. An example computing system may include one or more processors and one or more computer-readable media, the one or more computer-readable media including instructions that, when executed by the one or more processors, cause the one or more processors to recognize the identity of a custom class.

[0022] When executed by one or more processors, this instruction enables one or more processors to select a pre-trained base model trained to predict a base class, the selection being based at least in part on a base class belonging to the same domain as the custom class.

[0023] When executed by one or more processors, this instruction can cause one or more processors to generate an untrained inner class expander model trained to identify custom classes from data points associated with custom classes. The untrained inner class expander model includes an untrained fusion layer, a first untrained inner class expander layer, and a second untrained expander layer.

[0024] When executed by one or more processors, this instruction allows one or more processors to input training data to connect an untrained inner class expander model to a pre-trained base model, thereby generating an inner class augmentation model.

[0025] When executed by one or more processors, this instruction enables one or more processors to input training images into an in-class augmentation model, the training images including data points associated with a custom category.

[0026] When executed by one or more processors, this instruction can cause one or more processors to output a first plurality of activation values, at least in part, based on the training images, the first block of the pre-trained base model.

[0027] When executed by one or more processors, this instruction can cause one or more processors to output a second plurality of activation values, at least in part, based on the training images, from a first untrained inner class expander layer.

[0028] When executed by one or more processors, this instruction can cause one or more processors to generate multiple weights for weighting a first plurality of activation values ​​and a second plurality of activation values ​​from an untrained fusion layer. These multiple weights are configured to enable the in-class enhancement model to recognize a custom category from the training images.

[0029] When executed by one or more processors, this instruction can cause one or more processors to make a second block of the pre-trained base model at least in part based on a weighted sum to output a third plurality of activation values.

[0030] When executed by one or more processors, this instruction can cause one or more processors to output a fourth plurality of activation values ​​based at least in part on a weighted sum of the second untrained expander layer.

[0031] When executed by one or more processors, this instruction enables one or more processors to make the in-class enhancement model predict custom categories from training images, at least in part, based on a third and a fourth set of activation values.

[0032] The embodiments described herein relate to techniques for maintaining a data repository, wherein the techniques may include, for example, methods, computing systems, or one or more computer-readable media. Example methods may include a data repository receiving a message requesting that a model be trained to predict custom categories.

[0033] The method may also include a data repository that selects pre-trained base models from multiple pre-trained base models to predict base categories belonging to the same domain as the custom category.

[0034] The method may also include a data repository selecting a first pre-trained expander model from multiple pre-trained expander models. The first pre-trained expander model may be associated with a pre-trained base model, at least in part, based on the selection of the pre-trained base model. The first pre-trained expander model can be trained to predict custom categories.

[0035] The method may further include having a data repository trained, at least in part, on a first pre-trained expander model to predict custom categories, and connecting a pre-trained base model to the first pre-trained expander model to generate an augmented model. The pre-trained base model may be connected to the first pre-trained expander model such that a first output of a first block of the pre-trained base model is received by a first pre-trained fusion layer of the pre-trained expander model.

[0036] The method may also include a data repository transmitting the enhanced model to the message sender. Attached Figure Description

[0037] Figure 1A This is an example processing flow for generating an extender model according to one or more embodiments.

[0038] Figure 1B This is an example processing flow for generating inter-class extender models according to one or more embodiments.

[0039] Figure 1C This is an example processing flow for generating an inner class extender model according to one or more embodiments.

[0040] Figure 2 This is an illustration of an example system for generating an enhanced model according to one or more embodiments.

[0041] Figure 3 This is an illustration of an example augmentation model training system for generating a target augmentation model, according to one or more embodiments.

[0042] Figure 4 This is an illustration of an example inter-class enhancement model objective based on one or more embodiments.

[0043] Figure 5 This is an illustration of an example fusion layer according to one or more embodiments.

[0044] Figure 6 It is an example enhancement model having one or more inner class extender models according to one or more embodiments.

[0045] Figure 7 This is an illustration of an example in-class extender unit according to one or more embodiments.

[0046] Figure 8 This is an illustration of an example in-class extender unit according to one or more embodiments.

[0047] Figure 9 This is an illustration of an example in-class extender unit according to one or more embodiments.

[0048] Figure 10 This is an example processing flow according to one or more embodiments for using an enhanced model having one or more inner class extender models.

[0049] Figure 11 This is an example processing flow for training an inter-class expander model according to one or more embodiments.

[0050] Figure 12 This is an example processing flow for training an inner class extender model according to one or more embodiments.

[0051] Figure 13 This is an example processing flow for maintaining a data repository, according to one or more embodiments.

[0052] Figure 14 This is a block diagram illustrating a pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0053] Figure 15 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0054] Figure 16 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0055] Figure 17 This is a block diagram illustrating another pattern for implementing a cloud infrastructure-as-a-service system according to at least one embodiment.

[0056] Figure 18 This is a block diagram illustrating an example computer system according to at least one embodiment. Detailed Implementation

[0057] In the following description, various embodiments will be described. Specific configurations and details are set forth for illustrative purposes in order to provide a thorough understanding of the embodiments. However, it will also be apparent to those skilled in the art that these embodiments can be practiced without specific details. Furthermore, well-known features may be omitted or simplified so as not to obscure the described embodiments.

[0058] For cloud service providers (CSPs), category expansion becomes necessary when customers want to leverage the capabilities of pre-trained models while simultaneously extending them to a few novel categories. Once a model has been fine-tuned on a customer's task, its weights may have undergone significant changes, limiting its reusability on subsequent related downstream tasks. This imposes a constraint on a specific set of model parameters for each dataset. This may be suboptimal in terms of cost, time, and effort for potential customer use cases centered around seasonally varied datasets, where domain data exhibits periodic subtle differences. Meeting customer requests for tasks may require fine-tuning large pre-trained models from scratch. Existing class expansion (CL) techniques designed to address some of these issues have several drawbacks that make them less suitable for commercial use cases. For example, existing CL methods typically require storing a subset of training data for previous tasks for memory replay (e.g., showing the model the previous dataset again). This poses a security risk to the storage and access of confidential customer data stored in their respective leases. Furthermore, existing CL methods require longer inference times associated with higher model complexity, or longer training durations (due to multiple optimization stages used to prune the resulting model to obtain a better-optimized model, but with a slight performance decrease, often referred to as "quantization"). Additionally, existing CL methods are associated with a lack of fine-grained control over the domain-specific learnable model parameters relevant to a given task.

[0059] The embodiments in this paper address the aforementioned problems by introducing a technique that uses a unified framework to connect a base model to an expander model to generate an augmented model. This unified framework enables efficient single-stage training of CL models. The unified framework includes between-class and within-class expanders, which can extract / isolate important information about a given dataset and domain. Expanders can be attached / inserted into the base model to facilitate predictions on relevant domains without prior task data. For the between-class expanders, the expanders allow the base model to retain its previous learning and therefore do not cause any performance degradation in pre-trained classes, which is a standard bottleneck of CL methods.

[0060] This unified framework may also include a fusion layer, which learns to effectively combine knowledge from previous learning tasks to facilitate faster knowledge transfer or adaptation to new datasets.

[0061] The expander can encode information from the previously trained dataset. Therefore, it is not necessary to explicitly store the previous task dataset for the CL training job. The learnable blocks and fusion layers enable the addition of learnable blocks to any ML model architecture (unimodal or multimodal) and allow for fine-tuning for optimization and performance.

[0062] This unified framework supports indexing extenders in a data repository based on domain metadata. Therefore, extenders from relevant domains can be retrieved from the data repository on demand and fused during training to facilitate faster knowledge transfer to associated client tasks in a plug-and-play manner. This unified framework enables faster learning of new tasks and domain data while preserving the performance of previously trained base or custom classes.

[0063] Furthermore, the data repository can store mappings from domains to dataset-specific expanders from past iterations, providing finer-grained control over expanders to be used for new client tasks. Due to the portable nature of learnable blocks, the unified framework CL can utilize a single training phase with on-demand selection and fusion of machine learning blocks.

[0064] Figure 1 is an example processing flow for generating an expander model according to one or more embodiments. At 102, a computational system (such as an augmentation model training system) can receive a request for a model trained to perform classification from a set of one or more categories. The computational system described herein can be implemented through various computational systems (such as...) Figure 14-18 This can be achieved through a system described in [the document]. For example, a CSP's client might request a model that can classify custom categories, such as cat breeds and dog breeds.

[0065] At point 104, the computational system may search one or more model data repositories to determine if a model trained to perform classification on the exact category identified at point 102 exists. This one or more model data repositories may include various models, such as previously trained base models and previously trained extender models. In some cases, the model may include an augmentation model that includes a base model connected to the trained extender model.

[0066] At point 106, the computational system can determine whether a model trained to perform classification on the exact category identified in point 102 has been found. If a model is found in point 104, then at point 108, the model is used as a response to the request received in point 102, and the processing ends.

[0067] However, if no model is found at 104, the computational system can search at 110 for one or more models that might fit the request received in 102, based on the categories and domains identified in 102. The goal of the computational system could be to identify models trained to classify one or more categories closest to the one or more categories requested in 102. Consider an example where the computational system uses the domain "animals" to narrow its search. The search results may produce multiple different models. The computational system can then further narrow its search by identifying which of these multiple different models (if any) is closest to being trained to classify one or more categories identified in 102 (e.g., breeds of cats and breeds of dogs). If the search produces a first base model classifying cats, a second model classifying dogs, a third model classifying cats and dogs, and a fourth model classifying cats and horses, then the closest model is the third model classifying cats and dogs, and therefore the computational system can select the third model at 110.

[0068] At point 112, the computational system can determine one or more categories for which the model selected in 110 is trained to perform classification. At point 114, the computational system can, based on the set of categories identified in 112, determine whether to extend the base model by internally modifying the base model (e.g., intra-class modification) or by externally connecting the base model (e.g., inter-class modification). For example, if one or more categories from the request in 102 include cats, dogs, and horses, and the model selected in 110 is trained to classify cats and dogs, then inter-class modification can be selected to add an extender model for classifying horses. However, if one or more categories from the request in 102 are cat breeds, and the model selected in 110 is trained to classify cats, then intra-class modification can be selected to modify the model to classify cat breeds. At point 116, depending on whether inter-class modification or intra-class modification was selected, the processing flow can proceed to point 120 or 150.

[0069] Figure 1B This is an example processing flow for generating inter-class expander models according to one or more embodiments. At 120, based on the categories identified in 102, one or more base categories identified in 112, and metadata associated with the base model identified in 110, the computing system can identify any previously trained inter-class expander models to be included in the target augmentation model. The metadata may include information associated with the base model and identify each augmentation model to which the base model is a part. For example, the base model may be linked to a first inter-class expander model, and a copy of the base model may be linked to a second inter-class expander model. In any case, the metadata may provide information about each inter-class expander model. For example, if one or more requested categories identified in 102 include cats, dogs, and horses, and the base model is trained to classify cats, then the computing system can use the metadata and the categories identified in 102 to determine whether there are one or more inter-class expander models that can be trained to classify dogs, horses, or both.

[0070] At point 122, the computational system can determine whether any previously trained inter-class expander models were identified at point 120. If previously trained inter-class expanders were identified, then at point 124, zero or more inter-class expander models to be included in the training phase of the augmented model are identified. The decision to include inter-class expander models can be based on various factors, such as the architectural compatibility of the inter-class expander models and the base model, and the one or more classes that the inter-class expander models were trained on. However, if no previously trained inter-class expander models were identified at point 122, then the computational system can create inter-class expander models at point 126. The description of the inter-class expander models is as follows: Figure 4 To provide a more detailed description.

[0071] At point 128, the computational system can create an augmented model by connecting the base model identified at point 110, any previously trained inter-class expander models, and any new untrained inter-class expander models created at point 126. The inter-class expander models can be connected in parallel with the base model. Specifically, the inter-class expander models can include a sequence of inter-class expander layers, where each inter-class expander layer includes a fusion layer and an inter-class expander. Each inter-class expander layer can be connected to a corresponding base block of the base model. The augmented model, including the inter-class expander models connected to the base model, is related to... Figure 6 To provide a more detailed description.

[0072] At point 130, the computational system can train the augmentation model created at point 128 using a custom-class training dataset to generate a trained augmentation model. Training may include training a new, untrained inter-class expander model. Training may include locked-configuration training or unlocked-configuration training. Locked-configuration training may include locking the parameters (e.g., weights) of the base model. In this configuration, a cost function is used to determine the accuracy of the inter-class expander model. Furthermore, backpropagation can be used to tune the parameters of the inter-class expander model. The parameters of the base model are locked and not tuned. Unlocked-configuration training may include using a cost function to evaluate the accuracy of any inter-class expander model and the accuracy of the base model. The parameters of both the inter-class expander model and the base model can be tuned using backpropagation. At point 132, the computational system can determine whether to use a locked or unlocked configuration. This decision may be based on, for example, user input.

[0073] If at point 132 the computation system determines to use a locked configuration, then the computation system can store the trained inter-class expander model in the model data repository at point 134. However, if the computation system determines to use an unlocked configuration, then the computation system can store the inter-class expander model at point 136, as well as any other models included in the augmented model created at point 128, because the parameters of the base model and any pre-trained inter-class expander models may have changed.

[0074] At point 138, the computing system can update the metadata information associated with the base model to indicate the creation of the inter-class augmentation model, zero or more inter-class expander models identified in point 120, and newly trained inter-class expander models.

[0075] Figure 1C This is an example processing flow for generating an inner class expander model according to one or more embodiments. Steps 150 to 164 relate to the inner class expander model. At 150, based on the category identified at 102, one or more categories identified at 112, and metadata associated with the base model identified at 110, the computing system can identify any previously trained inner class expander models to be included in the augmented model to be created.

[0076] At point 152, the computational system can create a new untrained inner class expander model. It should be recognized that, for the inter-class augmentation model, the previously trained inter-class expander model connected to the base model can be disconnected and replaced by the untrained inter-class expander model. For the inner class expander model, if the base model is already connected to the inner class expander model, and the inner class expander model has been trained together with the base model, then the new untrained inner class expander model will be connected to the inner class augmentation model, which includes both the base model and the previously trained inner class expander model.

[0077] At 154, the computational system can create an inner class augmentation model by connecting any previously trained inner class extender model identified at 150 and a new untrained inner class extender model created at 152 to the base model identified at 110.

[0078] At point 156, the computational system can train the inner-class augmentation model created at point 154 using a custom-class training dataset to generate a trained between-class augmentation model. Training may include training a new, untrained inner-class expander model.

[0079] If the computational system uses a locked configuration to train the inner class augmentation model at position 158, then the computational system can store the trained inner class expander model in the model data repository at position 160. However, if the computational system uses an unlocked configuration to train the inner class augmentation model, then the computational system can store the trained inner class augmentation model created at position 154 at position 162, because the parameters of the base model and any pre-trained inner class expander models may have changed.

[0080] At point 164, the computational system can update the metadata information associated with the base model to indicate the creation of the inner class enhancement model, at point 150 the zero or more inner class expander models identified, and the newly trained inner class expander model.

[0081] Figure 2 This is an illustration of a computational system 200 for generating augmented (e.g., between-class or within-class) models according to one or more embodiments. The computational system (CSP) can receive requests for machine learning models trained to perform classification of one or more custom categories. The computational system 200 can receive input 202 from the CSP, including control instructions for training a base model 204 for the requested one or more categories. The base model 204 can be implemented by a neural network (such as a convolutional neural network (CNN)) and includes single, binary, or multi-class classifiers. The base model 204 may have been previously trained using a base model training unit 206, which can be a computational system configured to train machine learning models. The base model training unit 206 can receive model parameters and an architecture 208. Model parameters can include various hyperparameters, such as the number of hidden layers, learning rate, and regularization strength. The architecture can include definitions of input layers, hidden layers, and output layers. The base model 204 can also be trained using a base dataset 210 that includes various training instances for training the base model 204. For example, in a supervised learning environment, the base dataset 210 may include labeled images of one or more base categories. For instance, if the base model 204 is to be trained to classify cats and dogs, then the base dataset 210 may include labeled images of cats and dogs.

[0082] At different times, a customer may request a model to classify cat breeds. As indicated above, the base model 204 has previously been trained to classify cats and dogs. The technique described herein allows a model to be trained to classify cat breeds using the base model's ability to classify cats and dogs.

[0083] like Figure 1B and 1C As indicated in 152, the computing system can access model data repository 212 to determine if any previously trained models exist that have been trained to classify custom categories (e.g., cat breeds). If the computing system cannot identify a trained model in the model data repository that has been trained to identify custom categories, then the computing system can create an inter-class expander or an intra-class expander to be trained to classify custom categories. If the control instructions instruct the model to be trained to classify only new categories, then the computing system 200 can create an intra-class expander model using intra-class expander unit 214, as indicated in 152. However, if the control instructions instruct the model to be trained to classify both new categories and any categories that the base model 204 was trained to target, then the computing system 200 can create an inter-class expander model using inter-class expander unit 216, as indicated in 126.

[0084] When the computing system 200 creates a new inner class expander model, the computing system can use plug-ins and fusion units 218 to create an augmented model 220 (e.g., an inner class) by connecting the base model 204 to the new untrained inner class expander model, as described in 154. When the computing system 200 creates a new inter-class expander model, the computing system can use plug-ins and fusion units 218 to create an augmented model 220 by connecting the base model 204 to the untrained inter-class expander model, as described in 128.

[0085] Figure 3 This is an illustration of an augmentation model training system 200 for generating a trained augmentation model according to one or more embodiments. The augmentation model training system 200 can receive an augmentation model for training. The augmentation model can be an inter-class augmentation model (see [link]). Figure 1B ) or inner class enhancement model (see Figure 1C The augmented model training system 300 can also receive a custom category training dataset 302. The custom category training dataset 302 can be a training set used to train the augmented model 220 to classify one or more categories identified in step 102. For example, the custom category training dataset 302 can be an image with data points for objects (such as animals) or images of other objects.

[0086] The custom category training dataset 302 can be based on the categories that the augmentation model 220 will be trained to classify, as indicated in 102. This can also depend on whether the augmentation model is an inter-class augmentation model or an within-class augmentation model. For example, for an inter-class augmentation model, there are at least two variations, where a first variation may include a base model connected to an untrained inter-class expander model, and a second variation may include a base model connected to one or more previously trained inter-class expander models as well as an untrained inter-class expander model. The one or more categories identified in 102 may include cats, dogs, and cat breeds. Using the first variation, in a first case, the base model may be trained to classify only cats. In this case, the untrained inter-class expander model may need to be trained to classify both dog and cat breeds. Therefore, the custom category training dataset 302 may include labeled images of dog and cat breeds. In a second case of the first variation, the base model may be trained to classify both cats and dogs. In this case, the untrained inter-class expander model may need to be trained to classify cat breeds. Therefore, the custom category training dataset 302 may include labeled images of cat breeds. For the second variant, a similar example can be envisioned. The augmentation model training system 300 can train the inter-class augmentation model using a custom class training dataset 302 as described in step 130 to generate a trained augmentation model 304 (an inter-class augmentation model in this example).

[0087] For the in-class augmentation model, at least two variations are possible. A first variation may include a base model connected to an untrained in-class expander model, and a second variation may include a base model connected to one or more previously trained in-class expander models as well as an untrained in-class expander model. Using the first variation, in a first case, the base model might be trained to classify cats and dogs, and one or more categories identified in 102 could be cat breeds and dog breeds. In this case, the custom category training dataset 302 could include labeled images of cat breeds and dog breeds. In a second case of the first variation, the base model might be trained to classify cats and dogs, and one or more categories identified in 102 could be cat breeds and dog breeds. In this case, the custom category training dataset 302 could include labeled images of dogs, cat breeds, and dog breeds. For the second variation, similar examples are conceivable. The augmentation model training system 300 can train the in-class augmentation model using the custom category training dataset 302 as described in step 156 to generate a trained augmentation model 304 (in this example, the in-class augmentation model).

[0088] As indicated in steps 132 and 158, training may include a locked configuration or an unlocked configuration. For a locked configuration and an inter-class augmentation model, the augmentation model training system 300 may store the trained new inter-class expander model in the model data repository 306, as described in step 134. For an unlocked configuration, the augmentation model training system 300 may store the trained new inter-class augmentation model in the model data repository 306, as described in step 136.

[0089] For the locked configuration and the inner class augmentation model, the augmentation model training system 300 can store the trained new inner class expander model in the model data repository 306, as described in step 160. For the unlocked configuration, the augmentation model training system 300 can store the trained new inner class augmentation model in the model data repository 306, as described in step 162.

[0090] Inter-class expander model

[0091] Figure 4 This is an illustration of an inter-class augmentation model 400 according to one or more embodiments. As shown, the inter-class augmentation model 400 may include a pre-trained base model 402, zero or more pre-trained inter-class extender models 404, and an untrained inter-class extender model 406. It should be appreciated, and as stated above, that in another embodiment, the inter-class augmentation model 400 may include a pre-trained base model 402 and an untrained inter-class extender model 406. In this case, the inter-class augmentation model 400 will not include zero or more inter-class extender models 404.

[0092] The pre-trained inter-class expander layer 410 of the pre-trained inter-class expander model 408 can be connected to the output of the corresponding first block 412 of the pre-trained base model 402. Specifically, the output of the first block 412 can be received by the pre-trained fusion layer 414 of the pre-trained inter-class expander layer 410. The first block 412 and each subsequent block of the pre-trained base model 402 can include a set of convolutional layers that collectively perform one or more tasks (e.g., object detection, feature detection, contour detection, color detection). The output of each convolutional layer can be determined by a set of weights for the block. The second block 417 can also include a set of layers that collectively perform one or more tasks. The first block 412 can be selected to correspond to the pre-trained inter-class expander layer 410 based on various criteria (such as functionality, training class, or other appropriate criteria).

[0093] It should be recognized that the first block 412 may not be the initial block of the pre-trained base model 402. For example, the pre-trained base model 402 can process the custom class training dataset 302 through various initial blocks until the processing reaches an intermediate block, which in Figure 4The first block is described as 412. For the sake of brevity, the initial block is not shown.

[0094] Furthermore, the untrained inter-class expander layer 418 of the untrained inter-class expander model 406 can also be connected to and receive the output of the corresponding first block 412 from the pre-trained base model 402. Specifically, the untrained fusion layer 424 of the untrained inter-class expander layer 418 can receive the custom class training dataset 302 and the output of the first block 412 via an input interface (e.g., an input layer). Regarding... Figure 5 A more detailed description of the untrained fusion layer 424 is provided.

[0095] An untrained inter-class expander model 406 can be trained by the augmentation model training system 300. Furthermore, the untrained inter-class expander model 406 can be an untrained inter-class expander model created in step 126. The untrained inter-class expander model 406 can be connected to a pre-trained base model 402 (e.g., the base model identified in step 110), as indicated in step 128. Using the methods described herein, the inter-class augmentation model 400 can be trained such that it can perform one or more classifications (e.g., cat breeds) that differ from the classifications of the pre-trained base model 402 (e.g., cats). Zero or more pre-trained inter-class expander models 404 can be connected to the pre-trained base model 402 to perform one or more classifications (e.g., dogs) that also differ from the classifications of the pre-trained base model 402 (e.g., cats). Therefore, in the case where a customer wants to expand the inter-class augmentation model for a new classification (e.g., cat breeds), an untrained inter-class expander model 406 can be connected to a pre-trained base model 402 and zero or more pre-trained inter-class expander models 404 (if any). The untrained inter-class expander model can then be trained to classify cat breeds as indicated in step 130.

[0096] The inter-class augmentation model 400 can receive a custom-class training dataset 302. For example, the custom-class training dataset 302 can be images (e.g., labeled images of cat breeds) where a pre-trained base model 402 is trained to classify cats, and a pre-trained inter-class expander model 408 is trained to classify dogs. In this scenario, zero or more pre-trained inter-class expander models 404 may include only pre-trained inter-class expander model 408. In other scenarios, zero or more pre-trained inter-class expander models 404 may not include any pre-trained inter-class expander models or may include more than one pre-trained inter-class expander model. The custom-class training dataset 302 can be received by a pre-trained base model 402, which can process the custom-class training dataset 302 according to its training. For example, if the pre-trained base model 402 is trained to classify cats, then the pre-trained base model 402 can identify cats from the images in the custom-class training dataset 302. The output of the first block 412 can be a set of activation values, which can be used by subsequent blocks (e.g., the second block 417) to classify cats from images. The first linear head 420 can output a prediction about whether an image in the custom category training dataset 302 contains a cat.

[0097] The first block 412 can also feed its output to the trained fusion layer 414. For example, the first block 412 can generate an input sequence represented by a set of activation values ​​that describe a portion of the training images. The pre-trained fusion layer 414 can use an attention mechanism to generate attention scores from the input sequence. For example, the attention mechanism can be used to generate query and key vectors from the input sequence. The attention mechanism can then take the dot product of the corresponding key and query vectors. The dot product can be a measure of the similarity between the key and query vectors. The pre-trained fusion layer 414 can then use a function (e.g., the SoftMax function) to generate a probability distribution that indicates the relative importance of each input in the input sequence to the desired category based on the dot product. For example, the pre-trained in-class expander model 408 can be trained to classify dogs. Therefore, the probability distribution can indicate the relative importance of each input in the input sequence for classifying dogs. Thus, even though the first block 412 of the pre-trained base model 402 is generating activation values ​​to assist in classifying cats, the weighted activation mechanism allows the pre-trained in-class expander model 408 to focus on dogs. This probability distribution can be used to generate a weighted sum. The weighted sum can be passed to the pre-trained inter-class expander 426. The pre-trained inter-class expander 426 can pass an image through a set of convolutional layers and use the weighted sum as a guide to identify features associated with dogs.

[0098] The untrained inter-class expander model 406 can also receive the custom category training dataset 302 and the output of the first block 412 (e.g., the input sequence). The untrained fusion layer 424 can be trained to generate attention scores using an attention mechanism on the input sequence. The attention mechanism can be used to generate query and key vectors from the input sequence. The attention mechanism can then take the dot product of the corresponding key and query vectors. The untrained fusion layer 424 can be trained to generate a probability distribution using a function (e.g., the SoftMax function) to indicate the relative importance of each input in the input sequence to the desired category based on the dot product. It should be recognized that the untrained inter-class expander model 406 will be trained to predict the custom category (e.g., cat breed). Therefore, the probability distribution can indicate the relative importance of each input in the input sequence to the custom category. This probability distribution can be used to generate a weighted sum. The weighted sum can be passed to the untrained inter-class expander 432. The untrained inter-class expander 432 can be trained to pass images through a set of convolutional layers and use the weighted sum as a guide to identify features associated with the custom category.

[0099] The output of the untrained inter-class expander 432 can be fed to a second untrained fusion layer 434. The second untrained fusion layer 434 can also receive the output of the second block 417 and the output of the pre-trained inter-class expander 426 (if zero or more pre-trained inter-class expander models 404 include a pre-trained inter-class expander model 408). The second untrained fusion layer 434 can be trained to similarly generate a weighted sum using the outputs of the untrained inter-class expander 432, the pre-trained inter-class expander 426, and the second block 417. The weighted sum can be fed to the second inter-class expander 436. This process can be repeated until the untrained inter-class expander model 406 outputs a classification using the third linear head 438. For example, the third linear head 438 can output a prediction of the breed of cat in images of a custom-category training dataset 302.

[0100] As used herein, each block (e.g., the first block 412 and any subsequent blocks) and expanders (e.g., any subsequent expanders of the pre-trained inter-class expander 426 and the pre-trained inter-class expander model 408, and any subsequent expanders of the untrained inter-class expander 432 and the untrained inter-class expander model 406) may include one or more layers (e.g., convolutional layers) that implement a forward method to transform the input into the output. Each must also be able to implement a backward method to compute gradients that can be used to determine the accuracy of the block.

[0101] The number of blocks in the pre-trained base model 402 is not necessarily the same as the number of inter-class expanders in the pre-trained inter-class expander model 408 or the untrained inter-class expander model 406. In many cases, the initial blocks of the pre-trained model perform basic tasks, and the outputs of these tasks are not fed to any fusion layer associated with the pre-trained inter-class expander model 408 or the untrained inter-class expander model 406. Instead, the intermediate blocks of the pre-trained base model 402 perform more complex tasks, which can be used as the basis for training the pre-trained inter-class expander model 408 or the untrained inter-class expander model 406. Therefore, in some cases, the number of blocks in the pre-trained model is greater than the number of inter-class expanders in the pre-trained inter-class expander model 408 or the untrained inter-class expander model 406.

[0102] As indicated in step 132, training can be in a locked or unlocked configuration. If the configuration is locked, then the parameters are for the pre-trained base model 402. In the locked configuration, the augmentation model training system 300 can adjust the weights of the untrained inter-class expander model 406. For example, the augmentation model training system 300 uses an algorithm to determine the accuracy of the untrained inter-class expander model 406. The augmentation model training system 300 can use backpropagation to adjust the weights of the untrained inter-class expander model 406. The weights of the pre-trained base model 402 and zero or more pre-trained inter-class expander models 404 can be locked so that their weights are not adjusted.

[0103] In an unlocked configuration, the augmentation model training system 300 can determine the accuracy of the untrained inter-class expander model 406, zero or more pre-trained inter-class expander models 404, and the pre-trained base model 402. The augmentation model training system 300 can use backpropagation to adjust the weights of the untrained inter-class expander model 406, zero or more pre-trained inter-class expander models 404, and the pre-trained base model 402.

[0104] It should be recognized that the techniques used in this paper can be applied to tasks other than object detection. For example, the pre-trained base model 402 can be trained for audio recognition, and the training data can be audio signals.

[0105] The augmented model training system 300 can also store the untrained inter-class expander model 406 (which has now been trained) in the model data repository 212. As indicated in step 136, if the untrained inter-class expander model 406 was trained using an unlocked configuration, then the augmented model training system 300 can store the untrained inter-class expander model 406, zero or more pre-trained inter-class expander models 404, and pre-trained base model 402 in the data repository. This is because the weights of the untrained inter-class expander model 406, zero or more pre-trained inter-class expander models 404, and pre-trained base model 402 have been adjusted during the training process. Alternatively, as indicated in step 138, if the untrained inter-class expander model 406 was trained using a locked configuration, then the augmented model training system 300 can store the untrained inter-class expander model 406 in the model data repository 212. This is because only the weights of the untrained inter-class expander model 406 have been adjusted during the training process.

[0106] In either case, the augmentation model training system 300 can update the metadata associated with the pre-trained base model 402 and zero or more pre-trained inter-class expander models 404 to associate these models with the untrained inter-class expander model 406. In this sense, if a user wants a machine learning model to predict the categories that the untrained inter-class expander model 406 has been trained to classify, the user can retrieve the corresponding pre-trained base model 402 and zero or more pre-trained inter-class expander models 404 (if desired). Alternatively, if a user wants to add a new untrained inter-class expander model (e.g., a model for classifying dog breeds), the user can retrieve the corresponding pre-trained base model 402, zero or more pre-trained inter-class expander models 404, and the untrained inter-class expander model 406.

[0107] Figure 5This is an illustration of an example fusion layer 500 according to one or more embodiments. Fusion layer 500 may be part of an inter-class expander layer (e.g., a pre-trained inter-class expander layer 410 or an untrained inter-class expander layer 418). Fusion layer 500 may use an attention mechanism to generate attention scores based on an input sequence represented as activation values. For example, fusion layer 500 may use an attention mechanism to generate a key vector and a corresponding query vector based on the input sequence. The attention mechanism may also determine the dot product of the key vector and the corresponding query vector. Fusion layer 500 may also use a SoftMax function 502 to transform the attention scores into a probability distribution. The probability distribution may indicate the relative importance of each input in the input sequence to the class. Fusion layer 500 may then generate a value vector from the input sequence. Fusion layer 500 may multiply the probability distribution by the value vector to generate a weighted sum 504. The weighted sum may be passed to an inter-class expander 506 to extract features from the image using the weighted sum as a guide.

[0108] The attention mechanism determines a set of attention scores that indicate the relevance of each input in the input sequence to a category. For example, if a pre-trained base model is to be trained to identify cat breeds, then the scores indicate the relevance of each input in the input sequence to the cat breed. A softmax function 502 can be used to transform each of these scores into a probability distribution. The probability distribution associates each attention score with a number between 0 and 1, where the sum of the numbers equals 1. The fusion layer 500 can then generate a weighted sum 504 and pass the weighted sum 504 to the inter-class expander 506.

[0109] Figure 6This is a processing flow 600 using an inter-class extender model according to one or more embodiments. While the operations of processes 600, 1000, 1100, and 1200 are described as being performed by a general-purpose computer, it should be understood that any suitable device can be used to perform one or more operations of these processes. Processes 600, 1000, 1100, and 1200 (described below) are illustrated as logic flowcharts, where each operation represents a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, an operation represents a computer-executable instruction stored on one or more computer-readable storage media that, when executed by one or more processors, performs the operation. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc., that perform a particular function or implement a particular data type. The order in which operations are described is not intended to be construed as limiting, and any number of described operations can be combined in any order and / or in parallel to implement the processing. At 602, the computing system can identify an augmentation model for performing data classification. For example, a user can provide data and request a category prediction from the data. For example, a user can provide an image of an animal and a request to categorize the cat breeds from the image.

[0110] At 604, the computational system can construct the model identified in 602 using a pre-trained base model and one or more pre-trained inter-class expanders. For example, the computational system can access data repository 212 and search for models that can classify cat breeds. The computational system can identify the domain (e.g., cat) associated with cat breeds and identify the pre-trained base model trained to classify cats. The computational system can access the metadata associated with the pre-trained base model and use a mapping from the metadata associated with the pre-trained base model to the metadata associated with one or more inter-class expander models. The computational system can then concatenate the pre-trained base model with one or more inter-class expander models to generate an augmented model.

[0111] At 606, the computational system can provide the data to be classified as input to the augmented model constructed in 604. The data may include, for example, a set of animal images containing one or more breeds of cats.

[0112] At point 608, the computational system can receive one or more outputs from the augmentation model that identify the category of the input data. For example, the computational system can receive an indication from a pre-trained base model that an image contains a cat. The computational system can also receive an indication of the breed of cat in the image from one or more pre-trained inter-class extender models.

[0113] Inner class extender

[0114] Figure 7-9 You can view them together. Figure 7This is a diagram 700 of an example convolutional neural network according to one or more embodiments. Specifically, Figure 7 This is a diagram of a Mobile-Net V3 architecture with inner class extender units. It should be recognized that providing... Figure 7 This is for illustrative purposes only, and the inner class expander is compatible with various neural networks. Figure 8 This is an illustration of a portion of an inner class extender unit according to one or more embodiments. Figure 9 This is an illustration of an example portion of an inner-class expander unit according to one or more embodiments. Unlike the inter-class expander, which keeps the inner activation values ​​of the backbone network and previous inter-class branches constant, adding an inner-class expander to the base architecture results in changes to the output activation values ​​to adapt to the new task dataset. In other words, supplementary task-specific parameters are fused at multiple instances within the pre-trained base model before producing the final block output.

[0115] A computational system can create an inner class enhancement model 702 by connecting a pre-trained base model to zero or more pre-trained inner class expander models and by connecting an untrained inner class expander model to a pre-trained base model. The zero or more pre-trained inner class expander models and the untrained inner class expander model are related to... Figure 8 and Figure 9 To describe this in more detail, zero or more pre-trained inner class expander models and untrained inner class expander models are uniformly illustrated as follows: Figure 7 The inner class extender unit 704 in the middle.

[0116] The inner-class augmentation model 702 can be trained by the augmentation model training system 300, as indicated in 158. The inner-class augmentation model 702 can receive a custom-class training dataset 302 via the augmentation model training system 300. In this example, the custom-class training dataset 302 may include images of horses instead of images of cats and dogs. In some embodiments, upon receiving an image, the inner-class augmentation model 702 may perform an expansion and depthwise convolution 706 operation, where a different filter is used for each input channel, and the input channels may correspond to features of the input. For example, for an image, the input channels may correspond to pixel colors (e.g., red, green, and blue). The inner-class augmentation model 702 may include filters for each color. For audio training instances, the input channels may correspond to different frequency bands, and the inner-class augmentation model 702 may include filters for different frequency bands. In some cases, this operation may result in several modified expanded channels 708. For example, for an image, the channels may be expanded to include additional colors (e.g., yellow, black, etc.). The modified extended channel 708 can be received by the inner class extender unit 704, and the inner class extender unit 704 is related to... Figure 8 and Figure 9 To describe in more detail. The continuation of this process is also about Figure 8 and Figure 9 Further description is required.

[0117] Figure 8 This is an illustration of a portion 800 of an example inner class expander unit 704 according to one or more embodiments. In some embodiments, an adaptive average pooling operation is applied to each channel of the modified expanded channel 708 to modify the spatial dimension of each channel to a desired spatial dimension. For example, input channels may each be represented as input feature maps to which the adaptive average pooling operation is applied. The inner class enhancement model 702 may also have access to the desired output feature map size. The inner class enhancement model 702 may also determine the dimension of the output feature map based on the desired map size and perform a pooling operation (e.g., an adaptive average pooling operation). For each region of the output feature map, the neural network may determine a value corresponding to the region of the input feature map. The inner class enhancement model 702 may then generate an output feature map 802 based on this determination. The output feature map 802 may be used as input and propagated to a first base block 804, a first pre-trained inner class expander layer 806 of zero or more pre-trained inner class expander models 808, and a first untrained inner class expander layer 810 of an untrained inner class expander model 812. It should be recognized that, in some embodiments, a pre-trained inter-class expander model is not present. For example, if a pre-trained inter-class expander model is not identified at step 150, then a pre-trained inter-class expander model is not present.

[0118] As shown in the figure, the output feature map 802 propagates through a first base block 804, a first untrained inner class expander layer 810, and a first untrained inner class expander layer. The first base block 804, the first pre-trained inner class expander layer 806, and the first untrained inner class expander layer 810 may each include one or more corresponding filters that can process the output feature map 802 based on the respective categories on which they are trained or to be trained. For example, if the first base block 804 is pre-trained to classify horses in an image, then the first base block 804 can use one or more filters to process horse-related features in the output feature map. If the first pre-trained inner class expander layer 806 is pre-trained to classify zebra breeds, then the first pre-trained inner class expander layer 806 can use one or more filters to identify zebra breed-related features. If the first untrained in-class expander layer 810 is to be trained to classify horse breeds, then the first untrained in-class expander layer 810 can process the features associated with horse breeds to output feature map 802. Each output (e.g., first output 814, second output 816, and third output 818) of the first base block 804, the first pre-trained in-class expander layer 806, and the first untrained in-class expander layer 810 can be passed to the first fusion layer 820. As shown, the outputs are passed to the common first fusion layer 820 compared to the inter-class enhancement model 400. Furthermore, the first fusion layer 820 is to be trained for classifying new categories (e.g., horse breeds). Therefore, the first fusion layer 820 can be trained to generate a set of common attention scores associated with the desired new category. The first fusion layer 820 can also generate a probability distribution based on the attention scores and further generate a weighted sum based on the probability distribution.

[0119] The first fusion layer 820 can transmit the first fusion layer output 822 (e.g., a weighted sum) to the second base block 824, the second pre-trained inner class expander layer 826, and the second untrained inner class expander layer 828. The second base block 824, the second pre-trained inner class expander layer 826, and the second untrained inner class expander layer 828 can perform functions similar to those of the first base block 804, the first pre-trained inner class expander layer 806, and the first untrained inner class expander layer 810. However, given the first fusion layer output 822, the second base block 824, the second pre-trained inner class expander layer 826, and the second untrained inner class expander layer 828 can use a weighted sum to determine a deeper correspondence between features and new categories. The second base block 824, the second pre-trained inner class expander layer 826, and the second untrained inner class expander layer 828 can transmit their outputs (e.g., a fourth output 830, a fifth output 832, and a sixth output 834) to the second fusion layer 836. The second fusion layer 836 can be functionally similar to the first fusion layer 820.

[0120] The inner-class enhancement model 702 can then perform a channel-wise multiplication operation, such that element-wise multiplication is performed between the output of the second fusion layer 836 and the channel-wise multiplier. This operation can be performed for feature modulation, allowing the inner-class enhancement model 702 to emphasize one or more channels over others. For example, the inner-class enhancement model 702 can use a channel-wise multiplication operation to emphasize channels that correspond more strongly to a new category (e.g., a breed of horse). The channel-wise multiplication operation can produce a set of expanded channels 838.

[0121] Figure 9 This is an illustration of an example portion 900 of an inner class extender unit according to one or more embodiments. Figure 9 As shown in the figure Figure 8 This is a continuation of the inner class expander unit 704. This set of expansion channels 838 can be transmitted to the third base block 902, the third pre-trained inner class expander layer 904, and the third untrained inner class expander layer 906. The third base block 902, the third pre-trained inner class expander layer 904, and the third untrained inner class expander layer 906 are functionally similar to the second base block 824, the second pre-trained inner class expander layer 826, and the second untrained inner class expander layer 828. The third base block 902, the third pre-trained inner class expander layer 904, and the third untrained inner class expander layer 906 can also transmit outputs (e.g., the seventh output 908, the eighth output 910, and the ninth output 912) to the third fusion layer 914. The third fusion layer is functionally similar to the second fusion layer 836. It should be recognized that, as shown, the inner class expander unit 704 includes three fusion layers. Those skilled in the art will envision that the inner class expander unit 704 may include various numbers of fusion layers to perform the functions described herein. The output of the third fusion layer 914 may be transmitted for downstream processing by the inner class enhancement model 702.

[0122] The in-class enhancement model 702 may include an output layer 916 for outputting predictions about a new category (e.g., the breed of horse). For example, output layer 916 may generate predictions about the breed of horse displayed on images in the custom category training dataset 302.

[0123] The augmentation model training system 300 can use algorithms to determine the accuracy of the inner class augmentation model 702. The augmentation model training system 300 can compare the accuracy to a threshold accuracy. The augmentation model training system 300 can use backpropagation to adjust the weights of the inner class augmentation model 702 to improve accuracy. As indicated in step 158, training can be in a locked or unlocked configuration. If training is in a locked configuration, then only the weights of the untrained inner class expander model 812 are adjusted. However, if training is in an unlocked configuration, then the augmentation model training system 300 adjusts the weights of the base models (e.g., the first base block 804, the second base block 824, and the third base block 902), zero or more pre-trained inner class expander models 808, and the untrained inner class expander model 812.

[0124] If training is in an unlocked configuration, the augmentation model training system 300 can store the base model, zero or more pre-trained inner class expander models 808, and untrained inner class expander models 812 (now trained) in the data repository 212, as indicated in 162. If training is in a locked configuration, the augmentation model training system 300 can store the untrained inner class expander models 812 (now trained) in the data repository 212, as indicated in 164. The augmentation model training system 300 can then update the metadata associated with the base model to indicate the creation of the inner class augmentation model 702 (including zero or more pre-trained inner class expander models 808 and untrained inner class expander models 812 (now trained)).

[0125] At some point, a user might want access to an untrained inner class expander model 812 (which is now trained). For example, a user might want a model that can classify horse breeds from an input image. The computational system can access a data repository and identify the untrained inner class expander model 812 (which is now trained) based on metadata. For example, the computational system can use keyword search or other appropriate search methods to identify the untrained inner class expander model 812 (which is now trained). In this case, the computational system can retrieve the base model, zero or more pre-trained inner class expander models 808, and the untrained inner class expander model 812 (which is now trained) from the data repository 212. This is because the fusion layers (e.g., the first fusion layer 820, the second fusion layer 836, and the third fusion layer 914) have not yet been trained for the recognition of different categories, as the inter-class augmentation model 400 has. Instead, the fusion layers have been trained using the outputs from the base block, the pre-trained inner class expander layers (if any), and the untrained inner class expander layers to generate a common output for the new category. The output of each fusion layer indicates that it receives outputs from the base model, zero or more pre-trained inner class expander models 808, and untrained inner class expander models 812. Therefore, the untrained inner class expander model 812 cannot be decoupled from the base model and zero or more pre-trained inner class expander models 808 (if any).

[0126] The embodiments described herein provide a single-stage CL training method feasible for industrial-custom modeling settings using relatively few effective training resources. The embodiments eliminate the neural network's dependence on previous task datasets provided by the client. The embodiments described herein enable the creation of meta-repositories / model repositories with domain-specific model blocks. The embodiments described herein can be used to create target models on the fly by selectively combining task-specific parameters from their respective domains. The embodiments described herein further provide clients with fine-grained control over the selection and fusion of domain and task-specific parameters from past iterations. The embodiments described herein can continuously improve performance on the target domain without impacting the performance of previous tasks and datasets. The embodiments described herein can reduce the overall cost of adoption for clients in terms of time, effort, and financial resources.

[0127] Figure 10 This is a processing flow 1000 using an inter-class expander model according to one or more embodiments. At 1002, the computing system can identify an augmented model for performing data classification. For example, a user can provide data and a request to predict categories from the data. For example, a user can provide an image of an animal and a request to classify the breed of horse from the image.

[0128] At 1004, the computational system can construct the model identified in 1002 using a pre-trained base model and one or more pre-trained in-class expanders. For example, the computational system can access the model data repository 212 and search for models that can classify horse breeds. The computational system can identify the domain (e.g., horse) associated with horse breeds and identify the pre-trained base model trained to classify horses. The computational system can access the metadata associated with the pre-trained base model and use a mapping from the metadata associated with the pre-trained base model to the metadata associated with one or more in-class expander models. The computational system can then concatenate the pre-trained base model with one or more in-class expander models to generate an augmented model.

[0129] At point 1006, the computational system can provide the data to be classified as input to the augmented model built in point 1004. The data may include, for example, a set of animal images containing one or more breeds of horses.

[0130] At point 1008, the computing system can receive one or more outputs from the augmented model that identify categories applicable to the input data. For example, the computing system can receive indications of one or more horse breeds contained in an image.

[0131] Figure 11 This is an example processing flow 1100 for training an inter-class expander model according to one or more embodiments. At 1102, a computing system (e.g., an augmentation model training system) can identify custom categories. For example, the computing system can receive a request for a model that can classify custom categories.

[0132] At point 1104, the computational system can select a pre-trained base model trained to predict base classes. This selection can be based at least in part on base classes belonging to the same domain as the custom class. For example, the computational system can determine the domain associated with the custom class and access a data repository. The computational system can also map domain custom classes to domains identified by metadata associated with the pre-trained base model.

[0133] At point 1106, the computational system can generate an untrained inter-class expander model, which is trained to predict a custom class from data points associated with that class. The untrained inter-class expander model can include an untrained fusion layer and an untrained inter-class expander. For example, if the computational system cannot identify the pre-trained model, it can determine how to train an augmented model.

[0134] At point 1108, the computational system can concatenate the pre-trained base model with the untrained inter-class expander model in parallel to generate an inter-class augmentation model. The pre-trained models can be concatenated such that the first block of the pre-trained base model is connected to the fusion layer of the untrained inter-class expander model, as indicated above.

[0135] At 1110, the computational system can input training images into the inter-class augmentation model, which include data points associated with a custom class. The training images can be images from the custom class training dataset 302.

[0136] At 1112, the computational system can, at least in part, base its output on the training images by associating the first block of the pre-trained base model with the first set of activation values ​​associated with the base class. The pre-trained base model can pass images through a set of convolutional layers and output the input sequence represented by the activation values.

[0137] At 1114, the computational system can enable the untrained fusion layer to generate a set of weights for weighting the first set of activation values. This set of weights is configured to enable the untrained inter-class expander model to recognize a custom class from the training images. The untrained fusion layer can use an attention mechanism to generate this set of weights, which is configured to enable the untrained inter-class expander model to recognize a custom class from the training images.

[0138] At 1116, the computational system can enable the untrained fusion layer to generate a weighted sum, at least partially, based on weighting the first set of activations using that set of weights. The untrained fusion layer can generate a value vector based on the output of the first block. The untrained fusion layer can also multiply the set of weights by the value vector to generate a weighted sum.

[0139] At 1118, the computational system can enable an untrained inter-class expander to generate a second set of activation values, at least partially based on a weighted sum. The untrained inter-class expander can use the weighted sum as a guide to pass the image through a set of convolutional layers to generate the second set of activation values.

[0140] At 1120, the computational system can enable an untrained inter-class expander model to predict a custom class from the training images, at least partially, based on a second set of activation values. At 1122, the computational system can enable a pre-trained base model to predict a base class from the training images, at least partially, based on a first set of activation values.

[0141] Figure 12 This is an example processing flow 1200 for training an inner class expander model according to one or more embodiments. At 1202, the computing system (e.g., an augmentation model training system) can identify custom categories.

[0142] At point 1204, the computational system can select a pre-trained base model trained to predict base classes. This selection can be based at least in part on a base class belonging to the same domain as the custom class. For example, the computational system can determine the domain associated with the custom class and access a data repository. The computational system can also map domain custom classes to domains identified by metadata associated with the pre-trained base model.

[0143] At 1206, the computing system can generate an untrained inner class expander model, which is trained to identify custom classes from data points associated with custom classes. The untrained inner class expander model includes an untrained fusion layer, a first untrained inner class expander layer, and a second untrained expander layer.

[0144] At point 1208, the computational system can connect the inner class expander machine learning model to the pre-trained base machine learning model to generate an inner class augmentation model. The pre-trained model can be connected such that the first block of the pre-trained base model and the inner class expander are connected to a common fusion layer, as indicated above.

[0145] At 1210, the computational system can input the training image into the in-class augmentation model, which includes data points associated with a custom category.

[0146] At 1212, the computational system can, at least in part, base the pre-trained base model on the training image, causing the first block of the pre-trained base model to output a first plurality of activation values. The first block of the pre-trained base model can pass the image through a set of convolutional layers and output the input sequence represented by the activation values ​​for reception by a common fusion layer.

[0147] At 1214, the computational system can, at least in part, base its operation on the training image, causing the first untrained inner class expander layer to output a second plurality of activation values. The first untrained inner class expander layer can pass the image through a set of convolutional layers and output an input sequence represented by activation values ​​for reception by a common fusion layer.

[0148] At 1216, the computational system can enable the untrained fusion layer to generate multiple weights for weighting a first plurality of activations and a second plurality of activations, which are configured to enable the in-class enhancement model to recognize custom categories from training images.

[0149] At point 1218, the computational system can enable the untrained fusion layer to generate a weighted sum, at least in part, based on weighting a first plurality of activations and a second plurality of activations using multiple weights. The common fusion layer can generate a value vector based on the output of the first block. The untrained fusion layer can also multiply the set of weights by the value vector to generate a weighted sum.

[0150] At 1220, the computational system can make the second block of the pre-trained base model at least partially based on a weighted sum to output a third plurality of activation values, and at 1222, the computational system can make the second untrained expander layer at least partially based on a weighted sum to output a fourth plurality of activation values.

[0151] At point 1222, the computational system enables the in-class augmentation model to predict custom categories from training images, at least in part, based on a third and fourth set of activation values. For example, the in-class augmentation model can predict the breed of a horse in an image.

[0152] Figure 13 This is an example processing flow 1300 for maintaining a data repository according to one or more embodiments. At 1302, a computing system (e.g., data repository 212) may receive a message requesting a model trained to predict custom categories. For example, the computing system may receive a user request for a model that can predict custom categories.

[0153] At point 1304, the computational system can select from multiple pre-trained base models that have been trained to predict base categories belonging to the same domain as the custom category. For example, the computational system can determine the domain associated with the custom category. The computational system can also map the domain custom category to the domain identified by the metadata associated with the pre-trained base model.

[0154] At point 1306, the computational system can select a first pre-trained expander model from multiple pre-trained expander models. The first pre-trained expander model can be at least partially based on the selection of a pre-trained base model associated with it. The first pre-trained expander model can be trained to predict custom categories.

[0155] At 1308, the computational system can generate an augmented model by connecting a pre-trained base model to the first pre-trained expander model, which is trained at least in part to predict custom categories. The pre-trained base model can be connected to the first pre-trained expander model such that the first output of the first block of the pre-trained base model is received by the first pre-trained fusion layer of the pre-trained expander model.

[0156] At point 1310, the computing system can transmit the augmented model to the message sender.

[0157] Example

[0158] Further exemplary embodiments are provided in the following sections.

[0159] Example 1 includes a method comprising: identifying a custom category by a computational system; selecting a pre-trained base model trained by the computational system to predict a base category, the selection being at least partially based on a base category belonging to the same domain as the custom category; generating an untrained inter-class expander model by the computational system, the model being trained to predict the custom category from data points associated with the custom category, the untrained inter-class expander model including an untrained fusion layer and an untrained inter-class expander; concatenating the pre-trained base model and the untrained inter-class expander model in parallel by the computational system to generate an inter-class augmentation model; inputting training images into the inter-class augmentation model by the computational system, the training images including data points associated with the custom category; and enabling the pre-trained base model to predict the custom category from data points associated with the custom category by the computational system at least partially based on the training images. The first block output of the base model is associated with a first plurality of activation values ​​of a base class; the computation system causes an untrained fusion layer to generate a plurality of weights for weighting the first plurality of activation values, the plurality of weights being configured to enable an untrained inter-class expander model to identify a custom class from training images; the computation system causes the untrained fusion layer to generate a weighted sum based at least partially on weighting the first plurality of activation values ​​using the plurality of weights; the computation system causes the untrained inter-class expander to generate a second plurality of activation values ​​based at least partially on the weighted sum; the computation system causes the untrained inter-class expander model to predict a custom class from training images based at least partially on the second plurality of activation values; and the computation system causes a pre-trained base model to predict a base class from training images based at least partially on the first plurality of activation values.

[0160] Example 2 includes the method of Example 1, wherein the method further includes: generating multiple vectors at least partially based on training images; generating multiple activation scores at least partially based on the vectors; and generating a probability distribution at least partially based on the multiple activation scores, wherein the probability distribution includes multiple weights.

[0161] Example 3 includes the method of any one of Examples 1 and 2, wherein the inter-class augmentation model is trained using a locked configuration, and wherein the method further includes: determining the accuracy of the untrained inter-class expander model based at least in part on a loss function; and adjusting the weights of a plurality of untrained inter-class expander models based at least in part on the accuracy, wherein the weights of a plurality of pre-trained base models remain unadjusted.

[0162] Example 4 includes the method of any one of Examples 1 and 2, wherein the inter-class augmentation model is trained using an unlocked configuration, and wherein the method further includes: determining the accuracy of the untrained inter-class expander model based at least in part on a loss function; adjusting the weights of a plurality of pre-trained base models based at least in part on the accuracy; and adjusting the weights of a plurality of untrained inter-class expander models based at least in part on the accuracy.

[0163] Example 5 includes a method of any one of Examples 1-4, wherein the method further includes: accessing a data repository storing a pre-trained base model; identifying a pre-trained inter-class extender model based at least in part on metadata stored in the data repository and associated with the pre-trained inter-class extender model, wherein the pre-trained inter-class extender model is trained to identify categories belonging to the same domain as a custom category; and concatenating the pre-trained inter-class extender model to the pre-trained base model and an untrained inter-class extender model.

[0164] Example 6 includes the method of Example 5, wherein the method further includes: in response to an untrained inter-class expander model recognizing a custom category from training images, transmitting a first control instruction to a data repository to store the untrained inter-class expander model; and transmitting a second control instruction to the data repository to update metadata associated with the pre-trained base model to indicate the generation of the inter-class augmentation model.

[0165] Example 7 includes a method from any of Examples 1-6, wherein the method further includes: determining, at least in part, whether to generate an inter-class enhancement model based on a custom category.

[0166] Example 8 includes a computing system comprising one or more processors; and one or more non-transitory computer-readable media comprising a sequence of instructions that, when executed on the one or more processors, cause the one or more processors to perform any of the steps in Examples 1-7.

[0167] Example 9 includes one or more non-transitory computer-readable media, which include a sequence of instructions that, when executed by one or more processors, cause a computing system to perform any of the steps in Examples 1-7.

[0168] Example 10 includes a computing system comprising: one or more processors; and one or more non-transitory computer-readable media including a sequence of instructions that, when executed on the one or more processors, cause the one or more processors to: identify the identity of a custom class; select a pre-trained base model trained to predict a base class, the selection being at least partially based on a base class belonging to the same domain as the custom class; generate an untrained inner class expander model trained to identify the custom class from data points associated with the custom class, the untrained inner class expander model including an untrained fusion layer, a first untrained inner class expander layer, and a second untrained expander layer; connect the untrained inner class expander model to the pre-trained base model to generate an inner class augmentation model; and input training images into the inner class augmentation model, the training images being... The method includes: including data points associated with a custom category; at least partially based on training images, causing a first block of a pre-trained base model to output a first plurality of activation values; at least partially based on training images, causing a first untrained inner class expander layer to output a second plurality of activation values; causing an untrained fusion layer to generate a plurality of weights for weighting the first plurality of activation values ​​and the second plurality of activation values, the plurality of weights being configured to enable the inner class enhancement model to identify a custom category from the training images; causing the untrained fusion layer to generate a weighted sum at least partially based on weighting the first plurality of activation values ​​and the second plurality of activation values ​​using the plurality of weights; causing a second block of the pre-trained base model to output a third plurality of activation values ​​at least partially based on the weighted sum; causing a second untrained expander layer to output a fourth plurality of activation values ​​at least partially based on the weighted sum; and causing the inner class enhancement model to predict a custom category from the training images at least partially based on the third plurality of activation values ​​and the fourth plurality of activation values.

[0169] Example 11 includes the computational system of Example 10, wherein when the instruction sequence is executed on one or more processors, the one or more processors further cause the one or more processors to: generate a plurality of vectors at least partially based on training images; generate a plurality of activation scores at least partially based on the vectors; and generate a probability distribution at least partially based on the plurality of activation scores, wherein the probability distribution includes a plurality of weights.

[0170] Example 12 includes the computational system of Example 11, wherein the inner class augmentation model is trained using a locked configuration, and wherein when the instruction sequence is executed on one or more processors, the one or more processors also cause the one or more processors to: determine the accuracy of the untrained inner class expander model at least in part based on a loss function; and adjust the weights of a plurality of untrained inner class expander models at least in part based on the accuracy, wherein the weights of a plurality of pretrained base models remain unadjusted.

[0171] Example 13 includes the computational system of Example 11, wherein the inner class augmentation model is trained using an unlocked configuration, and wherein when the instruction sequence is executed on one or more processors, the one or more processors also cause the one or more processors to: determine the accuracy of the untrained inner class expander model at least in part based on a loss function; adjust the weights of a plurality of pretrained base models at least in part based on the accuracy; and adjust the weights of a plurality of untrained inner class expander models at least in part based on the accuracy.

[0172] Example 14 includes a computing system comprising any one of Examples 10-13, wherein when the sequence of instructions is executed on one or more processors, the one or more processors also cause the one or more processors to: access a data repository storing a pre-trained base model; identify a pre-trained inner class extender model based at least in part on metadata stored in the data repository and associated with the pre-trained inner class extender model, wherein the pre-trained inner class extender model is trained to identify categories belonging to the same domain as a custom category; and connect the pre-trained inner class extender model to the pre-trained base model and an untrained inner class extender model.

[0173] Example 15 includes the computing system of Example 14, wherein when the sequence of instructions is executed on one or more processors, the one or more processors also cause the one or more processors to: transmit a first control instruction to a data repository to store the untrained inner class expander model in response to an untrained inner class expander model recognizing custom categories from training images; and transmit a second control instruction to the data repository to update the metadata associated with the pretrained base model to indicate the generation of the inner class augmentation model.

[0174] Example 16 includes a computing system of any of Examples 10-15, wherein when the sequence of instructions is executed on one or more processors, the one or more processors also cause the one or more processors to: determine, at least in part, whether to generate an inner class enhancement model based on a custom category.

[0175] Example 17 includes a method for performing any of the steps in Examples 10-16.

[0176] Example 18 includes one or more non-transitory computer-readable media, which include a sequence of instructions that, when executed by one or more processors, cause a computing system to perform any of the steps in Examples 10-16.

[0177] Example 19 includes one or more non-transitory computer-readable media, the one or more non-transitory computer-readable media including a sequence of instructions that, when executed by one or more processors, cause a computing system to: identify a custom class; select a pre-trained base model trained to predict a base class, the selection being at least partially based on a base class belonging to the same domain as the custom class; generate an untrained inter-class expander model trained to predict the custom class from data points associated with the custom class, the untrained inter-class expander model including an untrained fusion layer and an untrained inter-class expander; concatenate the pre-trained base model with the untrained inter-class expander model in parallel to generate an inter-class augmentation model; and input training images into the inter-class augmentation model, the training images including data points associated with the custom class. The system comprises: class-associated data points; first plurality of activation values ​​associated with a first block of output of a pre-trained base model with a base class, based at least in part on training images; an untrained fusion layer generating a plurality of weights for weighting the first plurality of activation values, the plurality of weights being configured to enable an untrained inter-class expander model to identify a custom class from the training images; the untrained fusion layer generating a weighted sum based at least in part on weighting the first plurality of activation values ​​using the plurality of weights; the untrained inter-class expander generating a second plurality of activation values ​​based at least in part on the weighted sum; the untrained inter-class expander model predicting a custom class from the training images based at least in part on the second plurality of activation values; and the pre-trained base model predicting a base class from the training images based at least in part on the first plurality of activation values.

[0178] Example 20 includes one or more non-transitory computer-readable media of Example 19, wherein when the sequence of instructions is executed on one or more processors, the one or more processors further cause the one or more processors to: generate a plurality of vectors at least partially based on training images; generate a plurality of activation scores at least partially based on the vectors; and generate a probability distribution at least partially based on the plurality of activation scores, wherein the probability distribution includes a plurality of weights.

[0179] Example 21 includes one or more non-transitory computer-readable media of Example 20, wherein the inter-class augmentation model is trained using a locked configuration, and wherein when the instruction sequence is executed on one or more processors, the one or more processors further cause the one or more processors to: determine the accuracy of the untrained inter-class expander model at least in part based on a loss function; and adjust multiple weights of the untrained inter-class expander model at least in part based on the accuracy, wherein the weights of the pre-trained base model remain unadjusted.

[0180] Example 22 includes one or more non-transitory computer-readable media of Example 20, wherein the inter-class augmentation model is trained using an unlocked configuration, and wherein when the instruction sequence is executed on one or more processors, the one or more processors further cause the one or more processors to: determine the accuracy of an untrained inter-class expander model based at least in part on a loss function; adjust the weights of a plurality of pre-trained base models based at least in part on the accuracy; and adjust the weights of a plurality of untrained inter-class expander models based at least in part on the accuracy.

[0181] Example 23 includes one or more non-transitory computer-readable media of any of Examples 19-22, wherein when the sequence of instructions is executed on one or more processors, it also causes one or more processors to: access a data repository storing a pre-trained base model; identify a pre-trained inter-class extender model based at least in part on metadata stored in the data repository and associated with the pre-trained inter-class extender model, wherein the pre-trained inter-class extender model is trained to identify categories belonging to the same domain as a custom category; and connect the pre-trained inter-class extender model to the pre-trained base model and an untrained inter-class extender model.

[0182] Example 24 includes one or more non-transitory computer-readable media of any of Examples 19-23, wherein when the sequence of instructions is executed on one or more processors, it further causes one or more processors to: transmit a first control instruction to a data repository to store the untrained inter-class expander model in response to an untrained inter-class expander model recognizing custom categories from training images; and transmit a second control instruction to the data repository to update metadata associated with the pre-trained base model to indicate the generation of the inter-class augmentation model.

[0183] Example 25 includes a method for performing any of the steps in Examples 19-24.

[0184] Example 26 includes a computing system comprising one or more processors; and one or more non-transitory computer-readable media comprising a sequence of instructions that, when executed on the one or more processors, cause the one or more processors to perform any of the steps in Examples 19-24.

[0185] Example 27 includes a method comprising: receiving a message from a data repository requesting a model trained to predict a custom category; selecting from a plurality of pre-trained base models a pre-trained base model trained to predict a base category belonging to the same domain as the custom category; selecting from a plurality of pre-trained extender models a first pre-trained extender model, the first pre-trained extender model being associated with the selected pre-trained base model at least in part based on the selection of the pre-trained base model, the first pre-trained extender model being trained to predict the custom category; concatenating the pre-trained base model to the first pre-trained extender model by the data repository to generate an augmented model at least in part based on the first pre-trained extender model being trained to predict the custom category, the pre-trained base model being concatenated to the first pre-trained extender model such that a first output of a first block of the pre-trained base model is received by a first pre-trained fusion layer of the pre-trained extender model; and transmitting the augmented model to the sender of the message by the data repository.

[0186] Example 28 includes the method of Example 27, wherein the method further includes: receiving a message indicating that a pre-trained base model was used to train a second pre-trained extender model, the second pre-trained extender model being trained to predict categories belonging to the same domain as the base category and the custom category; generating metadata to associate the pre-trained base model with the second pre-trained extender model; identifying the second pre-trained extender model at least in part based on the metadata; and connecting the second pre-trained extender model to the pre-trained base model, wherein the augmented model includes the pre-trained base model, the first pre-trained extender model, and the second pre-trained extender model.

[0187] Example 29 includes the method of Example 28, wherein selecting a pre-trained base model trained to predict a base category belonging to the same domain as a custom category from a plurality of pre-trained base models includes: accessing first metadata stored in a data repository and associated with the pre-trained base model, wherein the first metadata includes the identity of the base category; and determining an association between the base category and the custom category using a mapping based at least in part on the first metadata and a message, wherein the pre-trained base model is selected based at least in part on the association.

[0188] Example 30 includes a method of any one of Examples 27-29, wherein selecting a first pre-trained expander model from a plurality of pre-trained expander models comprises: accessing first metadata stored in a data repository and associated with a pre-trained base model, wherein the first metadata includes the identity of a base category; accessing second metadata stored in a data repository and associated with the first pre-trained expander model, wherein the second metadata includes the identity of a custom category; and using a mapping to determine an association between the base category and the custom category based at least in part on the first metadata and the second metadata, wherein the first pre-trained expander model is selected at least in part based on the association.

[0189] Example 31 includes a method of any one of Examples 27-30, wherein the augmentation model is an inter-class augmentation model, and wherein connecting a pre-trained base model to a first pre-trained extender model to generate the augmentation model includes: connecting a first pre-trained fusion layer to an input interface of the augmentation model for receiving training instances, such that the first pre-trained fusion layer is configured to receive the output of a first block and training instances; connecting the first pre-trained fusion layer to a first pre-trained inter-class extender of the first pre-trained extender model; and connecting the first pre-trained inter-class extender to a second pre-trained fusion layer of the first pre-trained extender model.

[0190] Example 32 includes a method of any one of Examples 27-31, wherein the augmentation model is an in-class augmentation model, and wherein connecting a pre-trained base model to a first pre-trained extender model to generate the augmentation model includes: connecting a first pre-trained in-class extender layer to a first pre-trained fusion layer such that a second output of the first pre-trained in-class extender layer is received; and connecting the first pre-trained fusion layer to a second block of the pre-trained base model and a second pre-trained in-class extender layer such that a third output of the first pre-trained fusion layer is received by the second block and the second pre-trained in-class extender layer.

[0191] Example 32 includes a method of any one of Examples 27-32, wherein the method further includes: generating metadata that indicates the generation of the augmented model; and storing the metadata in a data repository.

[0192] Example 33 includes a computing system comprising one or more processors; and one or more non-transitory computer-readable media comprising a sequence of instructions that, when executed on the one or more processors, cause the one or more processors to perform any of the steps in Examples 27-33.

[0193] Example 34 includes one or more non-transitory computer-readable media, which include a sequence of instructions that, when executed by one or more processors, cause a computing system to perform any of the steps in Examples 27-33.

[0194] As mentioned above, Infrastructure as a Service (IaaS) is a specific type of cloud computing. IaaS can be configured to provide virtualized computing resources over a public network (e.g., the Internet). In the IaaS model, cloud providers can host infrastructure components (e.g., servers, storage devices, network nodes (e.g., hardware), deployment software, platform virtualization (e.g., hypervisor layer), etc.). In some cases, IaaS providers can also offer various services to accompany these infrastructure components (example services include billing software, monitoring software, logging software, load balancing software, clustering software, etc.). Therefore, because these services may be policy-driven, IaaS users can implement policies to drive load balancing to maintain application availability and performance.

[0195] In some cases, IaaS customers can access resources and services over a wide area network (WAN) such as the Internet and can use the cloud provider's services to install the remaining elements of the application stack. For example, a user can log in to the IaaS platform to create virtual machines (VMs), install an operating system (OS) on each VM, deploy middleware such as databases, create buckets for workloads and backups, and even install enterprise software into that VM. The customer can then use the provider's services to perform various functions, including balancing network traffic, troubleshooting application issues, monitoring performance, and managing disaster recovery.

[0196] In most cases, cloud computing models will require the involvement of cloud providers. Cloud providers can, but are not necessarily, third-party providers specializing in (e.g., provisioning, renting, selling) IaaS services. Entities may also choose to deploy private clouds, thus becoming their own infrastructure service providers.

[0197] In some examples, IaaS deployment is the process of placing a new application or a new version of an application onto a prepared application server, etc. It may also include the processing of server preparation (e.g., installation libraries, daemons, etc.). This is typically managed by the cloud provider, below the hypervisor layer (e.g., servers, storage devices, network hardware, and virtualization). Therefore, the customer can be responsible for processing (OS), middleware, and / or application deployment (e.g., on self-service virtual machines, etc., which can be started on demand).

[0198] In some examples, IaaS provisioning can refer to acquiring computers or virtual hosts for use, or even installing necessary libraries or services on them. In most cases, deployment does not include provisioning, and provisioning may need to be performed first.

[0199] In some cases, IaaS provisioning presents two distinct challenges. First, there's the initial challenge of provisioning the initial infrastructure set before anything is operational. Second, once everything is provisioned, there's the challenge of evolving the existing infrastructure (e.g., adding new services, changing services, removing services, etc.). In some cases, both challenges can be addressed by enabling configuration that declaratively defines the infrastructure. In other words, the infrastructure (e.g., which components are needed and how they interact) can be defined by one or more configuration files. Therefore, the overall topology of the infrastructure (e.g., which resources depend on which resources and how they work together) can be described declaratively. In some cases, once the topology is defined, workflows for creating and / or managing the different components described in the configuration files can be generated.

[0200] In some examples, the infrastructure can have many interconnected elements. For example, there may be one or more Virtual Private Clouds (VPCs) (e.g., potential on-demand pools of configurable and / or shared computing resources), also known as the core network. In some examples, there may also be one or more inbound / outbound traffic group rules, provisioned to define how inbound and / or outbound traffic will be configured for the network, and one or more Virtual Machines (VMs). Other infrastructure elements, such as load balancers, databases, etc., may also be provisioned. The infrastructure can evolve incrementally as more and / or additional infrastructure elements are desired.

[0201] In some cases, continuous deployment techniques can be used to enable the deployment of infrastructure code across various virtual computing environments. Furthermore, the described techniques enable infrastructure management within these environments. In some examples, service teams may write code that they expect to deploy to one or more, but often many, different production environments (e.g., across various geographical locations, sometimes spanning the entire world). However, in some examples, the infrastructure on which the code will be deployed must first be set up. In some cases, provisioning can be done manually, resources can be provisioned using provisioning tools, and / or once the infrastructure is provisioned, the code can be deployed using deployment tools.

[0202] Figure 14This is a block diagram 1400 illustrating an example pattern of an IaaS architecture according to at least one embodiment. A service provider 1402 may be communicatively coupled to a secure host lease 1404, which may include a virtual cloud network (VCN) 1406 and a secure host subnet 1408. In some examples, the service provider 1402 may use one or more client computing devices, which may be portable handheld devices (e.g., iPhone®, cellular phone, iPad®, computing tablet, personal digital assistant (PDA)) or wearable devices (e.g., Google Glass® head-mounted display), running software such as Microsoft Windows Mobile® and / or various mobile operating systems such as iOS, Windows Phone, Android, BlackBerry 8, Palm OS, and enabled for the Internet, email, short message service (SMS), Blackberry®, or other communication protocols. Alternatively, client computing devices may be general-purpose personal computers, including, for example, personal computers and / or laptops running various versions of Microsoft Windows®, Apple Macintosh®, and / or Linux operating systems. The client computing device can be a workstation computer running a variety of commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, such as Google Chrome OS). Alternatively, or additionally, the client computing device can be any other electronic device, such as a thin client computer, an internet-enabled gaming system (e.g., a Microsoft Xbox game console with or without Kinect® gesture input), and / or a personal messaging device capable of communicating over a network that can access VCN 1406 and / or the internet.

[0203] VCN 1406 may include a local peering gateway (LPG) 1410, which may be communicatively coupled to a secure shell (SSH) VCN 1412 via an LPG 1410 included in an SSH VCN 1412. SSH VCN 1412 may include an SSH subnet 1414, and SSH VCN 1412 may be communicatively coupled to a control plane VCN 1416 via an LPG 1410 included in a control plane VCN 1416. Furthermore, SSH VCN 1412 may be communicatively coupled to a data plane VCN 1418 via an LPG 1410. Control plane VCN 1416 and data plane VCN 1418 may be contained within a service lease 1419 that may be owned and / or operated by an IaaS provider.

[0204] The control plane VCN 1416 may include a control plane demilitarized zone (DMZ) layer 1420 that acts as a peripheral network (e.g., a portion of a corporate network between a corporate intranet and an external network). DMZ-based servers can assume limited liability and help control vulnerabilities. Furthermore, the DMZ layer 1420 may include one or more load balancer (LB) subnets 1422, a control plane application layer 1424 that may include one or more application subnets 1426, and a control plane data layer 1428 that may include one or more database (DB) subnets 1430 (e.g., one or more front-end DB subnets and / or one or more back-end DB subnets). One or more LB subnets 1422 contained in the control plane DMZ layer 1420 may be communicatively coupled to one or more application subnets 1426 contained in the control plane application layer 1424 and an Internet gateway 1434 that may be contained in the control plane VCN 1416. The application subnets 1426 may be communicatively coupled to one or more DB subnets 1430 contained in the control plane data layer 1428, as well as a service gateway 1436 and a Network Address Translation (NAT) gateway 1438. The control plane VCN 1416 may include the service gateway 1436 and the NAT gateway 1438.

[0205] The control plane VCN 1416 may include a data plane mirror application layer 1440, which may include one or more application subnets 1426. The one or more application subnets 1426 included in the data plane mirror application layer 1440 may include a virtual network interface controller (VNIC) 1442 capable of executing a compute instance 1444. The compute instance 1444 may communicatively couple the one or more application subnets 1426 of the data plane mirror application layer 1440 to the one or more application subnets 1426 that may be included in the data plane application layer 1446.

[0206] Data plane VCN 1418 may include data plane application layer 1446, data plane DMZ layer 1448, and data plane data layer 1450. Data plane DMZ layer 1448 may include one or more LB subnets 1422 communicatively coupled to one or more application subnets 1426 of data plane application layer 1446 and Internet gateway 1434 of data plane VCN 1418. One or more application subnets 1426 may be communicatively coupled to service gateway 1436 and NAT gateway 1438 of data plane VCN 1418. Data plane data layer 1450 may also include one or more DB subnets 1430 communicatively coupled to one or more application subnets 1426 of data plane application layer 1446.

[0207] Internet gateway 1434 of control plane VCN 1416 and data plane VCN 1418 can be communicatively coupled to metadata management service 1452, and metadata management service 1452 can be communicatively coupled to public internet 1454. Public internet 1454 can be communicatively coupled to NAT gateway 1438 of control plane VCN 1416 and data plane VCN 1418. Service gateway 1436 of control plane VCN 1416 and data plane VCN 1418 can be communicatively coupled to cloud service 1456.

[0208] In some examples, service gateway 1436, either control plane VCN 1416 or data plane VCN 1418, can make application programming interface (API) calls to cloud service 1456 without traversing the public internet 1454. API calls from service gateway 1436 to cloud service 1456 can be unidirectional: service gateway 1436 can make API calls to cloud service 1456, and cloud service 1456 can send requested data to service gateway 1436. However, cloud service 1456 may not initiate API calls to service gateway 1436.

[0209] In some examples, secure host lease 1404 can be directly connected to service lease 1419, which would otherwise be isolated. Secure host subnet 1408 can communicate with SSH subnet 1414 via LPG 1410, which enables bidirectional communication between otherwise isolated systems. Connecting secure host subnet 1408 to SSH subnet 1414 allows secure host subnet 1408 to access other entities within service lease 1419.

[0210] Control plane VCN 1416 may allow users of service lease 1419 to configure or otherwise provision desired resources. Desired resources provisioned in control plane VCN 1416 may be deployed or otherwise used in data plane VCN 1418. In some examples, control plane VCN 1416 may be isolated from data plane VCN 1418, and the data plane mirror application layer 1440 of control plane VCN 1416 may communicate with the data plane application layer 1446 of data plane VCN 1418 via VNIC 1442, which may be included in both the data plane mirror application layer 1440 and the data plane application layer 1446.

[0211] In some examples, users or clients of the system can make requests, such as create, read, update, or delete (CRUD) operations, via the public internet 1454, which can transmit requests to the metadata management service 1452. The metadata management service 1452 can transmit requests to the control plane VCN 1416 via internet gateway 1434. Requests can be received by one or more LB subnets 1422 contained in the control plane DMZ layer 1420. The LB subnets 1422 can determine that the request is valid, and in response to this determination, they can transmit the request to one or more application subnets 1426 contained in the control plane application layer 1424. If the request is validated and requires a call to the public internet 1454, the call to the public internet 1454 can be transmitted to a NAT gateway 1438 that can make calls to the public internet 1454. The request may expect stored metadata to be stored in one or more DB subnets 1430.

[0212] In some examples, the data plane mirroring application layer 1440 can facilitate direct communication between the control plane VCN 1416 and the data plane VCN 1418. For example, it might be desirable to apply configuration changes, updates, or other appropriate modifications to resources contained in the data plane VCN 1418. Through VNIC 1442, the control plane VCN 1416 can communicate directly with the resources contained in the data plane VCN 1418, and thus can perform configuration changes, updates, or other appropriate modifications.

[0213] In some embodiments, the control plane VCN 1416 and data plane VCN 1418 may be contained within a service lease 1419. In this case, the system's users or customers may not own or operate the control plane VCN 1416 or data plane VCN 1418. Alternatively, the IaaS provider may own or operate both the control plane VCN 1416 and data plane VCN 1418, and both planes may be contained within the service lease 1419. This embodiment enables the isolation of networks that might prevent users or customers from interacting with resources from other users or customers. Moreover, this embodiment allows users or customers of the system to privately store databases without relying on the public internet 1454, which may not have the desired level of threat prevention for storage.

[0214] In other embodiments, one or more LB subnets 1422 included in the control plane VCN 1416 may be configured to receive signals from the service gateway 1436. In this embodiment, the control plane VCN 1416 and the data plane VCN 1418 may be configured to be invoked by the IaaS provider's customers without invoking the public internet 1454. The IaaS provider's customers may expect this embodiment because the database(s) used by the customer can be controlled by the IaaS provider and can be stored on service lease 1419, which may be isolated from the public internet 1454.

[0215] Figure 15 This is a block diagram 1400 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1502 (e.g., Figure 14 The service provider 1402 can communicatively couple to the secure host lease 1504 (e.g., Figure 14 Secure hosting lease 1404), the secure hosting lease 1504 may include a virtual cloud network (VCN) 1506 (e.g., Figure 14 VCN 1406) and Secure Host Subnet 1508 (e.g., Figure 14 The secure host subnet 1408). VCN 1506 may include a local peering gateway (LPG) 1510 (e.g., Figure 14 The LPG 1410), which can be communicatively coupled to the Secure Shell (SSH) VCN 1512 (e.g., via the LPG 1410 contained in the SSH VCN 1512) Figure 14 SSH VCN 1412). SSH VCN 1512 can include SSH subnet 1514 (e.g., Figure 14 SSH subnet 1414), and SSH VCN 1512 can be communicatively coupled to control plane VCN 1516 via LPG 1510 included in control plane VCN 1516 (e.g., Figure 14 Control plane VCN 1416). Control plane VCN 1516 may be included in service lease 1519 (e.g., Figure 14 In the service lease 1419), and the data plane VCN 1518 (e.g., Figure 14 The data plane VCN 1418 may be included in a customer lease 1521 that may be owned or operated by the system’s users or customers.

[0216] Control plane VCN 1516 may include control plane DMZ layer 1520 (e.g., Figure 14 The control plane DMZ layer 1420), which may include one or more LB subnets 1522 (e.g., Figure 14 (one or more) LB subnets 1422), may include (one or more) application subnets 1526 (e.g., Figure 14 The control plane application layer 1524 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424 may include one or more database (DB) subnets 1530 (e.g., similar to...). Figure 14 The control plane data layer 1528 of (one or more) DB subnets 1430 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1522 contained in the control plane DMZ layer 1520 can be communicatively coupled to one or more application subnets 1526 contained in the control plane application layer 1524 and an Internet gateway 1534 that can be contained in the control plane VCN 1516 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1526 can communicatively couple to DB subnet(s) 1530 contained in control plane data layer 1528 and service gateway 1536 (e.g., Figure 14 Service gateway 1436) and Network Address Translation (NAT) gateway 1538 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1516 may include the service gateway 1536 and the NAT gateway 1538.

[0217] The control plane VCN 1516 may include a data plane mirror of the application layer 1540, which may include one or more application subnets 1526 (e.g., Figure 14 The data plane mirror application layer 1440). One or more application subnets 1526 contained in the data plane mirror application layer 1540 may include computational instances 1544 (e.g., similar to...). Figure 14 The virtual network interface controller (VNIC) 1542 (e.g., the VNIC of 1442) of the computing instance 1444. The computing instance 1544 may facilitate the mirroring of the application subnet(s) 1526 of the application layer 1540 in the data plane and may be included in the application layer 1546 in the data plane (e.g., Figure 14 Communication between one or more application subnets 1526 in the data plane application layer 1446 via VNIC 1542 contained in the data plane mirror application layer 1540 and VNIC 1542 contained in the data plane application layer 1546.

[0218] The Internet gateway 1534, included in the control plane VCN 1516, can be communicatively coupled to the metadata management service 1552 (e.g., Figure 14 Metadata management service 1452), which can communicatively couple to the public Internet 1554 (e.g., Figure 14 The public internet 1554 can communicatively couple to a NAT gateway 1538 included in the control plane VCN 1516. The service gateway 1536 included in the control plane VCN 1416 can communicatively couple to a cloud service 1556 (e.g., ...). Figure 14 Cloud services 1456).

[0219] In some examples, data plane VCN 1518 may be included in customer lease 1521. In this case, the IaaS provider may provide control plane VCN 1516 for each customer, and the IaaS provider may set up a unique compute instance 1544 for each customer, included in service lease 1519. Each compute instance 1544 may allow communication between control plane VCN 1516 included in service lease 1519 and data plane VCN 1518 included in customer lease 1521. Compute instance 1544 may allow resources provisioned in control plane VCN 1516 included in service lease 1519 to be deployed or otherwise used in data plane VCN 1518 included in customer lease 1521.

[0220] In other examples, an IaaS provider's customer may have a database residing in customer lease 1521. In this example, control plane VCN 1516 may include data plane mirror application layer 1540, which may include one or more application subnets 1526. Data plane mirror application layer 1540 may reside in data plane VCN 1518, but may not reside in data plane VCN 1518. That is, data plane mirror application layer 1540 may have access to customer lease 1521, but may not reside in data plane VCN 1518 or be owned or operated by an IaaS provider's customer. Data plane mirror application layer 1540 may be configured to invoke data plane VCN 1518, but may not be configured to invoke any entity contained in control plane VCN 1516. Customers may expect to deploy or otherwise use resources provisioned in the control plane VCN 1516 in the data plane VCN 1518, and the data plane mirroring application layer 1540 can facilitate the customer's desired deployment or other use of resources.

[0221] In some embodiments, an IaaS provider's customer can apply filters to data plane VCN 1518. In this embodiment, the customer can determine what data plane VCN 1518 can access, and the customer can restrict access from data plane VCN 1518 to the public internet 1554. The IaaS provider may not be able to apply filters or otherwise control data plane VCN 1518's access to any external networks or databases. Applying filters and controls to data plane VCN 1518 contained in customer lease 1521 helps isolate data plane VCN 1518 from other customers and the public internet 1554.

[0222] In some embodiments, cloud service 1556 may be invoked by service gateway 1536 to access services that may not exist on public internet 1554, control plane VCN 1516, or data plane VCN 1518. The connection between cloud service 1556 and control plane VCN 1516 or data plane VCN 1518 may not be real-time or continuous. Cloud service 1556 may reside on different networks owned or operated by an IaaS provider. Cloud service 1556 may be configured to receive calls from service gateway 1536 and may be configured not to receive calls from public internet 1554. Some cloud services 1556 may be isolated from other cloud services 1556, and control plane VCN 1516 may be isolated from cloud services 1556 that may not be in the same region as control plane VCN 1516. For example, control plane VCN 1516 may be located in "Region 1," and cloud service "Deployment 14" may be located in both "Region 1" and "Region 2." If the service gateway 1536, contained in the control plane VCN 1516 located in region 1, makes a call to deployment 14, then that call can be transmitted to deployment 14 in region 1. In this example, the control plane VCN 1516 or deployment 14 in region 1 may be uncoupled from or otherwise communicate with deployment 14 in region 2.

[0223] Figure 16 This is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1602 (e.g., Figure 14 The service provider 1402 can communicatively couple to the secure host lease 1604 (e.g., Figure 14 Secure hosting lease 1404), the secure hosting lease 1604 may include a virtual cloud network (VCN) 1606 (e.g., Figure 14 VCN 1406) and Secure Host Subnet 1608 (e.g., Figure 14 The secure host subnet 1408). VCN 1606 can include LPG 1610 (e.g., Figure 14 The LPG 1410), which can be communicatively coupled to the SSH VCN 1612 via the LPG 1610 included in the SSH VCN 1612 (e.g., Figure 14 SSH VCN 1412). SSH VCN 1612 can include SSH subnet 1614 (e.g., Figure 14 SSH subnet 1414), and SSH VCN 1612 can be communicatively coupled to control plane VCN 1616 via LPG 1610 included in control plane VCN 1616 (e.g., Figure 14 The control plane VCN 1416) and coupled to the data plane VCN 1618 via the LPG 1610 contained in the data plane VCN 1618 (e.g., Figure 14 Data plane 1418). Control plane VCN 1616 and data plane VCN 1618 may be included in service lease 1619 (e.g., Figure 14 (Service rental 1419).

[0224] The control plane VCN 1616 may include one or more load balancer (LB) subnets 1622 (e.g., Figure 14 The control plane DMZ layer 1620 of (one or more) LB subnets 1422) (e.g., Figure 14 The control plane DMZ layer 1426 may include one or more application subnets 1626 (e.g., similar to...). Figure 14 The control plane application layer 1624 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424), and may include (one or more) DB subnets 1630, and the control plane data layer 1628 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1622 contained in the control plane DMZ layer 1620 can be communicatively coupled to one or more application subnets 1626 contained in the control plane application layer 1624 and an Internet gateway 1634 that can be contained in the control plane VCN 1616 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1626 can communicatively couple to DB subnet(s) 1630 contained in control plane data layer 1628 and service gateway 1636 (e.g., Figure 14 The service gateway) and Network Address Translation (NAT) gateway 1638 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1616 may include the service gateway 1636 and the NAT gateway 1638.

[0225] The data plane VCN 1618 may include the data plane application layer 1646 (e.g., Figure 14 Data plane application layer 1446), data plane DMZ layer 1648 (e.g., Figure 14 Data plane DMZ layer 1448), and data plane data layer 1650 (e.g., Figure 14 The data plane data layer 1450. The data plane DMZ layer 1648 may include one or more trusted application subnets 1660 and one or more untrusted application subnets 1662 that can be communicatively coupled to the data plane application layer 1646, and one or more LB subnets 1622 of the Internet gateway 1634 contained in the data plane VCN 1618. The one or more trusted application subnets 1660 may be communicatively coupled to the service gateway 1636 contained in the data plane VCN 1618, the NAT gateway 1638 contained in the data plane VCN 1618, and one or more DB subnets 1630 contained in the data plane data layer 1650. The one or more untrusted application subnets 1662 may be communicatively coupled to the service gateway 1636 contained in the data plane VCN 1618 and the one or more DB subnets 1630 contained in the data plane data layer 1650. The data plane data layer 1650 may include one or more DB subnets 1630 that can be communicatively coupled to the service gateway 1636 contained in the data plane VCN 1618.

[0226] One or more untrusted application subnets 1662 may include one or more primary VNICs 1664(1)-(N) that can be communicatively coupled to tenant virtual machines (VMs) 1666(1)-(N). Each tenant VM 1666(1)-(N) may be communicatively coupled to a corresponding application subnet 1667(1)-(N) that may be contained in a corresponding container egress VCN 1668(1)-(N), which may be contained in a corresponding customer lease 1670(1)-(N). A corresponding secondary VNIC 1672(1)-(N) may facilitate communication between one or more untrusted application subnets 1662 contained in data plane VCN 1618 and application subnets contained in container egress VCN 1668(1)-(N). Each container exit VCN 1668(1)-(N) may include a NAT gateway 1638, which can communicatively couple to the public Internet 1654 (e.g., Figure 14 The public internet (1454).

[0227] Internet gateway 1634, contained in control plane VCN 1616 and data plane VCN 1618, can be communicatively coupled to metadata management service 1652 (e.g., Figure 14 A metadata management system 1452, which is communicatively coupled to the public internet 1654, is also communicatively coupled to a NAT gateway 1638 contained in a control plane VCN 1616 and a data plane VCN 1618. A service gateway 1636 contained in both the control plane VCN 1616 and the data plane VCN 1618 is communicatively coupled to a cloud service 1656.

[0228] In some embodiments, the data plane VCN 1618 can be integrated with the customer lease 1670. Such integration may be useful or desired by the IaaS provider's customer in certain situations, such as when support may be expected during code execution. A customer may provide code that could be destructive, might communicate with other customer resources, or might otherwise cause undesirable effects. In response, the IaaS provider can determine whether to run the code provided by the customer to the IaaS provider.

[0229] In some examples, an IaaS provider's customer may grant the IaaS provider temporary network access and request functionality attached to the data plane application layer 1646. The code running this functionality may execute in VMs 1666(1)-(N) and may not be configured to run anywhere else on the data plane VCN 1618. Each VM 1666(1)-(N) may be connected to a customer lease 1670. The corresponding container 1671(1)-(N) contained in VMs 1666(1)-(N) may be configured to run the code. In this case, dual isolation may exist (e.g., container 1671(1)-(N) runs the code, where container 1671(1)-(N) may be contained in at least one or more untrusted application subnets 1662 containing VMs 1666(1)-(N)), which can help prevent incorrect or otherwise unintended code from corrupting the IaaS provider's network or the networks of different customers. Containers 1671(1)-(N) may be communicatively coupled to customer lease 1670 and may be configured to transmit or receive data from customer lease 1670. Containers 1671(1)-(N) may not be configured to transmit or receive data from any other entity in the data plane VCN 1618. After the code execution is complete, the IaaS provider may terminate or otherwise dispose of containers 1671(1)-(N).

[0230] In some embodiments, one or more trusted application subnets 1660 may run code that can be owned or operated by an IaaS provider. In this embodiment, one or more trusted application subnets 1660 may be communicatively coupled to one or more database subnets 1630 and configured to perform CRUD operations in one or more database subnets 1630. One or more untrusted application subnets 1662 may be communicatively coupled to one or more database subnets 1630, but in this embodiment, one or more untrusted application subnets may be configured to perform read operations in one or more database subnets 1630. Containers 1671(1)-(N) that may be contained in each customer's VM 1666(1)-(N) and may run code from the customer may not be communicatively coupled to one or more database subnets 1630.

[0231] In other embodiments, the control plane VCN 1616 and the data plane VCN 1618 may be coupled without direct communication. In this embodiment, there may be no direct communication between the control plane VCN 1616 and the data plane VCN 1618. However, communication can occur indirectly through at least one method. The LPG 1610 may be established by an IaaS provider, which can facilitate communication between the control plane VCN 1616 and the data plane VCN 1618. In another example, either the control plane VCN 1616 or the data plane VCN 1618 may invoke the cloud service 1656 via the service gateway 1636. For example, an invocation of the cloud service 1656 from the control plane VCN 1616 may include a request for a service that can communicate with the data plane VCN 1618.

[0232] Figure 17 This is a block diagram 1600 illustrating another example pattern of an IaaS architecture according to at least one embodiment. Service operator 1702 (e.g., Figure 14 The service provider 1402 can communicatively couple to the secure host lease 1704 (e.g., Figure 14 Secure hosting lease 1404), the secure hosting lease 1704 may include a virtual cloud network (VCN) 1706 (e.g., Figure 14 VCN 1406) and Secure Host Subnet 1708 (e.g., Figure 14 The secure host subnet 1408). VCN 1706 can include LPG 1710 (e.g., Figure 14 The LPG 1410), the LPG 1710 can be accessed via SSH VCN 1712 (e.g., LPG 1410), Figure 14The LPG 1710 in SSH VCN 1412 is communicatively coupled to SSH VCN 1712. SSH VCN 1712 may include SSH subnet 1714 (e.g., Figure 14 SSH subnet 1414), and SSH VCN 1712 can be communicatively coupled to control plane VCN 1716 via LPG 1710 contained in control plane VCN 1716 (e.g., Figure 14 The control plane VCN 1416) and coupled to the data plane VCN 1718 via the LPG 1710 contained in the data plane VCN 1718 (e.g., Figure 14 Data plane 1418). Control plane VCN 1716 and data plane VCN 1718 may be included in service lease 1719 (e.g., Figure 14 (Service rental 1419).

[0233] The control plane VCN 1716 may include one or more LB subnets 1722 (e.g., Figure 14 The control plane DMZ layer 1720 of (one or more) LB subnets 1422) (e.g., Figure 14 The control plane DMZ layer 1420 may include one or more application subnets 1726 (e.g., Figure 14 The control plane application layer 1724 of (one or more) application subnets 1426 (e.g., Figure 14 The control plane application layer 1424 may include (one or more) DB subnets 1730 (e.g., Figure 16 The control plane data layer 1728 of (one or more) DB subnets 1630 (e.g., Figure 14 The control plane data layer 1428). One or more LB subnets 1722 contained in the control plane DMZ layer 1720 can be communicatively coupled to one or more application subnets 1726 contained in the control plane application layer 1712 and an Internet gateway 1734 that can be contained in the control plane VCN 1716 (e.g., Figure 14 Internet gateway 1434), and application subnet(s) 1726 can communicatively couple to DB subnet(s) 1730 contained in control plane data layer 1728 and service gateway 1736 (e.g., Figure 14 The service gateway) and Network Address Translation (NAT) gateway 1738 (e.g., Figure 14 (NAT gateway 1438). The control plane VCN 1716 may include the service gateway 1736 and the NAT gateway 1738.

[0234] Data plane VCN 1718 may include data plane application layer 1746 (e.g., Figure 14 Data plane application layer 1446), data plane DMZ layer 1748 (e.g., Figure 14 Data plane DMZ layer 1448), and data plane data layer 1750 (e.g., Figure 14 The data plane data layer 1450). The data plane DMZ layer 1748 may include one or more trusted application subnets 1760 that can be communicatively coupled to the data plane application layer 1746 (e.g., Figure 16 (one or more) trusted application subnets 1660) and (one or more) untrusted application subnets 1762 (e.g., Figure 16 The data plane VCN 1718 may include one or more untrusted application subnets 1762 and one or more LB subnets 1722 of Internet gateway 1734. One or more trusted application subnets 1760 may communicatively couple to service gateway 1736, NAT gateway 1738, and DB subnets 1730 in data plane VCN 1750. One or more untrusted application subnets 1762 may communicatively couple to service gateway 1736 and DB subnets 1730 in data plane VCN 1718. Data plane VCN 1750 may include one or more DB subnets 1730 that may communicatively couple to service gateway 1736 in data plane VCN 1718.

[0235] One or more untrusted application subnets 1762 may include a primary VNIC 1764(1)-(N) communicatively coupled to tenant virtual machines (VMs) 1766(1)-(N) residing within one or more untrusted application subnets 1762. Each tenant VM 1766(1)-(N) may run code in a corresponding container 1767(1)-(N) and is communicatively coupled to an application subnet 1726 that may be contained in a data plane application layer 1746 that may be contained in a container egress VCN 1768. A corresponding secondary VNIC 1772(1)-(N) may facilitate communication between one or more untrusted application subnets 1762 contained in a data plane VCN 1718 and the application subnet contained in a container egress VCN 1768. The container egress VCN may include a public internet 1754 (e.g., Figure 14 The public internet (1454) uses NAT gateway 1738.

[0236] Internet gateway 1734, contained in control plane VCN 1716 and data plane VCN 1718, can be communicatively coupled to metadata management service 1752 (e.g., Figure 14 The metadata management system 1452, which is communicatively coupled to the public internet 1754, is also communicatively coupled to a NAT gateway 1738 contained in a control plane VCN 1716 and a data plane VCN 1718. A service gateway 1736 contained in both the control plane VCN 1716 and the data plane VCN 1718 is communicatively coupled to a cloud service 1756.

[0237] In some examples, Figure 17 The architecture shown in the block diagram 1600 can be considered as... Figure 16 This is an exception to the pattern shown in the architecture of block diagram 1500, and this pattern may be what the IaaS provider's customers would expect if the IaaS provider cannot communicate directly with the customer (e.g., in a disconnected region). The customer can access in real time the corresponding container 1767(1)-(N) contained in each customer's VM 1766(1)-(N). Container 1767(1)-(N) can be configured to invoke a corresponding auxiliary VNIC 1772(1)-(N) contained in one or more application subnets 1726 of the data plane application layer 1746, which may be contained in the container egress VCN 1768. The auxiliary VNIC 1772(1)-(N) can transmit the call to a NAT gateway 1738, which can then transmit the call to the public internet 1754. In this example, containers 1767(1)-(N), which can be accessed by clients in real time, can be isolated from the control plane VCN 1716 and from other entities contained in the data plane VCN 1718. Containers 1767(1)-(N) can also be isolated from resources from other clients.

[0238] In other examples, a client can use container 1767(1)-(N) to invoke cloud service 1756. In this example, the client can run code within container 1767(1)-(N) requesting a service from cloud service 1756. Container 1767(1)-(N) can then forward the request to auxiliary VNIC 1772(1)-(N), which can then forward the request to a NAT gateway, which can forward the request to the public internet 1754. The public internet 1754 can then forward the request via internet gateway 1734 to one or more LB subnets 1722 contained in control plane VCN 1716. In response to determining that the request is valid, one or more LB subnets can then forward the request to one or more application subnets 1726, which can then forward the request to cloud service 1756 via service gateway 1736.

[0239] It should be recognized that the IaaS architectures 1400, 1500, 1600, and 1700 depicted in the figures may have other components besides those depicted. Furthermore, the embodiments shown in the figures are merely some examples of cloud infrastructure systems that can be incorporated into embodiments of this disclosure. In some other embodiments, the IaaS system may have more or fewer components than shown in the figures, may combine two or more components, or may have different configurations or component arrangements.

[0240] In some embodiments, the IaaS system described herein may include application suites, middleware, and database service offerings delivered to customers in a self-service, subscription-based, elastically scalable, reliable, highly available, and secure manner. An example of such an IaaS system is the Oracle Cloud Infrastructure (OCI) provided by this assignee.

[0241] Figure 18 An example computer system 1800 in which various embodiments can be implemented is illustrated. System 1800 can be used to implement any of the computer systems described above. As shown, computer system 1800 includes a processing unit 1804 that communicates with a plurality of peripheral subsystems via a bus subsystem 1802. These peripheral subsystems may include a processing acceleration unit 1806, an I / O subsystem 1808, a storage subsystem 1818, and a communication subsystem 1824. Storage subsystem 1818 includes a tangible computer-readable storage medium 1822 and system memory 1810.

[0242] Bus subsystem 1802 provides a mechanism for allowing various components and subsystems of computer system 1800 to communicate with each other as intended. While bus subsystem 1802 is schematically shown as a single bus, alternative embodiments of the bus subsystem may utilize multiple buses. Bus subsystem 1802 can be any of several types of bus architectures, including memory buses or memory controllers, peripheral buses, and local buses using any of the various bus architectures available. For example, such architectures may include Industry Standard Architecture (ISA) buses, Microchannel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses, which may be implemented as Mezzanine buses manufactured according to the IEEE P1386.1 standard.

[0243] A processing unit 1804, which may be implemented as one or more integrated circuits (e.g., a conventional microprocessor or microcontroller), controls the operation of the computer system 1800. One or more processors may be included in the processing unit 1804. These processors may include single-core or multi-core processors. In some embodiments, the processing unit 1804 may be implemented as one or more independent processing units 1832 and / or 1834, wherein each processing unit includes a single-core or multi-core processor. In other embodiments, the processing unit 1804 may also be implemented as a quad-core processing unit formed by integrating two dual-core processors into a single chip.

[0244] In various embodiments, processing unit 1804 can execute various programs in response to program code and can maintain multiple concurrently executing programs or processes. At any given time, some or all of the program code to be executed can reside in processor(s) 1804 and / or storage subsystem 1818. With appropriate programming, processor(s) 1804 can provide the various functions described above. Computer system 1800 may additionally include processing acceleration unit 1806, which may include digital signal processor (DSP), dedicated processor, etc.

[0245] I / O subsystem 1808 may include user interface input devices and user interface output devices. User interface input devices may include keyboards, pointing devices such as mice or trackballs, touchpads or touchscreens integrated into a display, scroll wheels, click wheels, dials, buttons, switches, keyboards, audio input devices with voice command recognition systems, microphones, and other types of input devices. User interface input devices may include, for example, motion sensing and / or gesture recognition devices, such as the Microsoft Kinect® motion sensor, which enables users to control and interact with input devices such as the Microsoft Xbox® 360 game controller via a natural user interface using gestures and voice commands. User interface input devices may also include eye posture recognition devices, such as the Google Glass® blink detector, which detects eye activity from the user (e.g., “blinking” when taking a photo and / or making menu selections) and translates the eye posture into input in an input device (e.g., Google Glass®). Furthermore, user interface input devices may include voice recognition sensing devices that enable users to interact with a voice recognition system (e.g., the Siri® navigator) via voice commands.

[0246] User interface input devices may also include, but are not limited to, 3D mice, joysticks or pointing sticks, game panels and drawing tablets, as well as audio / video devices such as speakers, digital cameras, digital camcorders, portable media players, webcams, image scanners, fingerprint scanners, barcode readers, 3D scanners, 3D printers, laser rangefinders, and eye-tracking devices. Furthermore, user interface input devices may include, for example, medical imaging input devices such as computed tomography (CT), magnetic resonance imaging (MRI), positron emission tomography (PET), and medical ultrasound equipment. User interface input devices may also include, for example, audio input devices such as MIDI keyboards, digital musical instruments, etc.

[0247] User interface output devices may include display subsystems, indicator lights, or non-visual displays such as audio output devices, etc. Display subsystems may be cathode ray tubes (CRTs), flat panel devices such as those using liquid crystal displays (LCDs) or plasma displays, projection devices, touchscreens, etc. Generally, the term "output device" is intended to include all possible types of devices and mechanisms for outputting information from computer system 1800 to the user or other computers. For example, user interface output devices may include, but are not limited to, various display devices that visually convey text, graphics, and audio / video information, such as monitors, printers, speakers, headphones, car navigation systems, plotters, voice output devices, and modems.

[0248] Computer system 1800 may include a storage subsystem 1818 that provides a tangible, non-transitory, computer-readable storage medium for storing software and data constructs that provide the functionality of the embodiments described in this disclosure. The software may include programs, code modules, instructions, scripts, etc., which provide the aforementioned functionality when executed by one or more cores or processors of processing unit 1804. Storage subsystem 1818 may also provide a repository for storing data used according to this disclosure.

[0249] like Figure 18 As depicted in the example, the storage subsystem 1818 may include various components, including system memory 1810, computer-readable storage medium 1822, and computer-readable storage medium reader 1820. System memory 1810 may store program instructions that can be loaded and executed by processing unit 1804. System memory 1810 may also store data used during instruction execution and / or data generated during program instruction execution. Various types of programs may be loaded into system memory 1810, including but not limited to client applications, web browsers, middleware applications, relational database management systems (RDBMS), virtual machines, containers, etc.

[0250] System memory 1810 may also store operating system 1816. Examples of operating system 1816 may include various versions of Microsoft Windows®, Apple Macintosh® and / or Linux operating systems, various commercial UNIX® or UNIX-like operating systems (including, but not limited to, various GNU / Linux operating systems, Google Chrome® OS, etc.) and / or mobile operating systems (such as iOS, Windows® Phone, Android® OS, BlackBerry® OS, and Palm® OS). In some implementations where computer system 1800 executes one or more virtual machines, the virtual machine, along with its guest operating system (GOS), may be loaded into system memory 1810 and executed by one or more processors or cores of processing unit 1804.

[0251] System memory 1810 may be configured differently depending on the type of computer system 1800. For example, system memory 1810 may be volatile memory (such as random access memory (RAM)) and / or non-volatile memory (such as read-only memory (ROM), flash memory, etc.). Different types of RAM configurations may be provided, including static random access memory (SRAM), dynamic random access memory (DRAM), etc. In some embodiments, system memory 1810 may include a basic input / output system (BIOS) containing basic routines that facilitate the transfer of information between components within the computer system 1800, such as during startup.

[0252] Computer-readable storage medium 1822 may represent remote, local, fixed and / or removable storage devices and storage media for temporarily and / or more permanently containing and storing computer-readable information (including instructions executable by the processing unit 1804 of the computer system 1800) for use by the computer system 1800.

[0253] Computer-readable storage medium 1822 may include any suitable medium known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing and / or transmitting information. This may include tangible computer-readable storage media such as RAM, ROM, electrically erasable programmable ROM (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices, or other tangible computer-readable media.

[0254] For example, computer-readable storage medium 1822 may include hard disk drives that read from or write to non-removable non-volatile magnetic media, disk drives that read from or write to removable non-volatile magnetic disks, and optical disc drives that read from or write to removable non-volatile optical discs (such as CD ROMs, DVDs, and Blu-ray® discs or other optical media). Computer-readable storage medium 1822 may include, but is not limited to, Zip® drives, flash memory cards, Universal Serial Bus (USB) flash drives, Secure Digital (SD) cards, DVD discs, digital audio tapes, and so on. Computer-readable storage medium 1822 may also include solid-state drives (SSDs) based on non-volatile memory (such as flash memory-based SSDs, enterprise flash drives, solid-state ROMs, etc.), volatile memory-based SSDs (such as solid-state RAM, dynamic RAM, static RAM), DRAM-based SSDs, magnetoresistive RAM (MRAM) SSDs, and hybrid SSDs using a combination of DRAM-based and flash memory-based SSDs. Disk drives and their associated computer-readable media can provide non-volatile storage for computer-readable instructions, data structures, program modules and other data for computer system 1800.

[0255] Machine-readable instructions executable by one or more processors or cores of the processing unit 1804 may be stored on a non-transitory computer-readable storage medium. The non-transitory computer-readable storage medium may include physically tangible memory or storage devices, including volatile memory storage devices and / or non-volatile memory devices. Examples of non-transitory computer-readable storage media include magnetic storage media (e.g., disks or tapes), optical storage media (e.g., DVDs, CDs), various types of RAM, ROM, or flash memory, hard disk drives, floppy disk drives, removable memory drives (e.g., USB drives), or other types of storage devices.

[0256] The communication subsystem 1824 provides an interface to other computer systems and networks. The communication subsystem 1824 serves as an interface for receiving data from other systems and sending data from computer system 1800 to other systems. For example, the communication subsystem 1824 enables computer system 1800 to connect to one or more devices via the Internet. In some embodiments, the communication subsystem 1824 may include radio frequency (RF) transceiver components (e.g., advanced data network technologies using cellular telephone technology, such as 3G, 4G, or EDGE (Enhanced Data Rates for Global Evolution), WiFi (IEEE 802.11 series standards), or other mobile communication technologies, or any combination thereof), a global positioning system (GPS) receiver component, and / or other components for accessing wireless voice and / or data networks. In some embodiments, as an addition to or alternative to the wireless interface, the communication subsystem 1824 may provide a wired network connection (e.g., Ethernet).

[0257] In some embodiments, the communication subsystem 1824 may also represent one or more users who can use the computer system 1800 to receive input communications in the form of structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, etc.

[0258] For example, the communication subsystem 1824 can be configured to receive data feeds 1826 in real time from users of social networks and / or other communication services, such as Twitter® feeds, Facebook® updates, web feeds such as Rich Site Summary (RSS) feeds, and / or real-time updates from one or more third-party information sources.

[0259] Furthermore, the communication subsystem 1824 can also be configured to receive data in the form of a continuous data stream, which may include an event stream 1828 and / or event updates 1830 that are essentially continuous or unbounded real-time events without a clearly defined termination. Examples of applications that generate continuous data may include, for example, sensor data applications, financial quotation machines, network performance measurement tools (e.g., network monitoring and traffic management applications), clickstream analysis tools, vehicle traffic monitoring, and so on.

[0260] The communication subsystem 1824 can also be configured to output structured and / or unstructured data feeds 1826, event streams 1828, event updates 1830, etc. to one or more databases, which can communicate with one or more streaming data source computers coupled to the computer system 1800.

[0261] The computer system 1800 can be one of a variety of types, including handheld portable devices (e.g., iPhone® cellular phones, iPad® computing tablets, PDAs), wearable devices (e.g., Google® Glass head-mounted displays), PCs, workstations, mainframes, information stations, server racks, or any other data processing system.

[0262] Due to the ever-evolving nature of computers and networks, the description of the computer system 1800 depicted in the figures is merely a concrete example. Many other configurations with more or fewer components than the system depicted in the figures are possible. For example, custom hardware may be used and / or specific elements may be implemented using hardware, firmware, software (including applets), or a combination thereof. Additionally, connections to other computing devices, such as network input / output devices, may also be employed. Based on the disclosure and teachings provided herein, those skilled in the art will recognize other ways and / or methods for implementing the various embodiments.

[0263] While specific embodiments have been described, various modifications, alterations, alternative constructions, and equivalents are also included within the scope of this disclosure. The embodiments are not limited to operation within certain specific data processing environments, but can be freely operated within multiple data processing environments. Furthermore, although the embodiments have been described using a specific series of transactions and steps, those skilled in the art will understand that the scope of this disclosure is not limited to the described series of transactions and steps. Various features and aspects of the above embodiments can be used individually or in combination.

[0264] Furthermore, while embodiments have been described using specific combinations of hardware and software, it should be recognized that other combinations of hardware and software are also within the scope of this disclosure. Embodiments may be implemented using only hardware, or only software, or a combination thereof. The various processes described herein can be implemented in any combination on the same processor or on different processors. Accordingly, where a component or service is described as being configured to perform certain operations, such configuration can be accomplished, for example, by designing electronic circuits to perform operations, by programming programmable electronic circuits (such as microprocessors), or any combination thereof. Processes may communicate using a variety of technologies, including but not limited to conventional technologies for inter-process communication, and different pairs of processes may use different technologies, or the same pair of processes may use different technologies at different times.

[0265] Accordingly, the specification and drawings are intended to be illustrative rather than restrictive. However, it will be apparent that additions, omissions, deletions, and other modifications and alterations may be made thereto without departing from the broader spirit and scope set forth in the claims. Thus, while specific disclosed embodiments have been described, they are not intended to be limiting. Various modifications and equivalents are within the scope of the following claims.

[0266] In the context of describing the disclosed embodiments (particularly in the context of the following claims), the terms “a,” “an,” and “the,” and similar designations, are to be interpreted as encompassing both singular and plural, unless otherwise indicated herein or clearly contradicted by the context. Unless otherwise stated, the terms “comprising,” “having,” “including,” and “containing” are to be interpreted as open-ended terms (i.e., meaning “including but not limited to”). The term “connected” should be interpreted as partially or wholly contained in, attached to, or joined together, even if something exists in between. Unless otherwise indicated herein, the enumeration of value ranges herein is intended only as a shorthand method for individually referencing each individual value falling within that range, and each individual value is incorporated into the specification as if it were individually enumerated herein. Unless otherwise indicated herein or clearly contradicted by the context, all methods described herein can be performed in any suitable order. The use of any and all examples or exemplary language (e.g., “such as”) provided herein is intended only to better illustrate the embodiments and does not constitute a limitation on the scope of this disclosure, unless otherwise stated. Nothing in the specification should be construed as indicating that any unclaimed element is essential to the practice of this disclosure.

[0267] Disjunctive language, such as the phrase “at least one of X, Y, or Z”, unless otherwise explicitly stated, is intended to be understood in the context generally used to represent items, terms, etc., and may be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Therefore, such disjunctive language is generally not intended to, and should not, imply that some embodiments require the presence of at least one of X, at least one of Y, or at least one of Z, each individually.

[0268] This document describes preferred embodiments of the present disclosure, including known best modes for carrying out the present disclosure. Variations of these preferred embodiments will become apparent to those skilled in the art upon reading the foregoing description. Those skilled in the art should be able to appropriately employ such variations and may practice the present disclosure in ways other than those specifically described herein. Accordingly, the present disclosure includes all modifications and equivalents to the subject matter recited in the appended claims, where permitted by applicable law. Moreover, unless otherwise indicated herein, the present disclosure includes any combination of the foregoing elements in all its possible variations.

[0269] All references cited in this article, including publications, patent applications and patents, are incorporated into this article by reference to the same extent as if each reference individually and specifically indicated to be incorporated by reference and elaborated in full in this article.

[0270] In the foregoing specification, various aspects of this disclosure have been described with reference to specific embodiments thereof; however, those skilled in the art will recognize that this disclosure is not limited thereto. The various features and aspects of the foregoing disclosure may be used individually or in combination. Furthermore, embodiments may be used in any number of settings and applications other than those described herein without departing from the broader spirit and scope of this specification. Accordingly, this specification and the accompanying drawings should be considered illustrative rather than restrictive.

Claims

1. A method comprising: identifying, by a computing system, a custom class; selecting, by the computing system, a pre-trained base model trained to predict a base class, the selection based at least in part on the base class belonging to a same domain as the custom class; generating, by the computing system, an untrained inter-class extender model to be trained to predict the custom class, the untrained inter-class extender model comprising an untrained fusion layer and an untrained inter-class extender; connecting, by the computing system, the pre-trained base model in parallel with the untrained inter-class extender model to generate an inter-class augmented model; inputting, by the computing system, a training image into the inter-class augmented model, the training image comprising the custom class; causing, by the computing system, a first block of the pre-trained base model to output a first plurality of activation values associated with the base class based at least in part on the training image; causing, by the computing system, the untrained fusion layer to generate a plurality of weights for weighting the first plurality of activation values, the plurality of weights configured to cause the untrained inter-class extender model to identify the custom class from the training image; causing, by the computing system, the untrained fusion layer to generate a weighted sum based at least in part on weighting the first plurality of activation values using the plurality of weights; causing, by the computing system, the untrained inter-class extender to generate a second plurality of activation values based at least in part on the weighted sum; causing, by the computing system, the untrained inter-class extender model to predict the custom class from the training image based at least in part on the second plurality of activation values; and causing, by the computing system, the pre-trained base model to predict the base class from the training image based at least in part on the first plurality of activation values.

2. The method of claim 1, wherein the method further comprises: generating a plurality of vectors based at least in part on the training image; generating a plurality of activation scores based at least in part on the vectors; and generating a probability distribution based at least in part on the plurality of activation scores, wherein the probability distribution comprises the plurality of weights.

3. The method of any one of claims 1 or 2, wherein the method further comprises: determining an accuracy of the untrained inter-class extender model based at least in part on a loss function; and adjusting a plurality of untrained inter-class extender model weights based at least in part on the accuracy, wherein a plurality of pre-trained base model weights remain unadjusted.

4. The method of any one of claims 1-3, wherein the method further comprises: determining an accuracy of the untrained inter-class extender model based at least in part on a loss function; adjusting a plurality of pre-trained base model weights based at least in part on the accuracy; and adjusting a plurality of untrained inter-class extender model weights based at least in part on the accuracy.

5. The method of any one of claims 1-4, wherein the method further comprises: accessing a data store storing pre-trained base models; identifying a pre-trained inter-class extender model based at least in part on metadata stored in the data store and associated with the pre-trained inter-class extender model, wherein the pre-trained inter-class extender model is trained to identify a class belonging to a same domain as the custom class; and ​ ​ ​ ​ ​ connecting the pre-trained inter-class expander model to the pre-trained base model and the untrained inter-class expander model.

6. The method of claim 5, wherein the method further comprises: transmitting, to the data store, a first control instruction to store the untrained inter-class expander model in response to the untrained inter-class expander model identifying the custom class from the training image; and transmitting, to the data store, a second control instruction to update metadata associated with the pre-trained base model to indicate generation of the inter-class augmentation model.

7. The method of any one of claims 1-6, wherein the method further comprises: determining, based at least in part on the custom class, whether to generate the inter-class augmentation model.

8. A computing system comprising: one or more processors; and one or more non-transitory computer-readable media comprising sequences of instructions that, when executed on the one or more processors, cause the one or more processors to: identify a custom class; select a pre-trained base model trained to predict base classes, the selection based at least in part on a base class belonging to a same domain as the custom class; generate an untrained intra-class expander model to be trained to identify the custom class, the untrained intra-class expander model comprising an untrained fusion layer, a first untrained intra-class expander layer, and a second untrained expander layer; connect the untrained intra-class expander model to the pre-trained base model to generate an intra-class augmentation model; input a training image to the intra-class augmentation model, the training image comprising the custom class; cause, based at least in part on the training image, a first block of the pre-trained base model to output a first plurality of activation values; cause, based at least in part on the training image, the first untrained intra-class expander layer to output a second plurality of activation values; cause the untrained fusion layer to generate a plurality of weights for weighting the first plurality of activation values and the second plurality of activation values, the plurality of weights configured to cause the intra-class augmentation model to identify the custom class from the training image; cause the untrained fusion layer to generate a weighted sum based at least in part on weighting the first plurality of activation values and the second plurality of activation values using the plurality of weights; cause a second block of the pre-trained base model to output a third plurality of activation values based at least in part on the weighted sum; cause the second untrained expander layer to output a fourth plurality of activation values based at least in part on the weighted sum; cause the intra-class augmentation model to predict the custom class from the training image based at least in part on the third plurality of activation values and the fourth plurality of activation values.

9. The computing system of claim 8, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: generate a plurality of vectors based at least in part on the training image; generate a plurality of activation scores based at least in part on the vectors; and generate a probability distribution based at least in part on the plurality of activation scores, wherein the probability distribution comprises the plurality of weights. ​ 10. The computing system of claim 9, wherein the sequence of instructions, which when executed on the one or more processors, further cause the one or more processors to: determine an accuracy of the untrained intra-class expander model based at least in part on a loss function; and adjust a plurality of untrained intra-class expander model weights based at least in part on the accuracy, wherein a plurality of pre-trained base model weights remain unadjusted.

11. The computing system of claim 9, wherein the sequence of instructions, which when executed on the one or more processors, further cause the one or more processors to: determine an accuracy of the untrained intra-class expander model based at least in part on a loss function; adjust a plurality of pre-trained base model weights based at least in part on the accuracy; and adjust a plurality of untrained intra-class expander model weights based at least in part on the accuracy.

12. The computing system of any one of claims 8-11, wherein the sequence of instructions, which when executed on the one or more processors, further cause the one or more processors to: access a data store storing pre-trained base models; identify a pre-trained intra-class expander model based at least in part on metadata stored in the data store and associated with the pre-trained intra-class expander model, wherein the pre-trained intra-class expander model is trained to identify classes belonging to a same domain as the custom class; and connect the pre-trained intra-class expander model to the pre-trained base model and the untrained intra-class expander model.

13. The computing system of claim 12, wherein the sequence of instructions, which when executed on the one or more processors, further cause the one or more processors to: in response to the untrained intra-class expander model identifying the custom class from a training image, transmit a first control instruction to the data store to store the untrained intra-class expander model; and transmit a second control instruction to the data store to update metadata associated with the pre-trained base model to indicate generation of the intra-class enhancement model.

14. The computing system of any one of claims 8-13, wherein the sequence of instructions, which when executed on the one or more processors, further cause the one or more processors to: determine whether to generate an intra-class enhancement model based at least in part on the custom class.

15. One or more non-transitory computer-readable media comprising a sequence of instructions, which when executed by one or more processors, cause a computing system to: identify a custom class; select a pre-trained base model trained to predict base classes, the selection based at least in part on base classes belonging to a same domain as the custom class; generate an untrained inter-class expander model to be trained to predict the custom class, the untrained inter-class expander model comprising an untrained fusion layer and an untrained inter-class expander; connect the pre-trained base model in parallel with the untrained inter-class expander model to generate an inter-class enhancement model; input a training image into the inter-class enhancement model, the training image comprising the custom class; based at least in part on the training image, cause the first block of the pre-trained base model to output a first plurality of activation values associated with the base classes; cause the untrained fusion layer to generate a plurality of weights for weighting the first plurality of activation values, the plurality of weights configured to cause the untrained inter-class expander model to identify the custom class from the training image; cause the untrained fusion layer to generate a weighted sum based at least in part on weighting the first plurality of activation values using the plurality of weights; cause the untrained inter-class expander to generate a second plurality of activation values based at least in part on the weighted sum; cause the untrained inter-class expander model to predict the custom class from the training image based at least in part on the second plurality of activation values; and cause the pre-trained base model to predict the base classes from the training image based at least in part on the first plurality of activation values.

16. The one or more non-transitory computer-readable media of claim 15, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: generate a plurality of vectors based at least in part on the training image; generate a plurality of activation scores based at least in part on the vectors; and generate a probability distribution based at least in part on the plurality of activation scores, wherein the probability distribution comprises the plurality of weights.

17. The one or more non-transitory computer-readable media of claim 15, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: determine an accuracy of the untrained inter-class expander model based at least in part on a loss function; and adjust the plurality of weights of the untrained inter-class expander model based at least in part on the accuracy, wherein the weights of the pre-trained base model remain unadjusted.

18. The one or more non-transitory computer-readable media of claim 15, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: determine an accuracy of the untrained inter-class expander model based at least in part on a loss function; adjust a plurality of pre-trained base model weights based at least in part on the accuracy; and adjust a plurality of untrained inter-class expander model weights based at least in part on the accuracy.

19. The one or more non-transitory computer-readable media of any of claims 15-18, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: access a data store storing pre-trained base models; identify a pre-trained inter-class expander model based at least in part on metadata stored in the data store and associated with the pre-trained inter-class expander model, wherein the pre-trained inter-class expander model is trained to identify classes belonging to a same domain as the custom class; and connect the pre-trained inter-class expander model to the pre-trained base model and the untrained inter-class expander model. ​ 20. The one or more non-transitory computer-readable media of claim 19, wherein the sequences of instructions, when executed on the one or more processors, further cause the one or more processors to: in response to the untrained inter-class expander model identifying the custom class from the training image, transmit a first control instruction to the data store to store the untrained inter-class expander model; and transmit a second control instruction to the data store to update metadata associated with the pre-trained base model to indicate generation of the inter-class enhancement model.

21. A method comprising: receiving, by a data store, a message requesting a model trained to predict a custom class; selecting, by the data store from a plurality of pre-trained base models, a pre-trained base model trained to predict a base class belonging to a same domain as the custom class; selecting, by the data store from a plurality of pre-trained expander models, a first pre-trained expander model associated with the pre-trained base model based at least in part on selecting the pre-trained base model, the first pre-trained expander model trained to predict the custom class; connecting, by the data store, the pre-trained base model to the first pre-trained expander model to generate an enhancement model based at least in part on the first pre-trained expander model being trained to predict the custom class, the pre-trained base model connected to the first pre-trained expander model such that a first output of a first block of the pre-trained base model is received by a first pre-trained fusion layer of the pre-trained expander model; and transmitting, by the data store, the enhancement model to a sender of the message.

22. The method of claim 21, wherein the method further comprises: receiving a message indicating that the pre-trained base model was used to train a second pre-trained expander model trained to predict a class belonging to a same domain as the base class and the custom class; generating metadata to associate the pre-trained base model with the second pre-trained expander model; identifying the second pre-trained expander model based at least in part on the metadata; and connecting the second pre-trained expander model to the pre-trained base model, wherein the enhancement model comprises the pre-trained base model, the first pre-trained expander model, and the second pre-trained expander model.

23. The method of any one of claims 21 or 22, wherein selecting, from a plurality of pre-trained base models, a pre-trained base model trained to predict a base class belonging to a same domain as the custom class comprises: accessing first metadata stored in the data store and associated with the pre-trained base model, wherein the first metadata comprises an identity of the base class; and determining, using a mapping, an association between the base class and the custom class based at least in part on the first metadata and the message, wherein the pre-trained base model is selected based at least in part on the association.

24. The method of any one of claims 21-23, wherein selecting, from the plurality of pre-trained expander models, a first pre-trained expander model comprises: accessing first metadata stored in the data store and associated with the pre-trained base model, wherein the first metadata comprises an identity of the base class; ​ ​ ​ accessing second metadata stored in the data store and associated with the first pre-trained extender model, wherein the second metadata includes an identity of the custom class; and determining, using the mapping, an association between the base class and the custom class based at least in part on the first metadata and the second metadata, wherein the first pre-trained extender model is selected based at least in part on the association.

25. The method of any one of claims 21-24, wherein the enhanced model is an inter-class enhanced model, and wherein connecting the pre-trained base model to the first pre-trained extender model to generate the enhanced model comprises: connecting the first pre-trained fusion layer to an input interface of the enhanced model for receiving the training instance such that the first pre-trained fusion layer is configured to receive the output of the first block and the training instance; connecting the first pre-trained fusion layer to a first pre-trained inter-class extender of the first pre-trained extender model; and connecting the first pre-trained inter-class extender to a second pre-trained fusion layer of the first pre-trained extender model.

26. The method of any one of claims 21-25, wherein the enhanced model is an intra-class enhanced model, and wherein connecting the pre-trained base model to the first pre-trained extender model to generate the enhanced model comprises: connecting the first pre-trained intra-class extender layer to the first pre-trained fusion layer such that a second output of the first pre-trained intra-class extender layer; and connecting the first pre-trained fusion layer to a second block of the pre-trained base model and the second pre-trained intra-class extender layer such that a third output of the first pre-trained fusion layer is received by the second block and the second pre-trained intra-class extender layer.

27. The method of any one of claims 21-26, wherein the method further comprises: generating metadata indicating the generation of the enhanced model; and storing the metadata in the data store.

Images