Image classification system

Abstract

An image classification system is provided for determining a possible classification of an image using multiple machine learning models that share a base machine learning model. The image classification system can be a browser-based system on a user's computing device that obtains the multiple machine learning models from a remote system over a network at once, stores the models locally in the image classification system, and uses the models multiple times without subsequently requesting the machine learning models from the remote system. The image classification system can thus determine a classification that may be associated with an image by running the machine learning models on the user's computing device.

Description

Background Art

[0001] Computing devices are often used to view and interact with images. For example, a computing device may receive an image and automatically classify the image using a label or identifier. Some computing devices can train a machine learning model for classifying images, such that the model is likely to correctly identify or associate a label with any received image. During operation, the computing device can use various machine learning models to associate an image with a specific label, identifier, or other classification. A user can then confirm the computer-generated classification or select a different classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] Reference numerals may be repeated throughout the drawings to indicate corresponding relationships between referenced elements.The drawings are provided to illustrate example embodiments described herein and are not intended to limit the scope of the present disclosure.

[0003] Figure 1 is a block diagram of an illustrative computing environment including an image classification system in accordance with some embodiments.

[0004] Figure 2 is a flow chart of an illustrative process for image classification using a base model and multiple working models, according to some embodiments.

[0005] Figure 3 is a visual representation of an example model architecture utilized by an image classification system in accordance with some embodiments.

[0006] Figure 4 is a flow chart of an illustrative process for image classification using an image classification hierarchy, in accordance with some embodiments.

[0007] Figure 5 is a flow chart of another illustrative process for image classification using an image classification hierarchy in accordance with some embodiments.

[0008] Figure 6A is a user interface diagram of an image classification interface showing various interactive tools for uploading and classifying images, according to some embodiments.

[0009] Figure 6B is a user interface diagram illustrating an image classification interface for evaluating an uploaded image, according to some embodiments.

[0010] Figure 6C is a diagram illustrating an image classification hierarchy according to some embodiments.

[0011] Figure 7 is a block diagram of an illustrative computing system configured to implement features of the present disclosure, according to some embodiments. DETAILED DESCRIPTION

[0012] The present disclosure relates to an image classification system for determining a possible classification of an image using multiple machine learning models that share a base machine learning model. The image classification system can be a browser-based system on a user's computing device that obtains multiple machine learning models from a remote system over a network at once, stores the models locally on the user's computing device, and uses the models multiple times without subsequently requesting the models again from the remote system. Thus, by running the machine learning models any number of times on any number of images on the user's computing device, the image classification system can determine a possible classification associated with the image without requiring remote application programming interface (API) calls to the remote system.

[0013] Traditional systems for image recognition are associated with numerous costs and inefficiencies. Images are typically uploaded to a computing device and then manually classified. For example, after a user uploads a photo of a hotel room, the user may manually scroll through a menu to select a predetermined category from a list of possible categories. If the user wishes to associate an image with multiple categories, upload multiple photos, and so on, the user may need to repeat this process multiple times. Using traditional machine learning systems to predict the category associated with an image can be problematic. For example, some conventional systems rely on a single model to determine multiple possible labels associated with an image, which can lead to poor model training and inaccurate predictions. As another example, some conventional systems rely on multiple complete models to determine different classifications, and the execution of multiple models can result in redundant execution of certain operations or overall inefficient use of computing resources. As yet another example, because a remote API call may be executed each time a user performs an image classification process, or because the retrieval of one or more machine learning models may occur before or during each image classification session, some traditional image classification systems that use machine learning may place a burden on the user's computing resources or cause unacceptable user-perceived delays.

[0014] Aspects of the present disclosure address, among other things, image classification problems such as those discussed above. More specifically, an image classification system obtains multiple machine learning models from a remote system and stores these models in a local persistent data store. The image classification system can then quickly access the models without requiring additional requests to the remote system. The set of models can include a base machine learning model (also referred to as a "base model") and multiple additional machine learning models (also referred to as "working models"). The working models are each configured to accept and analyze the output of the base model. Thus, the base model can be executed once for a particular image, and the base model output can be analyzed by multiple working models to perform the classification task. Generating the base model output once for a particular image and caching the base model output allows it to be processed serially, in parallel, or asynchronously by any number of working models without having to regenerate the base model output for each working model. Furthermore, by separating out the operations performed by the base model and executing the operations once for any particular image, each working model can be smaller, include fewer operations, be more efficient to train, and require less time and computing resources to execute, if each working model includes the operations of the base model—or some analog of it.

[0015] In some embodiments, the working model is configured to determine a classification for an image in one or more levels of an image classification hierarchy. As an example, a user may indicate that a particular image is associated with a "pool / swimming pool" category classification label. The image classification system may then use the working model, trained specifically on images of swimming pools, to determine possible project-specific classification labels indicating the type of swimming pool to determine the type of swimming pool depicted in the image. A project-specific classification may be "possible" in the sense that, based on the output of the working model(s), the classification satisfies one or more classification criteria (e.g., the classification is associated with a score that satisfies a minimum threshold, the classification is associated with a score that is greater than a score associated with the second highest rated classification by at least a threshold amount, etc.).

[0016] Although aspects of some embodiments described in this disclosure will focus on specific examples of classifications, labels, devices, frameworks, machine learning models, algorithms, etc. for illustrative purposes, the examples are illustrative only and are not intended to be limiting. In some embodiments, the techniques described herein can be applied to additional or alternative classifications, labels, devices, frameworks, machine learning models, and algorithms. Various aspects of the present disclosure will now be described with respect to certain examples and embodiments, which are intended to illustrate, not to limit, the present disclosure.

[0017] Image classification environment

[0018] With reference to the illustrative embodiments, Figure 1An example network environment in which aspects of the present disclosure may be implemented is shown. As shown, the network environment may include a remote system 106 and an image classification system 118. The image classification system 118 and the remote system 106 may communicate with each other via one or more communication networks 114. The communication network 114 (also referred to simply as a "network") may be a publicly accessible network of linked networks, potentially operated by various parties such as the Internet. In some embodiments, the communication network 114 may be or may include a personal area network ("PAN"), a local area network ("LAN"), a wide area network ("WAN"), a global area network ("GAN"), or some combination thereof, any or all of which may or may not have access to and / or from the Internet.

[0019] Generally speaking, the remote system 106 can be a computing system operated by an entity (e.g., a business or a user). For example, the remote system 106 can be configured to manage listings and images of items, such as real estate offered for sale or lease. The remote system 106 can host and provide network resources, such as web pages, mobile device applications, scripts, programs, etc. The network resources can include information about the items, including images associated with the items based on a user-specified or automatically determined classification of the object, location, and / or subject matter depicted within the image.

[0020] In some embodiments, the remote system 106 may include a data store 110. In some embodiments, the data store 110 may be configured to store a machine learning model that may be transmitted to the image classification system 118. In some embodiments, the image classification system 118 may transmit each uploaded image and associated classification data to the remote system 106 for storage in the data store 110.

[0021] In some embodiments, as Figure 1As shown, the image classification system 118 may include a user interface unit 122, a model selection unit 126, a base model unit 130, a working model unit 134, a model data store 138, and a base model output data store 140. In some embodiments, the user interface unit 122 may generate a user interface that allows user interaction. For example, the user interface unit 122 may receive a network resource from a server such as the remote system 106. The user interface unit 122 may generate a display of the network resource that includes interactive controls to allow the user to initiate uploading one or more images to the remote system 106 and classifying the images using the image classification system 118. In some embodiments, the model selection unit 126 may access the model data store 138 and select a machine learning model to classify the images to be uploaded to the remote system 106. For example, the model selection unit 126 may select a base model and one or more working models. In some embodiments, the machine learning model stored in the model data store 138 is a convolutional neural network model.

[0022] Although Figure 1 The examples show a limited set of image classification system components, but it should be understood that other arrangements may exist in other embodiments.

[0023] Image classification using a base model and multiple working models

[0024] Figure 2 An example process 200 is shown that may be performed by the image classification system 118 to classify an image using a base model and multiple working models that process the output of the same base model. For example, the process 200 may use Figure 3 A base model network 300 and a working model network 302 are shown.

[0025] When process 200 begins, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., a hard drive, flash memory, removable media, etc.) may be loaded into a memory (e.g., random access memory or "RAM") of a computing device such as a computer. Figure 7 7. In some embodiments, process 200 or portions thereof may be implemented serially or in parallel on multiple processors (on the same computing device or on separate computing devices).

[0026] At block 202, the image classification system 118 may obtain an image to be classified. For example, the image classification system 118 may receive an image selected by a user.

[0027] At decision block 204, the image classification system 118 may determine whether base model outputs for the image have been cached. The image classification system 118 may check the base model output data store 140 to determine whether base model outputs for the image exist. For example, the base model output data store 140 may include base model outputs for the most recent image, as generated by the base model unit 130. As another example, the base model output data store 140 may include base model outputs for each image classified by the image classification system 118. As a further example, the base model output data store 140 may include base model outputs that represent only a subset of the images classified by the image classification system 118 (e.g., only the most recent x images (x being a number), only base model outputs that can fit within the maximum size of the data store 140, etc.). In some embodiments, the base model outputs stored in the base model output data store 140 may be associated with an identifier (e.g., a hash of the image, the name of the image, or some other unique identifier).

[0028] If a base model output for the current image is available in the base model output data store 140, the process 200 may proceed to block 210. Otherwise, if no base model output for the current image is available, the process 200 may proceed to block 206.

[0029] At block 206, the base model unit 130 may analyze the image using a base model, such as a base model stored in the model data store 138. In some embodiments, the base model unit 130 may first generate a base model input representing the image, such as a base model input in the form of a feature vector of data derived from the image. In some embodiments, the base model input may be the image itself, a scaled version of the image, or some other standardized or otherwise pre-processed form of the image for analysis by the base model. The base model unit 130 may process the base model input using the base model to generate a base model output. In some embodiments, the base model output may be a feature vector. A feature vector may be a multi-dimensional vector of data (e.g., a 1280-dimensional vector) that represents the image in a form that can be input into and processed by one or more working models. Such a feature vector may be referred to as a base model output vector.

[0030] At block 208, the base model unit 130 may store the base model output in the base model output data store 140. Generating the base model output once for a particular image and storing it in the base model output data store 140 allows the base model output to be available for processing by any number of working models (serially, in parallel, or asynchronously) without having to regenerate the base model output for each working model.

[0031] At block 210, the model selection unit 126 or some other component of the image classification system 118 may select one or more working models to use for processing the base model output for the current image. For example, the image may be associated with a top level, a middle level, and / or a bottom level of a multi-level hierarchy. The model selection unit 126 may select a working model configured to determine a possible top level classification, a working model configured to determine a possible middle level classification, and / or a working model configured to determine a possible bottom level classification.

[0032] At block 212, the working model unit 134 may process the base model outputs for the current image using the selected working model(s). Each working model may use the base model outputs as input, and each working model may output a classification, or output data from which a possible classification may be determined. In some embodiments, the working model unit 134 may execute multiple working models simultaneously, each of which is trained for a specific task. For example, an image classification system may be configured to analyze images relative to a multi-level hierarchy, where the first level corresponds to a category group, the second level corresponds to individual categories within a higher-level category group, and the third level corresponds to subcategories within the higher-level category. Subcategories may also be referred to as items. The working model unit 134 may execute the following models: a working model configured to "predict" or otherwise determine a possible category group label (e.g., attribute features) associated with an image, a working model configured to determine a possible category associated with an image (e.g., swimming pool), and / or a working model configured to determine a possible item label (e.g., outdoor swimming pool) associated with an image. It will be appreciated that in some embodiments, the working model unit 134 may execute the working models serially (e.g., one at a time) or asynchronously rather than simultaneously. Illustratively, a working model can be trained specifically to predict a narrow range of categories, labels, or other identifiers. Consequently, each working model can advantageously require fewer computing resources to store and execute than a base model. For example, an example base model can be 100MB in size and take perhaps 1 second to execute, while a working model can be 1% the size of the base model and take only 0.01 seconds to execute.

[0033] Model Architecture

[0034] Figure 3is a visual representation of an example model architecture utilized by the image classification system 118, according to some embodiments. Compared to traditional machine learning systems (which rely on a single machine learning model to handle possible classification determinations for any and all categories into which the system can classify an image, or for each possible item label), the image classification system 118 can instead process images via multiple machine learning models, with some machine learning models processing the output of one or more other models to produce a subset or superset of the classification determinations of the other models, etc. Avoiding the use of a single large machine learning model allows the image classification system 118 to maintain high accuracy predictions by executing only small, well-trained working models when necessary, while advantageously reducing the burden on computing resources.

[0035] The model used by image classification system 118 can be implemented as a neural network. Generally speaking, a neural network has multiple layers of nodes. Illustratively, a neural network can include an input layer, an output layer, and any number of intermediate or "hidden" layers between the input and output layers. Each layer can include any number of individual nodes. Nodes in adjacent layers can be connected to each other, and each connection between nodes in adjacent layers can be associated with a corresponding trainable parameter or "weight." Conceptually, a node can be thought of as a computational unit that computes an output value as a function of multiple different input values. The input value can include the output of a node in the previous layer, multiplied by the weight associated with the connection between the node in the previous layer and the current node. When the neural network processes input data in the form of an input vector or a matrix of input vectors, the neural network can perform a "forward pass" to generate a matrix of output vectors. The input vectors can each include n individual data elements or "dimensions" (where n is some positive integer), corresponding to the n nodes of the neural network's input layer. Each data element can be a value, such as a floating-point number or an integer. The forward pass involves multiplying the matrix of input vectors by a matrix representing the weights associated with the connections between nodes in the input layer and nodes in the next layer, and applying an activation function to the result. This process is then repeated for each subsequent neural network layer until an output is generated.

[0036] In some embodiments, convolutional neural networks such as the NASNet Large model, the Inception_V3 model, the Inception_V4 model, the Mobilenet_V2_1.0_224 model, and the Mobilenet_V2_1.0_224_quant model can be used as base models for application to images. The base model can be selected based on considerations such as file size and accuracy. For example, the Mobilenet_V2_1.0_224_quant model can be selected for its smaller file size (e.g., 3.4 MB), while the NASNet Large model can be selected for its enhanced accuracy at the expense of an increased file size (e.g., 355.3 MB).

[0037] like Figure 3 As shown, the image classification system 118 can use a set of neural networks, including a base model network 300 and multiple working model networks 302A, 302B, 302C, and 302D. The image is processed by the base model network 300 (shown as a set of diagonally shaded nodes) to generate a base model output 310. The base model output 310 is passed as an input (a subset of the input) to one or more working model networks 302A, 302B, 302C, and / or 302D, collectively referred to as "302A to 302D" or "302" (shown as a set of unshaded nodes). The working model network 302 generates a working model output 318 representing one or more classification determinations of the image.

[0038] The base model network 300 includes an input layer 304 corresponding to an image selected or uploaded by a user. The base model network 302 also includes one or more hidden layers 306. Although the base model hidden layer 306 is shown as having 5 nodes, it will be understood that in other embodiments, a different number of nodes (e.g., dozens, hundreds, or more nodes) may be used. Additionally, although the base model hidden layer 306 is shown as a single layer, it will be understood that the base model network 300 may be implemented as a deep neural network having multiple hidden layers, each of which may or may not have the same number of nodes as any other hidden layer of the base model network 300. For example, the base model network 300 may have 20, 40, or more base model hidden layers 306.

[0039] In some embodiments, the base model output 310 may comprise a multi-dimensional vector (shown as a rectangle comprising multiple items). For example, the base model output 310 may be a 1280-dimensional output vector that is an encoded representation of the input image. The encoded representation includes information about the input image that is used by the various working model networks 302 to determine the likely classification of the image. The base model network 300 may be run only once for each image, and the base model output 310 may be cached and / or provided to any number of working model networks 302 without rerunning the base model network 300. Advantageously, the image classification system 118 may rely on client-side execution to run the base model network 300. For example, the image classification system 118 runs the base model network 300 within a browser or other client application by utilizing a client-side graphics processing unit and WebGL.

[0040] The output 310 of the base model network 300 is input to one or more working model networks 302. In one example, the multidimensional vector can be passed through one or more working model hidden layers 314 of the working model 302A. The working model 302A can generate a working model output 318 (shown as a sparsely shaded square). In some embodiments, the working model hidden layer 314 can be a 256-dimensional neural network layer that utilizes a rectified linear unit (ReLU) or rectifier as an activation function. Although each working model network 302 is shown as a three-layer neural network, it should be understood that in some embodiments, the working model network can include additional working model hidden layers (e.g., to potentially improve the accuracy of classification determinations), or can exclude working model hidden layers (e.g., to minimize the overall size and complexity of the working model network, thereby reducing the amount of computing resources required to train, transmit, store, and use the working model network 302). Furthermore, while each working model network 302 is shown as having the same structure, in some embodiments, the working model networks may have different structures (e.g., some working model networks may have a different number of working model hidden layers and / or a different number of nodes per layer than other working model networks).

[0041] The working model output 318 from any particular working model network 302 may include a set of elements, such as a vector or array, and each element may correspond to a particular classification that the working model network 302 is configured to determine. The value of each element may represent the likelihood that the image is correctly classified as the classification of the corresponding element. Thus, the working model output 318 may include the likelihood that the image corresponds to each classification that the working model is configured to determine. In some embodiments, a softmax function may be used as the activation function to generate the working model output 318.

[0042] In some embodiments, the image classification system 118 may include a confidence score threshold so that only accurate or confident recommendations are displayed to the end user. A working model utilizing a softmax function can automatically calculate the likelihood of an image being correctly classified in each possible classification in the output layer, the working model being configured to determine each of the above possible classifications. For example, the working model can generate a floating-point number for each possible classification that ranges between a minimum and a maximum value for each classification (e.g., between 0.0 and 1.0). The floating-point numbers for each classification can be summed to a predetermined value (e.g., 1.0). If the corresponding likelihood calculated by the softmax function meets a specific threshold, the image classification system 118 can only present recommendations for a specific classification or result to the end user (e.g., only when the probability is greater than 0.5). In some embodiments, the image classification system 118 can only present recommendations for a specific classification based on additional or alternative criteria, such as only when the corresponding likelihood is greater than the second highest likelihood determined for the set of possible classifications by at least a threshold amount. It should be understood that in some embodiments, other algorithms or functions can be used.

[0043] In embodiments where each working model is configured and trained to determine possible classifications based on a specific level of the hierarchy (e.g., a working model network for determining possible top-level category groups, a working model network for each category group to determine possible categories within the category group, a working model network for each category to determine possible items within the category, etc.), each working model network can be well trained for its specific task and produce higher accuracy than a general model configured to make classification determinations among hundreds of possible options (e.g., among all possible items). Utilizing multiple working model networks rather than relying on one larger model can also allow the system to make detailed classification determinations for subcategories without overlapping with the subcategories.

[0044] In some embodiments, each working model network 302 can advantageously be called using only one line of code, and each working model network 302 can require significantly fewer computational resources than the base model network 300. For example, assuming a user uploads a photo of a swimming pool, the image classification system 118 can dedicate computational resources to executing only those working model networks 302 that are specifically trained to predict the type of swimming pool.

[0045] The training data can be used to train the base model network 300 and each working model network 302. For example, the base model network 300 and / or each working model network 302 can be trained on images that are real-world photographs (e.g., PNG formatted files that include at least 650 pixels of proportional width and height and use 24-bit color). The user can provide any number of labeled images for training (e.g., 2000 photos per project), and the model(s) can then learn to label the images appropriately.

[0046] Advantageously, the image classification system 118 can utilize one or more working model networks while taking into account the user's interactive selections. For example, if the user interacts with the user interface in a manner that indicates that the image is associated with the category label "swimming pool" (e.g., by selecting a category label from a drop-down menu), a working model network trained to determine possible item labels associated with the category "swimming pool" (e.g., indoor pool, outdoor pool, or children's pool) can be selected and executed.

[0047] Hierarchical Image Classification

[0048] Figure 4 is a flow chart of an illustrative process 400 for hierarchical image classification according to some embodiments. When the process 400 begins, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drive, flash memory, removable media, etc.) may be loaded into a memory (e.g., random access memory or "RAM") of a computing device such as Figure 7 4. In some embodiments, process 400 or portions thereof may be implemented serially or in parallel on multiple processors (on the same computing device or separate computing devices).

[0049] At block 402, the image classification system 118 may receive an image. For example, a user may select an image to be classified and uploaded to the remote system 106. Illustratively, the image may be an image of a hotel room to be classified by the image classification system 118.

[0050] At block 404, the image classification system 118 may generate or otherwise obtain a base model output. Figure 2As discussed, the image classification system 118 can determine whether a base model output has been cached for the current image. In some embodiments, the base model output can be stored in the base model output data store 140 or cached in the browser. If no base model output is associated with the image, the image classification system 118 can generate a base model output by processing the image using a base model. The image classification system can then cache the base model output. The base model output can be cached in the browser or stored in the base model output data store 140.

[0051] At block 406, the image classification system 118 may process the base model output using one or more working models based, at least in part, on the multi-level hierarchy. In some embodiments, the model data store 138 may store multiple working models (e.g., dozens or hundreds of working models), and each model may be associated with a respective level of the multi-level hierarchy for image classification. Upon reaching block 406, the image classification system may utilize only models associated with a particular level or subset of the levels of the hierarchy. For example, the first time a particular image reaches block 406, the image classification system 118 may utilize the working model(s) associated with the lowest and / or highest level of the hierarchy. When the same image reaches block 406 again, the image classification system may utilize working model(s) associated with a higher and / or lower level of the hierarchy than previously utilized. The particular level(s) of the hierarchy utilized may depend on the number of times block 406 has been reached for the particular image, interaction data representing user interactions with previous classifications, or other contextual information.

[0052] At block 408, the image classification system 118 may determine a classification associated with the image based on the output of the working model(s) generated above. In some embodiments, the first classification may be associated with a broad, high-level label related to the image. For example, Figure 6CAn example image classification hierarchy is shown. According to the example image classification hierarchy, an image can be associated with a label from a list of category group labels 650, category labels 652, and subcategory labels 654 (subcategory labels may also be referred to as item labels). In this embodiment, the lists of labels 650, 652, and 654 maintain a hierarchical relationship with each other. Category group label 650 is the top-level label and, therefore, the broadest, and may include a limited number of possible options (e.g., three). Category label 652 may include intermediate labels that further expand on specific options within category group label 650. For example, category label 652 may include multiple "attribute feature" labels that may be associated with a broad "attribute feature" category group (e.g., swimming pool, dining, parking, etc.). Subcategory label 654 may include lower-level labels that further expand on the options within category label 652 and correspond to the final label to be applied to the image. For example, subcategory label 654 may include multiple labels associated with the "swimming pool" category label (e.g., infinity pool, children's pool, indoor pool, outdoor pool, etc.). It will be appreciated that other arrangements of labels and identifiers may be used in other embodiments.

[0053] If a user uploads a picture of a hotel room, the first classification may be a broad label from the category group labels 650, such as "attribute features." The image classification system 118 may also determine a second classification associated with the image, such as if a second working model or a working model at a different level of the hierarchy was used in block 406 above. For example, the second classification may be a more specific, lower-level label from the subcategory labels 654, such as "outdoor swimming pool."

[0054] Back to Figure 4 At decision block 410, the image classification system 118 may determine whether to use a working model at an additional level in the classification hierarchy to process the image. This determination may be static (e.g., using each level in the hierarchy) or dynamic (e.g., depending on user interaction or other contextual information). If an additional level in the hierarchy is to be used, the process 400 returns to block 406; otherwise, the process proceeds to block 412.

[0055] At block 412, the image classification system 118 may determine or otherwise generate a final classification output (e.g., a project label) for the image. The final classification output may be based on one or more classification criteria. For example, the final classification may be based on a comparison of the individual outputs (e.g., the classification with the highest likelihood). As another example, the final classification may be based on user interaction (e.g., a user's selection of a possible classification). In some embodiments, the final classification may be based on additional, alternative, and / or combinations of the classification criteria.

[0056] While the above-described embodiments include determining the first classification and the second classification in a sequential order, it will be understood that in other embodiments, other arrangements are possible. For example, in some embodiments, the image classification system may determine the possible first classification and the possible second classification simultaneously. In other embodiments, the image classification system may determine the possible second classification before determining the possible first classification. Furthermore, the present disclosure is not limited to only two identifiers. In some embodiments, the image classification system may determine more than two categories or labels associated with an image.

[0057] Hierarchical Image Classification with User Interaction

[0058] Figure 5 is used for classifying images using a three-level hierarchy (e.g. Figure 6C ) and user interaction with possible classifications. Figure 6A and 6B . When process 500 begins, a set of executable program instructions stored on one or more non-transitory computer-readable media (e.g., hard drive, flash memory, removable media, etc.) may be loaded into a memory (e.g., random access memory or "RAM") of a computing device such as a computer. Figure 7 7. In some embodiments, process 500 or portions thereof may be implemented serially or in parallel on multiple processors (on the same computing device or on separate computing devices).

[0059] At block 502, the image classification system 118 may receive an image. For example, a user may upload an image of a hotel that is locally stored on a user computing device configured to execute the image classification system 118. The image classification system 118 may then suggest subcategory labels at block 504 and category group labels at block 506.

[0060] As described above, in some embodiments, the image classification system 118 may first use a base model to process a received image and generate an output, such as a multi-dimensional vector (e.g., a 1280-dimensional vector), which may be used by multiple working models to determine possible classifications. The image classification system 118 may then cache the base model output for future use by the working models. The image classification system 118 selects one or more working models to determine possible classifications (e.g., categories or labels) associated with the image. In some embodiments, at least two working models are initially used to provide an initial classification determination for the image. One working model may be used to determine a possible top-level classification for the image based on the multi-level image hierarchy being used, while another working model may be used to determine a possible low-level classification for the image.

[0061] Figure 6A An interface 600 is shown through which a user uploads an image 602 (e.g., a photo of a swimming pool). The image classification system 118 can process the base model output of the image 602 using a working model and initially determine that the image 602 may belong to a Figure 6C Category group list 650 is shown, for example, a category group for "attribute features." Image classification system 118 may also use a second working model to process the same base model output for image 602 and determine that image 602 likely belongs to the "beach" subcategory. The determined classification may be displayed in user interface 600, such as in automatic identification section 604. As shown, automatic identification section 604 may provide recommendations for one or more possible classifications: a top-level recommendation (category group) and a bottom-level recommendation (subcategory or item "label"). The user may accept the recommended label for image 602 and complete the classification process, select a top-level recommendation to view different possible choices, or browse through the label or category groups. The user may also be allowed to bypass automatic identification section 604 entirely and provide a label or classification using manual input section 606. In some embodiments, the recommended classification(s) determined by image classification system 118 may only be displayed to the user if the classification is associated with a score (e.g., likelihood or confidence score) that exceeds a threshold. For example, if the working model is trained to distinguish between more than 200 subcategories, the highest scoring category may still have a score below the threshold, indicating that the accuracy of the determination is low. Recommendations based on the output of the working model may be de-emphasized from the display in the automatic identification portion 604, or may be suppressed from the display in the automatic identification portion 604.

[0062] At decision block 508, the image classification system 118 may determine whether the lowest-level classification (e.g., the subcategory label suggestion) from block 504 is confirmed. In some embodiments, the image classification system 118 may confirm the accuracy of its prediction by waiting for user input. For example, the image classification system 118 may display one or more recommended classifications, such as the top-level category group label and the lowest-level subcategory label as described above. The user may select 610 one of the recommended classifications, browse other classifications, or manually enter a classification. If the user selects a lowest-level recommended subcategory label, the process 500 may proceed to block 526 where the selection is processed. Otherwise, if the user does not select a lowest-level recommendation (e.g., if the user selects a top-level recommendation or selects a control to browse top-level labels), the process 500 may proceed to decision block 510. As another example, two selectable buttons displaying "Yes" or "No" may be presented to the user via a user interface. The user may click the "Yes" button to confirm the recommendation displayed in the user interface, or click the "No" button to indicate that the recommendation displayed in the user interface is inaccurate.

[0063] At decision block 510, the image classification system 118 determines whether the user has selected or otherwise confirmed a top-level (e.g., category group) recommendation. If the user has not confirmed the category group label suggestion, then at block 512, the user can manually select a category group label. For example, upon seeing that the image classification system 118 has suggested the category group label "Activities" for uploaded image 602 (a picture of a swimming pool), the user can reject the suggestion by manually selecting a different category group (e.g., attribute features) from a browsable list of links, a drop-down menu listing category group labels, or some other control or interface. In some embodiments, the user can manually enter a category group label to associate with the image, such as in manual input section 606. In some embodiments, the initial determination of subcategory labels and category group labels can be influenced by an initial action by the user. For example, in some embodiments, a user can upload an image that the user wishes to submit to the remote system 106 from a local device and begin browsing predetermined tags (e.g., from a browsable list of links displayed in automatic identification section 604). As the user browses tags, the image classification system 118 can recommend a category group from the possible category groups for display to the user. In other embodiments, a user may upload an image and then actively select a top-level classification, such as a category group label. Given that the user manually selects a category group label, the image classification system 118 may forgo executing a working model configured to generate category group label recommendations. The image classification system 118 may execute the working model to determine a recommended intermediate-level classification, such as a category label, associated with the image 602, where the image 602 is assigned the selected category group label.

[0064] At block 514, once the user selects a top-level tag, or if the initial top-level tag recommendation is confirmed, the image classification system 118 may suggest intermediate-level tags. As described above, the image classification system 118 may execute a working model that is specifically trained to determine intermediate-level tags, such as category tags, when a top-level tag, such as a category group tag, is selected. For example, if an image is associated with the "attribute features" category group tag, the image classification system 118 may execute a working model that is specifically configured to determine a recommended category (e.g., fitness, spa, pool, dining, parking, kitchen, etc.) from over 40 possible category options in the selected category group. In some embodiments, the image classification system 118 may execute a working model that is specifically trained to determine a category if the user previously selected a category group. For example, assuming the user selects the "attribute features" category group tag, the image classification system 118 may use a working model that is trained to determine a category associated with the attribute features (e.g., pool, dining, parking, etc.), rather than predicting a category that may belong to a different category group (e.g., activities or room features). The image classification system 118 may then wait for the user to confirm the category label suggestion, or wait for the user to select a different category label.

[0065] At decision block 516 , the image classification system 118 may determine whether the user has already selected the mid-level tag recommendation, or made a different mid-level tag selection. In some embodiments, the user may activate a link or button to confirm the category recommendation.

[0066] If the user has not yet confirmed the intermediate-level tag recommendation, the user may select an intermediate-level tag at block 518. For example, the user may select a category tag from a list of category tag options or manually type a category tag in the manual input section 606 to associate with the image.

[0067] At block 520, the image classification system 118 may recommend lower-level labels, such as subcategory labels. The recommendations may be based at least in part on the user's previous selections, such as selecting a top-level label or an intermediate-level label (e.g., a category group label or a class label) already associated with the image. For example, if the user has manually selected a category group label (e.g., an attribute feature) and a class label (e.g., swimming pool), the image classification system may use a working model trained to determine possible subcategories (e.g., types of swimming pools) given the selected category group label and class label. In some embodiments, the working model may calculate a score (e.g., likelihood) for each subcategory, but only present the subcategory options with the highest scores. In some embodiments, the image classification system 118 may present multiple subcategory recommendations simultaneously. Furthermore, the image classification system 118 may present subcategory recommendations based on score thresholds. For example, the working model may determine that the likelihood of the image being an indoor swimming pool is 0.3, the likelihood of the image being an outdoor swimming pool is 0.6, and the likelihood of the image being a children's pool is 0.1. However, the image classification system 118 may only display the outdoor swimming pool recommendation to the user because it is the only subcategory option with a score exceeding 0.5. It should be understood that the threshold value can be adjusted or changed to suit the user's preferences. For example, the user may be interested in checking all subcategory recommendations, regardless of the calculated score. Once a subcategory label is recommended, the image classification system 118 can wait for user confirmation.

[0068] At decision block 522, the image classification system 118 may determine whether the user has selected the recommended low-level tag. For example, the user may confirm the subcategory suggestion through user input (e.g., activating a link or clicking a "yes" button). If the user does not confirm the low-level tag recommendation, at block 524, the image classification system 118 may receive a manual selection of a low-level tag (e.g., a subcategory) by the user.

[0069] While the process 500 is described as recommending subcategory labels, category group labels, and category labels that are displayed or recommended in a particular order, it will be understood that other orders or arrangements may be used in other embodiments. For example, the image classification system 118 may initially determine possible or otherwise recommended subcategory labels, category group labels, and category labels associated with an image simultaneously. Additionally, while Figure 5 The example shown in utilizes a hierarchical organization of the labels (eg, subcategory labels are narrower than category labels, which are narrower than category group labels), but other embodiments do not necessarily require such an ordered relationship between the labels.

[0070] Example device components

[0071] Figure 7The general architecture of a computing device 700 including an arrangement of computer hardware and software that can be used to implement various aspects of the present disclosure is depicted. Illustratively, the computing device 700 can be a user device such as a personal computer, laptop computer, tablet computer, mobile phone, etc. The hardware components of the computing device 700 can be implemented on a physical electronic device, as discussed in more detail below. The computing device 700 may include, for example, Figure 7 More, fewer and / or alternative elements may be present in the drawings. However, it is not necessary to show all of these generally conventional elements in order to provide an enabling disclosure. Additionally, Figure 7 The general architecture shown in can be used to implement Figure 1 One or more other components shown in .

[0072] As shown, computing device 700 includes processing unit 702, network interface 704, computer readable medium drive 706 and input / output device interface 708, all of which can communicate with each other through a communication bus. Network interface 704 can provide a connection to one or more networks or computing systems (e.g., remote system 106). Therefore, processing unit 702 can receive information and instructions from other computing systems or services via network 114. Processing unit 702 can also communicate with memory 710 and also provide output information to an optional display (not shown) via input / output device interface 708. Input / output device interface 708 can also accept input from an optional input device (not shown).

[0073] The memory 710 may contain computer program instructions (grouped into units in some embodiments) that the processing unit 702 executes to implement one or more aspects of the present disclosure. The memory 710 corresponds to one or more layers of a memory device, including (but not limited to) RAM, 3D XPOINT memory, flash memory, magnetic storage, etc.

[0074] The memory 710 may store an operating system 712 that provides computer program instructions for use by the processing unit 702 in the general management and operation of the computing device 700. The memory 710 may also include computer program instructions and other information for implementing aspects of the present disclosure. For example, in one embodiment, the memory 710 includes user interface unit instructions 714 for the user interface unit 122 that generates a user interface (and / or instructions therefor) for display on the computing device 700, for example, via a navigation and / or browsing interface (such as a browser or application installed on the computing device 700).

[0075] In addition to and / or in combination with the user interface unit instructions 714, the memory 710 may include model selection unit instructions 716 for the model selection unit 126, which is configured to select and generate a machine learning model to apply to the received image. The memory 710 may also include basic model unit instructions 718 for the basic model unit 130 configured to execute and train a basic model (e.g., a neural network). The memory 710 may also include working model unit instructions 720 for the working model unit 134 configured to execute one or more working models (e.g., neural networks).

[0076] Example Embodiments

[0077] Embodiments of the present disclosure may be described in terms of the following:

[0078] Clause 1: A system for classifying an image, the system comprising:

[0079] a data storage unit including a plurality of machine learning models, the plurality of machine learning models including a basic model and a plurality of working models; and

[0080] One or more processors configured by computer-executable instructions to at least:

[0081] receiving an image selected by a user to be classified into one of a plurality of project labels;

[0082] generating a base model output vector using a base model and the image, wherein the base model output vector represents a plurality of features of the image;

[0083] determining a first item label associated with the image using the base model output vector and a first working model of the plurality of working models;

[0084] determining a class group label associated with the image using the base model output vector and a second working model from the plurality of working models;

[0085] receiving a first user input indicating a selection of a category group label;

[0086] determining a class label associated with the image using the base model output vector and a third working model from the plurality of working models, wherein the third working model is associated with a class group label;

[0087] receiving a second user input indicating a selection of a category label;

[0088] determining a second item label associated with the image using the base model output and a fourth working model of the plurality of working models, wherein the fourth working model is associated with a category label;

[0089] receiving a third user input indicating a selection of a second item label; and

[0090] An image and classification data associated with the image are transmitted, wherein the classification data represents a second item tag.

[0091] Clause 2: The system of clause 1, wherein the base model is used only once, to generate an output using the image.

[0092] Clause 3: The system of clause 1, wherein the base model and the plurality of working models are neural network-based models.

[0093] Clause 4: The system of clause 1, wherein the first working model generates a first confidence value associated with the first item label, and wherein the first item label is displayed based at least in part on the first confidence value exceeding a threshold.

[0094] Clause 5: A computer-implemented method comprising:

[0095] As implemented by a computing system comprising one or more computer processors, the one or more computer processors being configured to execute specific instructions:

[0096] receiving an image to be classified;

[0097] using the base model network and the image to generate a base model output associated with the image;

[0098] generating a plurality of working model outputs using the plurality of working model networks and the base model outputs;

[0099] displaying an image classification interface, the image classification interface being configured to accept user input regarding a classification of the image;

[0100] receiving user interaction data representing user input to the image classification interface; and

[0101] A label associated with the image is determined, wherein the label is determined based at least in part on the plurality of working model outputs and the user input.

[0102] Clause 6: The computer-implemented method of clause 5, wherein each of the plurality of working model outputs is generated using the same base model output.

[0103] Clause 7: The computer-implemented method of Clause 5, further comprising storing the base model output in a cache.

[0104] Clause 8: The computer-implemented method of clause 7, further comprising loading the base model output from a cache, wherein at least one of the plurality of working model outputs is generated after loading the base model output from the cache.

[0105] Clause 9: The computer-implemented method of clause 5, further comprising:

[0106] receiving a base model network and a plurality of working model networks from a remote system via a network; and

[0107] The base model network and the plurality of working model networks are stored in a local data storage portion of the computing system.

[0108] Clause 10: The computer-implemented method of clause 9, further comprising loading the base model network and the plurality of working model networks from a local data store, wherein the local data store comprises a browser-based data store.

[0109] Clause 11: The computer-implemented method of clause 5, wherein generating the plurality of working model outputs comprises: generating a first working model output using a first working model network while generating a second working model output using a second working model network.

[0110] Clause 12: The computer-implemented method of clause 5, wherein generating the plurality of working model outputs comprises: generating a first working model output using the first working model network before generating a second working model output using the second working model network.

[0111] Clause 13: The computer-implemented method of clause 5, wherein the first working model output comprises a first confidence score associated with the label and a second confidence score associated with the second label.

[0112] Clause 14: The computer-implemented method of clause 13, further comprising determining to display the label based on the first confidence score satisfying a threshold.

[0113] Clause 15: The computer-implemented method of clause 13, further comprising determining a display label based on a difference between the first confidence score and the second confidence score.

[0114] Clause 16: The computer-implemented method of clause 5, further comprising:

[0115] receiving a second image selected by a user for classification;

[0116] generating, using the base model network and the second image, a second base model output associated with the second image;

[0117] generating a plurality of second working model outputs using the plurality of working model networks and the base model outputs; and

[0118] A second label associated with the second image is determined, wherein the second label is determined based at least in part on the plurality of second working model outputs, and wherein the second label is different from the label.

[0119] Clause 17: A system comprising a computer-readable memory and one or more processors, the one or more processors being configured by computer-executable instructions to at least:

[0120] A request for a network resource is received from a user computing device, the network resource being configured to:

[0121] Using the base model, generating base model output associated with the image;

[0122] generating one or more working model outputs based at least in part on the base model outputs using the one or more working models; and

[0123] determining a label associated with the image, wherein the label is based at least in part on the one or more working model outputs;

[0124] In response to the request, transmitting the network resource to the user computing device; and

[0125] An image and image classification data representing tags associated with the image are received from the user computing device.

[0126] Clause 18: The system of clause 17, wherein the one or more processors are further configured by the computer-executable instructions to: transmit the base model and the one or more working models to a user computing device.

[0127] Clause 19: The system of clause 17, wherein the label is associated with a confidence score, and wherein the label is displayed based at least in part on the confidence score satisfying a threshold.

[0128] Clause 20: The system of clause 17, wherein the base model and the one or more working models comprise neural network-based models executed on the user computing device.

[0129] the term

[0130] Depending on the embodiment, certain actions, events, or functions of any process or algorithm described herein may be performed in a different order, may be added, combined, or omitted entirely (e.g., not all described operations or events may be required to practice the algorithm). Furthermore, in some embodiments, operations or events may be performed concurrently, such as through multithreading, interrupt handling, or execution on multiple processors or processor cores or other parallel architectures, rather than sequentially.

[0131] The various illustrative logical blocks, modules, routines, and algorithmic steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware or a combination of electronic hardware and computer software. To clearly illustrate this interchangeability, various illustrative components, blocks, modules, and steps have been described above generally in terms of their functionality. Whether this functionality is implemented as hardware or as software running on hardware depends on the specific application and the design constraints imposed on the overall system. The described functionality may be implemented in different ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of this disclosure.

[0132] Furthermore, the various illustrative logical blocks and modules described in connection with the embodiments disclosed herein may be implemented or executed by a machine, such as a computer processor device, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof, designed to perform the functions described herein. A computer processor device may be a microprocessor, but in alternative embodiments, the processor device may be a controller, a microcontroller, or a state machine, a combination thereof, or the like. A processor device may include circuitry configured to process computer-executable instructions. In another embodiment, the processor device includes an FPGA or other programmable device that performs logic operations without processing computer-executable instructions. A processor device may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in combination with a DSP core, or any other such configuration. Although primarily described herein with respect to digital technology, a processor device may also primarily include analog components. For example, some or all of the algorithms described herein may be implemented in analog circuitry or mixed analog and digital circuitry. The computing environment may include any type of computer system, including but not limited to a microprocessor-based computer system, a mainframe computer, a digital signal processor, a portable computing device, a device controller, or a computing engine within an appliance.

[0133] The elements of the methods, processes, routines, or algorithms described in conjunction with the embodiments disclosed herein may be embodied directly in hardware, in software modules executed by a processor device, or in a combination of the two. The software modules may reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, a hard disk, a removable disk, a CD-ROM, or any other form of non-transient computer-readable storage medium. An exemplary storage medium may be coupled to the processor device so that the processor device can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated with the processor device. The processor device and the storage medium may reside in an ASIC. The ASIC may reside in a user terminal. In an alternative, the processor device and the storage medium may reside in a user terminal as discrete components.

[0134] Conditional language used herein, for example, “can,” “could,” “might,” “may,” “e.g.,” and the like, unless otherwise specifically stated or understood in the context of use, is generally intended to indicate that certain embodiments include certain features, elements, and / or steps while other embodiments do not. Thus, such conditional language is generally not intended to imply that features, elements, and / or steps are in any way essential to one or more embodiments, or that one or more embodiments must include logic for determining, with or without other input or prompting, whether such features, elements, and / or steps are included or will be performed in any particular embodiment. The terms “comprising,” “including,” “having,” and the like are synonymous and are used inclusively in an open-ended manner and do not exclude additional elements, features, actions, operations, and the like. Furthermore, the term “or” is used in its inclusive sense (rather than its exclusive sense) such that, for example, when used to connect a list of elements, the term “or” means one, some, or all of the elements in the list.

[0135] Unless specifically stated otherwise, disjunctive language such as the phrase "at least one of X, Y, Z" is otherwise understood in the context, as generally used to present that an item, term, etc. can be X, Y, or Z, or any combination thereof (e.g., X, Y, and / or Z). Thus, such disjunctive language is generally not intended to, and should not, imply that certain embodiments require that at least one of X, at least one of Y, or at least one of Z each be present.

[0136] Unless expressly stated otherwise, articles such as "a" and "an" should generally be interpreted as including one or more of the described items. Thus, a phrase such as "a device configured to..." is intended to include one or more of the recited devices. Such one or more recited devices may also be collectively configured to perform the stated limitations. For example, "a processor configured to perform limitations A, B, and C" may include a first processor configured to perform limitation A in conjunction with a second processor configured to perform limitations B and C.

[0137] Although the above detailed description has shown, described and pointed out the novel features applied to various embodiments, it will be understood that various omissions, substitutions and changes can be made to the form and details of the described devices or algorithms without departing from the spirit of the present disclosure. As can be appreciated, certain embodiments described herein may be embodied in a form that does not provide all of the features and benefits set forth herein, as some features can be used or practiced separately from other features. The scope of certain embodiments disclosed herein is indicated by the appended claims, rather than by the foregoing description. All changes that come within the meaning and range of equivalence of the claims are intended to be included within their scope.

Claims

1. A computer-implemented method comprising: Implemented by a computing system comprising one or more computer processors configured to execute specific instructions: receiving an image to be classified; generating a base model output associated with the image using the base model network and the image; selecting a subset of working model networks from a hierarchy of working model networks for generating a working model output, wherein the subset of working model networks comprises a first working model network of the plurality of working model networks in a first level of the hierarchy and a second working model network of the plurality of working model networks in a second level of the hierarchy; After selecting the subset of the working model network, generating a plurality of working model outputs using the subset of the working model network and the base model outputs; displaying an image classification interface, the image classification interface being configured to accept user input regarding a classification of the image; receiving user interaction data representing user input into the image classification interface; as well as A label associated with the image is determined, wherein the label is determined based at least in part on the plurality of working model outputs and the user input. 2 . The computer-implemented method of claim 1 , further comprising storing the base model output in a cache.

3. The computer-implemented method of claim 2, further comprising loading the base model output from the cache, wherein at least one of the plurality of working model outputs is generated after the base model output is loaded from the cache.

4. The computer-implemented method of claim 1 , further comprising: receiving the base model network and a subset of the working model network from a remote system via a network; as well as The base model network and a subset of the working model network are stored in a local data store of the computing system.

5. The computer-implemented method of claim 4 , further comprising loading the base model network and a subset of the working model network from the local data store, wherein: The local data storage includes a browser-based data storage. 6 . The computer-implemented method of claim 1 , wherein generating the plurality of working model outputs comprises generating a first working model output using the first working model network while generating a second working model output using the second working model network.

7. The computer-implemented method of claim 1 , wherein generating the plurality of working model outputs comprises generating a first working model output using the first working model network before generating a second working model output using the second working model network.

8. The computer-implemented method of claim 6 or 7, wherein the first working model output comprises a first confidence score associated with the label and a second confidence score associated with a second label. 9 . The computer-implemented method of claim 8 , further comprising determining to display the label based at least in part on the first confidence score satisfying a threshold. 10 . The computer-implemented method of claim 8 , further comprising determining to display the label based at least in part on a difference between the first confidence score and the second confidence score.

11. The computer-implemented method of claim 1 , further comprising: receiving a second image selected by a user for classification; generating a second base model output associated with the second image using the base model network and the second image; generating a plurality of second working model outputs using a subset of the working model network and the second base model output; as well as A second label associated with the second image is determined, wherein the second label is determined based at least in part on the plurality of second working model outputs, and wherein the second label is different from the label.

12. A system for classifying an image, the system comprising: a data storage unit for storing executable instructions; and One or more processors in communication with the data store and programmed by the executable instructions to perform at least the following steps: receiving an image to be classified; generating a base model output associated with the image using the base model network and the image; selecting a subset of working model networks from a hierarchy of working model networks for generating a working model output, wherein the subset of working model networks comprises a first working model network of the plurality of working model networks in a first level of the hierarchy and a second working model network of the plurality of working model networks in a second level of the hierarchy; After selecting the subset of the working model network, generating a plurality of working model outputs using the subset of the working model network and the base model outputs; displaying an image classification interface, the image classification interface being configured to accept user input regarding a classification of the image; receiving user interaction data representing user input into the image classification interface; as well as A label associated with the image is determined, wherein the label is determined based at least in part on the plurality of working model outputs and the user input.

13. The system of claim 12, wherein the one or more processors are further programmed with executable instructions to: storing the base model output in a cache; and The base model output is loaded from the cache, wherein at least one of the plurality of working model outputs is generated after the base model output is loaded from the cache.

14. The system of claim 12, wherein the one or more processors are further programmed with executable instructions to: receiving the base model network and a subset of the working model network from a remote system via a network; as well as storing the base model network and a subset of the working model network in a local data storage of the system; as well as The base model network and a subset of the working model networks are loaded from the local data store, wherein the local data store comprises a browser-based data store.

15. The system of claim 12, wherein to generate the plurality of working model outputs, the one or more processors are further programmed by executable instructions to: generate a first working model output using the first working model network, and simultaneously generate a second working model output using the second working model network.

16. The system of claim 12, wherein to generate the plurality of working model outputs, the one or more processors are further programmed with executable instructions to: generate a first working model output using the first working model network before generating a second working model output using the second working model network.

17. The system of claim 15 or 16, wherein the first working model output comprises a first confidence score associated with the label and a second confidence score associated with a second label.

18. The system of claim 17, wherein the one or more processors are further programmed with executable instructions to determine to display the label based at least in part on at least one of the first confidence score satisfying a threshold and a difference between the first confidence score and the second confidence score.

19. The system of claim 12, wherein: The particular level of the hierarchy is selected based at least in part on at least one of a previous user input received prior to the user input and a previous working model output generated prior to a plurality of working model outputs.

20. The system of claim 12, wherein only the first working model network of the plurality of working model networks at the first level of the hierarchy and the second working model network of the plurality of working model networks at the second level of the hierarchy are used to generate a working model output from the base model output.

Images