Prepare training datasets for machine learning algorithms

Abstract

The computer system is configured to receive a dataset of image derived features for a plurality of images, reduce the dimensionality of the dataset, identify clusters in the reduced dimensionality dataset, and generate a visual representation of data points of the reduced dimensionality dataset as icons grouped in the clusters. User input is received to apply user classification labels to images for inclusion in the training dataset. A user interface may be used to present information to a user and receive information from the user to facilitate application of the user classification labels.

Description

Technical Field

[0001] The present disclosure generally relates to the creation of training data sets for training image recognition algorithms. Background Art

[0002] Image recognition algorithms enable machines to distinguish between different images, as well as identify objects represented in images, and in various cases, categorize the images accordingly. However, in order for an image recognition algorithm to perform these tasks, the algorithm must first be trained with a training dataset. A training dataset contains images that can be used to train the algorithm, as well as "truth labels" that indicate what the image represents. In various cases, these truth labels are assigned by humans who view the images in the training dataset. Summary of the invention

[0003] The present disclosure relates to using machine learning techniques to assist users in labeling images with classification labels, thereby preparing a training data set for a target machine learning algorithm. Since the accuracy of the target machine learning algorithm depends in part on the accuracy of the training data set, preparing the training data set typically requires a user to review each individual image and apply classification labels. For a training data set consisting of hundreds or thousands of images, the user review process can be very time consuming and expensive, especially when specially trained users are required to review the images. Therefore, enabling a computer system to facilitate the user review process so that the user review process is more accurate and faster can improve the process of making a training data set. Through a machine learning algorithm, the computer system can group images that may have the same classification label and predict the classification label in an embodiment.

[0004] In various embodiments, a computer system applies a first machine learning algorithm to derive features from a plurality of images. The computer system then applies a dimensionality reduction algorithm to reduce the dimensionality of a database of the image-derived features. Thereafter, the computer system uses a clustering algorithm to identify clusters of data points in the reduced dimensionality dataset. The computer system then generates a visual representation of the data points of the reduced dimensionality dataset and causes one or more of the images to be displayed to a user. The computer system receives user input to apply a user classification label to the image.

[0005] The present disclosure also relates to a user interface for presenting information to a user and receiving input from the user to apply classification labels to images included in a training data set for a target machine learning algorithm. In various embodiments, the user interface includes a two-dimensional or three-dimensional representation of a reduced dimension data set of image data, the image data being derived from a plurality of images and one or more of the plurality of images. The user interface can be used to receive user input to apply user classification labels to the images, thereby preparing a labeled training data set for training the target machine learning algorithm to classify the images. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] Figure 1 is a block diagram illustrating an embodiment of a computer system configured to prepare a training data set using a plurality of images.

[0007] Figure 2 is a sampling of images that may be used to label and include in a training dataset according to various embodiments.

[0008] Figure 3 is a flow chart illustrating an embodiment of a method for creating a training dataset according to various embodiments.

[0009] Figure 4 is a sample visual representation of a dimensionality reduction dataset for multiple images.

[0010] Figures 5A-5F is a display screen of an exemplary embodiment of a graphical user interface, the graphical user interface being composed of Figure 1 User interface operations.

[0011] Figure 6 is a flow chart illustrating an embodiment of a method for creating a training dataset according to various embodiments.

[0012] Figure 7 is a flow chart illustrating an embodiment of a method for creating a training dataset according to various embodiments.

[0013] Figure 8 It is achievable Figure 1 A block diagram of an exemplary computer system illustrating various components of the computer system.

[0014] This disclosure includes references to "one embodiment" or "an embodiment." The appearances of the phrases "in one embodiment" or "in an embodiment" are not necessarily referring to the same embodiment. The particular features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0015] In the present disclosure, different entities (which may be variously referred to as "units", "circuits", other components, etc.) may be described or referred to as being "configured" to perform one or more tasks or operations. This expression - [the entity] is configured to [perform one or more tasks] - is used herein to refer to a structure (i.e., a physical thing, such as an electronic circuit). More specifically, the expression is used to indicate that the structure is arranged to perform the one or more tasks during operation. A structure may be said to be "configured to" perform certain tasks even if the structure is not currently being operated. For example, "a computer system configured to generate a data set" is intended to cover a computer system having circuits that perform that function during operation, even if the computer system is not currently being used (e.g., not connected to a power source). Therefore, an entity described or detailed as being "configured to" perform certain tasks refers to something physical, such as a device, a circuit, a memory storing program instructions that can be executed to implement a task, and the like. The term is not used herein to refer to something intangible. Therefore, the "configured to" construct is not used herein to refer to a software entity, such as an application programming interface (API).

[0016] The term “configured to” is not intended to mean “configurable to.” For example, an unprogrammed FPGA is not considered to be “configured to” perform some specific function, although it may be “configurable to” perform that function and may be “configured to” perform that function after being programmed.

[0017] In the appended claims, the recitation of a structure as being "configured to" perform one or more tasks expressly states that it is not intended to invoke 35 U.S.C. § 112(f) of such claim elements. Accordingly, no claim presented in this application is intended to be construed as having a method-plus-function element. Should the applicant wish to invoke § 112(f) during prosecution, it will use the "method-for" [performing a function] structure to recite the claim elements.

[0018] As used herein, the terms "first," "second," and the like are used as labels for the nouns they precede, and do not imply any type of order (e.g., spatial, temporal, logical, etc.) unless otherwise specified. For example, reference to a "first" and a "second" machine learning algorithm does not imply an order between the two unless otherwise specified.

[0019] As used herein, the term "based on" is used to describe one or more factors that influence a decision. This term does not exclude the possibility that additional factors may influence a decision. That is, a decision may be based solely on specific factors or on specific factors as well as other unspecified factors. Consider the phrase "determining A based on B." This phrase specifies that B is a factor used to determine A or influence the determination of A. This phrase does not exclude that the determination of A may also be based on certain other factors, such as C. This phrase is also intended to cover embodiments in which A is determined solely based on B. As used herein, the phrase "based on" is therefore synonymous with the phrase "based at least in part on."

[0020] As used herein, the word "module" refers to a structure that stores or performs a set of operations. A module refers to hardware that implements the set of operations, or a memory that stores the set of instructions, which, when executed by one or more processors of a computer system, causes the computer system to perform the set of operations. Thus, a module may include a dedicated integrated circuit that implements the instructions, a memory that stores the instructions, and one or more processors that execute the instructions, or a combination of the two. DETAILED DESCRIPTION

[0021] Reference now Figure 1 , a block diagram of an exemplary embodiment of a computer system 100 is shown. In various embodiments, the computer system 100 receives a plurality of images 120 and uses the plurality of images 120 and user input to prepare a training data set 130. In various embodiments, the computer system 100 employs a first machine learning algorithm 102, a dimensionality reduction algorithm 103, a clustering algorithm 106, a second machine learning algorithm 110, and a user interface 108 to prepare a training data set 130 using the images 120.

[0022] In various embodiments, the computer system 100 is any one of a plurality of computer systems configured to receive the image 120, receive user input, and prepare a training data set 130. In various embodiments, the computer system 100 is implemented by a single computing system (e.g., a single server, a desktop computer, a laptop computer, a tablet computer, a smart phone), but in other embodiments, the computer system 100 is implemented by a plurality of computers working together (e.g., a server cloud). In various embodiments, a first portion of the computer system 100 (e.g., a server or a server cloud) is configured to execute various algorithms, while a second portion of the computer system 100 (e.g., a laptop computer, a tablet computer) is configured to implement a user interface 108 to present information to a user and receive information from a user.

[0023] In various embodiments, the plurality of images 120 may be any one of a set of images having metadata, such as user classification labels, available for inclusion in the training dataset 130. In various embodiments, for example, the images 120 include images of cells or other biological samples. In various embodiments, these cell images include a plurality of multispectral cell images, a plurality of multimodal cell images, or both. For example, such images may be created using fluorescence images, in which the sample is stained with a fluorescent dye and excited with a light source. However, the disclosed technology is not limited to cell images and may be used for any type of image that may be included in the training dataset 130 (e.g., plant pictures, animal pictures, pictures of the surroundings taken from a vehicle driving on the street, images of human faces, etc.). In various embodiments, the number of images 120 may vary depending on the criteria of the target machine learning algorithm, the amount of acceptable training time for the target machine learning algorithm, and the amount of desired accuracy in the target machine learning algorithm. For example, a larger set of images 120 may translate into a larger training dataset 130. The amount of time required to train the target machine learning algorithm increases as the size of the training dataset 130 increases, but in various embodiments, the accuracy of the target machine learning algorithm may also increase. In various embodiments, plurality of images 120 includes between 500 and 3000 images 120. In various embodiments, images 120 are randomly selected from a larger pool of images.

[0024] In various embodiments, the target machine learning algorithm may be trained with the training data set 130 to classify other images (i.e., images other than the image 120). In such embodiments, the training data set 130 includes some or all of the images 120, as well as user classification labels applied to these images 120 as described herein. As used herein, a "target machine learning algorithm" includes any image recognition algorithm that can be used to classify images when trained with the training data set 130. For example, in conjunction with Figures 5A-5F In the discussed embodiment, the training data set 130 includes cell images with user classification labels that identify the number of cell nuclei in each image (e.g., one, two, or three or more). After training with the training data set 130, the target machine learning algorithm can be used to determine whether other images contain one, two, or three or more cell nuclei.

[0025] In various embodiments, the first machine learning algorithm 102 is any one of a plurality of algorithms executable to analyze images (e.g., by analyzing pixels of the images) and derive features from these images 120 to generate a dataset of image-derived features. In various embodiments, the first machine learning algorithm 102 is a convolutional neural network (CNN). In some such embodiments, the first machine learning algorithm 102 is an Inception V3 convolutional neural network that has been trained on a large database of images from ImageNet. In embodiments where the first learning algorithm 102 is a CNN, the image-derived features of the images are "bottleneck features" that can be used to describe the content of the image and distinguish between different images 120. In various instances, there may be thousands of bottleneck features per image. The output of the first machine learning algorithm 102 includes a multidimensional dataset in various embodiments (e.g., one dimension for each feature of the image 120). For example, after analyzing the image 120, the first machine learning algorithm 102 generates a dataset of 2048 features for each channel of the image 120. In various instances, the plurality of images 120 includes one to twelve channels of the image.

[0026] In various embodiments, the dimensionality reduction algorithm 104 is any one of a plurality of algorithms executable to reduce the dimensionality of the multidimensional data set output by the first machine learning algorithm 102 into a reduced dimensionality data set by obtaining a set of principal variables to reduce the number of random variables considered. In various embodiments, the dimensionality reduction algorithm 104 reduces the dimensionality by several orders of magnitude. For example, in some embodiments, the first machine learning algorithm 102 outputs 2048 features for each image 120, and the dimensionality reduction algorithm 104 reduces the dimensionality of the data set to three or fewer dimensions. In various embodiments, the dimensionality reduction algorithm 104 can be one or more of principal component analysis (PCA), uniform manifold approximation and projection (UMAP), or t-distributed stochastic neighbor embedding (t-SNE).

[0027] In various embodiments, the dimensionality reduction algorithm 104 may also be executable to receive input from the second machine learning algorithm 110. As discussed herein, the second machine learning algorithm 110 may be executable to output predicted classification labels for the unlabeled images 120 based on user classification labels received via the user interface 108. For each unlabeled image 102, the dimensionality reduction algorithm 104 may be executable to consider these predicted classification labels together with the multidimensional dataset output by the first machine learning algorithm 102 to generate another reduced dimensionality dataset having, for example, three or fewer dimensions.

[0028] In various embodiments, the dimensionality reduction algorithm 104 may be executed to output the reduced dimensionality data set to the clustering algorithm 106. In such embodiments, the clustering algorithm 106 may be executed to determine the clusters of data points in the reduced dimensionality data set. The clustering algorithm 106 may be any one of a plurality of suitable clustering algorithms including, but not limited to, a k-means clustering algorithm or a spectral clustering algorithm. In various embodiments, the number of clusters is set by the user, and the various data points in the reduced dimensionality data set are classified into the nearest clusters. In various embodiments, the plurality of clusters is equal to X times Y clusters, where X is the number of groups into which the user wants to classify the image (e.g., the number of potential user classification labels), and Y is greater than or equal to 1. In various embodiments, for example, Y is equal to 5, but other numbers may also be used. In various embodiments during the second or subsequent iteration (i.e., the user has entered the user classification label, and the second machine learning algorithm 110 and the dimensionality reduction algorithm 104 have outputted the reduced dimensionality data set with the predicted classification label), the clustering algorithm 106 clusters the data points corresponding to the unlabeled image to the nearest classification label. The cluster is presented to the user as the predicted classification label through the user interface 10.

[0029] In various embodiments, user interface 108 is executable to present information to a user and receive input from the user, thereby enabling the user to prepare training dataset 130 using images 120. In various embodiments, user interface 108 is a graphical user interface (GUI) that is executable to present visual representations of data points in the dimensionality reduced dataset (e.g., as described herein with reference to Figure 4 The visual representation 400 discussed above is presented as icons grouped by clusters, where each icon represents one or more specific data points. In various embodiments, various portions of the user interface 108 may be selected so that it displays one or more images 120 associated with one or more specific data points (such as the icons themselves, a list of clusters in the data set, a list of user classification labels, a list of predicted classification labels, or a combination). The user interface 108 may also be executed to receive user input of user classification labels for various images in the images 120. Figures 5A-5F The user interface 108 is discussed in further detail.

[0030] In various embodiments, the second machine learning algorithm 110 may be executed to predict the classification labels of the unlabeled images 120 based on the user classification labels entered by the user for other images 120. In various embodiments, the second machine learning algorithm 110 is an iterative optimization algorithm. In various embodiments, the second machine learning algorithm 110 may be any suitable supervised learning algorithm, including but not limited to a stochastic gradient descent (SGD) model with logarithmic loss or a random forest model. As discussed herein, in various embodiments, the second learning algorithm 110 may be executed to output to the dimensionality reduction algorithm 104. Next, the clustering algorithm 106 clusters the data points of the unlabeled images 120 into the nearest classification labels. In such embodiments, the results of the clustering are presented to the user through the user interface 108 as predicted classification labels. In various embodiments, the user responds to the predicted classification labels by accepting the predicted classification labels as user classification labels or rejecting them and selecting different user classification labels, marking the classification labels of the images 120 as unknown (e.g., leaving them for the second user to review), or completely excluding the images 120 from the training data set 130. In various embodiments, Figure 1 The illustrated loop from dimensionality reduction algorithm 104 to clustering algorithm 106 to user interface 108 to second machine learning algorithm 110 and back to dimensionality reduction algorithm 104 continues to iterate until all images 120 have been labeled or excluded. Figure 3 , 6 , 7 discusses the process of labeling image 120 in further detail.

[0031] In various embodiments, the techniques disclosed herein enable users to prepare training data sets 130 from a plurality of unlabeled images 120 more quickly and accurately. No longer do users need to view each image 120 in isolation and assign a user classification label to that image 120 in order for it to be included in the training data set 130, instead, the various algorithms used by the computer system 100 provide the user with various aids in decision making to make the labeling process more efficient. This is particularly important in situations where the decision of which label to apply to a particular image 120 is reviewed by a specially trained individual (e.g., a microbiologist or radiologist) and their labor time is very expensive.

[0032] In various embodiments discussed herein, the first machine learning algorithm 102, the dimensionality reduction algorithm 104, and the clustering algorithm 106 use machine learning techniques to pre-sort the images 120 into various clusters that are predicted to share visual features and, in many cases, will be assigned the same user classification label. In various embodiments, a user interface 108 is used to display a visual representation of the clusters (e.g., in conjunction with Figure 4400) and the labeled images. As discussed herein, a user is able to view various clusters and assign user classification labels to a plurality of images 120 simultaneously (e.g., by highlighting a plurality of images 120 and applying a label to each highlighted image). As discussed herein, this clustering process uses an "unsupervised" (i.e., no user input is used in the initial clustering) training technique, which is then reviewed by the user to prepare labeled material for the training data set 130.

[0033] As discussed herein, after a plurality of user classification labels have been entered, using the second learning algorithm 120, in various embodiments, the computer system 100 can take into account the user's input to further streamline the labeling process. As discussed herein, by using the second learning algorithm 110, the computer system 100 can be used to predict which classification labels may be correct for images 120 that remain partially (or entirely) unlabeled. In various embodiments, the dimensionality reduction algorithm 104 takes into account the output of the second learning algorithm 110 to generate a second reduced dimensionality data set, and then clusters the second reduced dimensionality data set using the clustering algorithm 106. As discussed herein, the user interface 108 is updated to display to the user a cluster of predicted user classification labels (e.g., in combination with Figure 5E and 5F 512 discussed herein). As discussed herein, a user is able to view various clusters and assign user classification labels to multiple images 120 simultaneously (e.g., by highlighting multiple images 120 and applying labels to each highlighted image). As discussed herein, this clustering process uses a "semi-supervised" (i.e., previous user input is used for modified clusters, but the user has not yet viewed all images) training technique, which is then reviewed by the user to prepare labeled material for the training dataset 130. Thus, in various embodiments, the techniques disclosed herein provide a guided path from unsupervised clustering to semi-supervised prediction to a user who is labeling images 120 for the training dataset 130, while providing a visualization and intuitive user interface to aid in decision making.

[0034] Reference now Figure 2 , shows a sampling of images 120. In various embodiments, each image 120 includes a visual portion and metadata 202 (e.g., the name when the particular image 120 was created, etc.). Figures 5A-5F As described above, when a particular image 120 is viewed and tagged (e.g., using the user interface 108), in various embodiments, the image is represented using an object 200. In various embodiments, the object 200 includes metadata 202 about the particular image (such as Figure 2As discussed herein, in various embodiments, selection object 200 allows a user to apply a user classification label (and / or respond to a predicted classification label).

[0035] Figure 2 A small number of examples of image-derived features 204 are also included. In various embodiments, the first machine learning algorithm 102 derives various features from the image 120. In various examples, these features are represented by mathematical descriptions of the pixel data of the image 120. However, these image-derived features 204 are visually represented as parts of the image 120 that collectively describe the image 120, making it distinguishable from other images 120. Therefore, Figure 2 Three image-derived features 204a, 204b, and 204c are shown in FIG. 1 , but the number of image-derived features may be much greater than the three discussed above (eg, thousands of features per image 120 in various embodiments).

[0036] Reference now Figure 3 , a flow chart illustrating an embodiment of a training data set creation method 300 is shown. In various embodiments, various actions associated with the method 300 are performed using a computer system 100 .

[0037] At block 302, a user inputs an unlabeled (or insufficiently labeled) training data set (e.g., a plurality of images 120) for labeling to prepare a training data set 130. As discussed herein, the user may randomly select images 120 to be labeled from a larger set of images. The user may input the images 120 by any suitable method, including but not limited to inserting a storage medium (e.g., a disk or hard drive) or downloading the images 120 to the computer system 100. In various embodiments where some of the techniques discussed herein are performed by a computer system 100 implemented on a remote computer cloud, the user may upload the images 120 to the cloud for processing.

[0038] At block 304, the computer system 100 derives (e.g., using the first machine learning algorithm 102) features (e.g., Figure 2204 shown). At block 306, the computer system 100 reduces the dimensionality of the features 204 (e.g., using the dimensionality reduction algorithm 104). In various examples, the dimensionality reduction is performed on a dataset of derived features 204 prepared by the first machine learning algorithm 102 using the plurality of images. In other examples (e.g., when the method 300 enters block 306 from block 314), the dimensionality reduction is performed on the dataset of derived features 204 while taking into account user classification labels that have been applied to some of the plurality of images 120. In either example, the dimensionality reduction algorithm 104 receives a relatively large dimensionality dataset for each of the plurality of images 120 (e.g., a dataset of 2048 features for the images 120) and reduces it to a significantly smaller dimensionality, such as two dimensions in some embodiments, or three dimensions in other embodiments.

[0039] At block 308, the computer system 100 prepares a visual representation 400 (also referred to herein as an "object map") of the data points of the reduced dimension dataset. Figure 4 As discussed in further detail, in various embodiments, the visual representation 400 is a two-dimensional graph whose icons represent one or more data points in the reduced dimensionality database. In other embodiments, the visual representation 400 is a three-dimensional graph whose icons represent one or more data points in the reduced dimensionality database.

[0040] At block 310, a decision is made whether to predict classification labels for the plurality of images 120. In various embodiments, the computer system 100 is configured to make this decision based on the number of user classification labels that have been entered. For example, if the percentage of labeled images 120 is below a threshold (e.g., 30%, 40%, or any other threshold), or when no user classification is received, the decision is made automatically and the method 300 proceeds to block 312. If the percentage is above the threshold, the method 300 proceeds to block 310. In various embodiments, the decision is made by a user who determines whether the method 300 should proceed to block 312 or block 314, and the computer system 100 proceeds according to instructions from the user.

[0041] At block 312, in various embodiments, the computer system 100 clusters (e.g., using the clustering algorithm 106) the dimensionality reduction dataset into a predetermined number of clusters. In iterations of the method 300 where no predicted classification labels are generated, the clustering algorithm clusters the data points into X by Y clusters, where X is the number of groups into which the user wants to classify the image (e.g., the number of user classification labels); and Y is greater than or equal to 1 (e.g., 3, 4, 5). In various embodiments, these clusters are incorporated herein in conjunction with Figure 4 and 5A -5F Visual representation of discussion 400.

[0042] At box 316, after determining the predicted classification label, the computer system 100 (e.g., using the second learning algorithm 110) predicts the classification label for the unlabeled image in the plurality of images 120. In the iteration of the method 300 in which the classification label is predicted, the various data points are clustered into clusters for each user classification label. In such embodiments, the data points representing images 120 with the user classification label are clustered into clusters associated with their respective labels, while the unlabeled data points are clustered into the closest cluster as the predicted classification label. The computer system 100 generates a visual representation 400 including the predicted classification label. In various embodiments, as shown in reference Figure 5D , 5E As discussed in further detail in 5F, a visual representation 400 of the update appears on the user interface.

[0043] At blocks 314 and 318, the computer system 100 receives user input to apply a user classification label. At block 314, the computer system 100 receives user input to apply a user classification label to one or more unlabeled images 120. In various embodiments, as shown in FIG. Figure 5C As discussed in further detail, the input is received through a menu that appears on the user interface. Similarly, in various embodiments, at block 318, the computer system 100 receives user input to apply a user classification label to one or more unlabeled images 120 that have been assigned a predicted classification label. In various embodiments, as described in reference Fig. 5F As discussed in further detail, the input is received via a menu that appears on the user interface 108. In various embodiments, the user classification labels include labels for various images 120 that describe the content contained in the image for training the target machine learning algorithm (e.g., Figures 5A-5F As discussed, the image contains one core, two cores, or three or more cores. In various embodiments, the user classification label can also be a label that excludes the image from the training data set 130. In various embodiments, the user classification label can be unknown (e.g., the user cannot identify the label to be applied). In various embodiments, the images 120 marked as unknown and excluded are not included in the training data set 130. After blocks 314 and 318, in various embodiments, if some of the plurality of images 130 remain unlabeled, the method 300 loops back to block 306.

[0044] Reference now Figure 4 , depicts an example visual representation 400 of a reduced dimension dataset for a plurality of images 120. Figure 4In the illustrated embodiment, the graphical representation 400 is a two-dimensional presentation that includes a plurality of icons 404 representing data points in a dimensionality reduction dataset output by the dimensionality reduction algorithm 104 and grouped into clusters 402. Figure 4 In the illustrated embodiment, the data points in the dimensionality reduction data set have been grouped into 15 clusters (404a-404o). In various embodiments, the clustering is represented as a visual representation 400 using one or more techniques including, but not limited to (a) presenting the visual representation 400 such that icons 404 corresponding to data points in the same cluster 402 are placed close together, (b) presenting the visual representation 400 such that icons 404 corresponding to data points in the same cluster 402 are shaded in the same color (e.g., cluster 402a in red, cluster 402b in blue, and cluster 402c in green), (c) presenting the visual representation 400 such that icons 404 corresponding to data points in the same cluster 402 are surrounded by polygons or combinations thereof. In various embodiments, the various icons 404 are arranged in a plurality of different shapes. Figure 4 The positions in the two-dimensional embodiment of the illustrated visual representation 400 are based on two dimensions of the reduced dimensionality dataset (e.g., the X-axis coordinate is based on the first dimension, and the Y-axis coordinate is based on the second dimension). Similarly, when the reduced dimensionality dataset has three dimensions, the visual representation 400 is a three-dimensional image, wherein the positions of the various icons 404 are based on the three dimensions of the reduced dimensionality dataset (e.g., the X-axis coordinate is based on the first dimension, and the Y-axis coordinate is based on the second dimension, and the Z-axis coordinate is based on the third dimension). As discussed herein, the number of clusters may vary depending on the number of user classification labels and whether predicted classification labels have been generated.

[0045] As referenced in this article Figure 5E and Fig. 5F As discussed, in various cases an updated visual representation 512 is generated, the updated visual representation 512 showing the reduced dimension data set output by the dimension reduction algorithm 104 (this time taking into account the output of the second machine learning algorithm 111), the reduced dimension data set being grouped into clusters for each user classification label. In various embodiments, the clustering is represented as a visual representation 400, using one or more techniques including, but not limited to, (a) presenting the visual representation 512 so that icons 404 corresponding to data points in the same cluster 402 are placed closer together, (b) presenting the visual representation 512 so that icons 512 corresponding to data points in the same cluster 402 are shaded with the same color, (c) presenting the visual representation 512 so that icons 404 corresponding to data points in the same cluster 402 are surrounded by polygons or combinations thereof.

[0046] Reference now Figures 5A-5F, illustrates various display screens of an exemplary embodiment of a graphical user interface (GUI) 500 operated by the user interface 108 according to the disclosed embodiments. In various embodiments, the GUI 500 is displayed on a display screen (e.g., a monitor, a laptop display, a tablet display) coupled directly (e.g., via an HDMI cable) or indirectly (e.g., transmitted to the display screen via a WAN and / or LAN) to the computer system 100. As discussed herein, the GUI 500 can be used to present information to a user and receive input from a user (e.g., input of a classification image 130) to prepare a training data set 130 for training a target machine learning algorithm.

[0047] In each screen of the GUI 500, multiple areas are used to display various information discussed herein. In various embodiments, each screen includes a first area 510 that includes a two-dimensional visual representation 400 (or an updated visual representation 512). As discussed herein, the visual representation 400 represents a reduced dimension dataset of image data derived from a plurality of images. In various embodiments, the two-dimensional visual representation 400 includes a plurality of icons 404 and indications of clusters 402 in the dataset. In various embodiments, each screen also includes a second area that includes one or more of the plurality of images 130. In various embodiments, the various screens include a third area 530 for displaying a list of identified clusters (and, in embodiments, the number of images 130 grouped into each). In various embodiments, the various screens include an updated third area 532 for displaying a list of predicted classification labels (and, in embodiments, the number of images 130 grouped into each). In various embodiments, the various screens include a fourth area 540 for displaying a list of user classification labels (and, in embodiments, the number of images 130 labeled with each label). Figures 5A-5F , the first area 510 is set on the right side of the GUI 500, the second area 520 is set in the middle of the GUI 500, the third area 530 (and the updated third area 532) and the fourth area 540 are set on the left side of the GUI 500, but these various areas can be arranged in any order. Figures 5A-5F The various regions in FIG. 1 are depicted as part of the same window, but in other embodiments, some or all of the regions may be presented as separate windows. Figure 3 , the actions of blocks 302-312 are Figure 5A The actions of block 314 are performed prior to display of the screen depicted in FIG. Figure 5C The decision at block 310 is performed during display of the screen depicted in Figure 5D The actions of block 316 are performed during display of the screen depicted in FIG. Figure 5E, and the actions of block 316 are performed before the display of the screen depicted in Fig. 5F Executed during display of the screen depicted.

[0048] In various embodiments, the first area 510 may be used to display the Figure 4 The visual representation 400 discussed herein, or referenced herein Figure 5E and Fig. 5F In various embodiments, each icon 404 of the visual representation 400 (or updated visual representation 512) represents one or more data points in the reduced dimensionality dataset. Further, in such embodiments, each icon 404 represents one or more of the plurality of images 120 and can be selected to cause the represented image 120 to be displayed in the second area 520.

[0049] In various embodiments, the second area 520 may be used to display one or more images 120. In various embodiments, the display of the images 120 displayed in the second area 520 is responsive to a user selection of a portion of the first area 510 (e.g., causing one or more icons 404 representing an image 120 to be displayed), responsive to a user selection of a portion of the third area 530 or an updated third area 532 (e.g., corresponding to a particular cluster causing a portion of a list of images 120 associated with the cluster to be displayed), and / or responsive to a user selection of a portion of the fourth area 540 (e.g., corresponding to a user classification tag causing a portion of a list of images 120 tagged with a particular user classification tag to be displayed). In various embodiments, each image 120 displayed in the second area 520 is displayed as an object 200. As referred to herein, Figure 2 As discussed, each object is associated with metadata for the image 120 and is selectable. For example, as discussed herein, selecting an image allows a user to apply a user classification label or a response predicted classification label for the selected image 120.

[0050] In various embodiments, the third region 530 may be used to display a list of clusters in the data set. Similarly, the updated third region 532 (also referred to herein as the "fifth region") may be used to display a list of predicted classification labels according to which the remaining unlabeled images 120 are clustered. In either case, in various embodiments, each entry in the list may be selected to cause the image 120 associated with that cluster to be displayed in the second region 520. In various embodiments, the lists displayed in the third region 530 and the updated third region 532 include a corresponding indication of the number of images 120 associated with each cluster.

[0051] In various embodiments, the fourth area 540 may be used to display a list of user classification tags applied to the images 120. In some such embodiments, each entry of the list may be selected to cause an image 120 tagged with the user classification tag to be displayed in the second area 520. In various embodiments, the list displayed in the fourth area 540 includes a corresponding indication of the number of images 120 tagged with each user classification tag.

[0052] Reference now Figure 5A , showing a first screen of the GUI 500. Prior to display of this first screen, an image 120 has been received by the computer system, a first machine learning algorithm 102 has derived features from the image 120, the dimensionality of the data set of the derived features has been reduced by a dimensionality reduction algorithm 104, and clusters have been determined by a clustering algorithm 106. A visual representation 400 of the reduced dimensionality data set is displayed in a first area 510. Several images 120 are displayed in a second area 520, however, because a user selection has not yet been received, the displayed images 120 are not associated with a particular cluster (e.g., they may be displayed randomly; they may be displayed in the order in which they were captured; they may be displayed in alphabetical order by name). A list of clusters is displayed in a third area 530, with the list of clusters having an indication of the number of images 120 associated with each cluster. Finally, a list of three user classification labels used in this case is displayed in a fourth area 540. In the Figures 5A-5F In the example shown, the user classification labels are determined based on the number of cell nuclei present in each image 120: 1N for images 120 that include one nucleus; 2N for images 120 that include two nuclei; and 3-4N for images that include three or more nuclei 120. As discussed herein, more than three user classification labels may be used, and in each case, the criteria for determining which label should be applied to a particular image 120 may also vary.

[0053] Reference now Figure 5B , a user selection of one or more icons 404 in cluster 402b has been received. In response, images 120 associated with cluster 402b are displayed in second area 520, and a portion of the list in third area 530 associated with cluster 402b is highlighted.

[0054] Reference now Figure 5C In various embodiments, the user classification labels are received from the user via a menu 522 displayed in the GUI 500. In various embodiments, the menu 522 includes an indication of each user classification label, including various labels for the training data set 130 and additional labels, such as labels for excluding one or more images 120 from the training data set 130. Figure 5CIn the example shown in FIG. 5 , the menu 522 includes indications of three user classification labels 1N, 2N, and 3_4N, as well as commands of “Move to Unknown” and “Move to Excluded” to mark the selected image 120 accordingly. Figure 5C As shown in , a number of images 120 are highlighted in the second area 520 , and user input to the menu 522 will apply a user classification label or command to the highlighted images 120 .

[0055] Reference now Figure 5D , user input for applying user classification labels to various images 120 has been received. As shown in the fourth area 540, 170 images are labeled 1N, 110 images are labeled 2N, and 146 images 120 are labeled 3_4N. Figure 5D In the embodiment shown in , by clicking button 524, the user can enter a command to predict a classification label. In response to the command, the predicted classification label is assigned as discussed in block 316 of method 300. Alternatively, after a threshold number of images 120 are labeled, the prediction of the classification label is automatically completed.

[0056] Reference now Figure 5E , the GUI 500 now includes an updated visual representation 512 and an updated third area 532. As discussed herein, the remaining unlabeled images 120 are assigned predicted classification labels and clustered with the predicted classification labels. Accordingly, the visual representation 512 includes four clusters: one associated with each user classification label, and one for images whose classification labels have not (or cannot be) determined for any reason. The updated third area 532 includes a list of predicted classification labels: Prediction 1N, Prediction 2N, Prediction 3_4N, and Unknown, as well as an indication of the number of images 120 with each predicted label. In various embodiments, the updated visual representation 512 also includes data points associated with images 120 with user classification labels. The icons 404 representing these tagged images 120 may be visually different from the icons 404 representing untagged images, including, but not limited to, being different colors (e.g., icons 404 representing tagged images 120 being darker than icons 404 representing untagged images 120, but within the same color family, such as dark green and light green, or dark blue and light blue) or being different shapes (e.g., circular icons 404 for untagged images and star-shaped icons for tagged images 120). An icon 404 of the visual representation 512 updated as discussed herein with the visual representation 400 may be selected to cause display of the image 120 represented by the selected icon 404. Figure 5E In the screen shown in , the predicted 1N cluster is selected. In response to the selection, the images 120 in the predicted 1N cluster are displayed in the second area 520.

[0057] Reference now Fig. 5F In various embodiments, the user responds to the predicted classification label by a command received via a menu 522 displayed in the GUI 500. In various embodiments, the menu 522 allows the user to accept the predicted classification label as the user classification label by selecting an indication of the user classification label corresponding to the predicted classification label in the menu 522, or to reject the predicted classification label by selecting a different indication in the menu 522. In various embodiments, the menu 522 includes an indication of each user classification label, including various labels for the training data set 130 and additional labels, such as labels for excluding one or more images 120 from the training data set 130. Fig. 5F In the example shown in FIG. 5 , the menu 522 includes indications of three user classification labels 1N, 2N, and 3_4N, as well as commands of “Move to Unknown” and “Move to Excluded” to mark the selected image 120 accordingly. Fig. 5F As shown in , a number of images 120 are highlighted in the second area 520 , and user input to the menu 522 will apply a user classification label or command to the highlighted images 120 .

[0058] Reference now Figure 6 , a flow chart illustrating an embodiment of a training data set creation method 600 is shown. In various embodiments, various actions associated with the method 600 are performed using a computer system 100. At box 602, the computer system 100 receives a data set of image-derived features for each of a plurality of images, wherein the image-derived features are determined by analyzing the plurality of images 120 using a first machine learning algorithm 102. At box 604, the computer system 100 generates a reduced dimension data set from the data set of image-derived features using a dimensionality reduction algorithm 106. At box 606, the computer system identifies a plurality of clusters of data points in the reduced dimension data set using a clustering algorithm 106. At box 608, the computer system 100 generates a visual representation 400 of the data points as icons 404 grouped in clusters 402. Each icon 404 represents one or more specific data points and can be selected to cause display of one or more images 120 associated with the one or more specific data points. At box 610, the computer system 100 receives a selection of one or more icons 404. At block 612, the computer system 100 causes display of images 120 associated with the one or more specific data points represented by the one or more selected icons 404. At block 614, the computer system 100 receives a user classification label for at least one of the displayed images. At block 616, the computer system 100 predicts a classification label for the unlabeled image 120 and receives a user response to the predicted classification label.

[0059] Reference now Figure 7 , a flow chart illustrating an embodiment of a training data set creation method 700 is shown. In various embodiments, various actions associated with the method 700 are performed using a computer system 100. At box 702, the computer system 100 causes a user interface (e.g., a GUI 500) to prepare a labeled training data set 130, which is used to train a target machine learning algorithm to classify images to be displayed on a user device. The user interface includes a first area 510, which includes a two-dimensional visual representation 400 of a reduced dimension data set of image data derived from a plurality of images 120. The two-dimensional visual representation 400 includes a plurality of icons 404 and indications of clusters 404 in the data set. The user interface also includes a second area 520, which includes one or more of the plurality of images 120. At box 704, the computer system 100 receives a user input that applies a user classification label to one or more images 120 displayed in the second area 520.

[0060] Exemplary Computer System

[0061] Now turn to Figure 8 , depicts a block diagram of an exemplary computer system 800 that may implement various components of computer system 100. Computer system 800 includes a processor subsystem 880 coupled to system memory 820 and I / O interface 840 via an interconnect 860 (e.g., a system bus). I / O interface 840 is coupled to one or more I / O devices 850. Computer system 800 may be any type of device, including, but not limited to, a server system, a personal computer system, a desktop computer, a laptop or notebook computer, a mainframe computer system, a tablet computer, a handheld computer, a workstation, a network computer, a consumer device (such as a cell phone), a music player, or a personal digital assistant (PDA). Although for convenience, the computer system 800 is not described herein. Figure 8 While a single computer system 800 is shown, system 800 may also be implemented as two or more computer systems operating together.

[0062] Processor subsystem 880 may include one or more processors or processing units. In various embodiments of computer system 800, multiple instances of processor subsystem 880 may be coupled to interconnect 860. In various embodiments, processor subsystem 880 (or each processor unit in 880) may include cache memory or other forms of onboard memory.

[0063] The system memory 820 may be used to store program instructions executable by the processor subsystem 880 to cause the system 800 to perform various operations described herein. The system memory 820 may be implemented using different physical memory media, such as hard disk storage, floppy disk storage, removable disk storage, flash memory, random access memory (RAM—SRAM, EDORAM, SDRAM, DDR SDRAM, RAMBUS RAM, etc.), read-only memory (PROM, EEPROM, etc.), etc. The memory in the computer system 800 is not limited to primary storage, such as memory 820. Instead, the computer system 800 may also include other forms of storage, such as cache memory in the processor subsystem 880 and auxiliary storage on the I / O device 850 (e.g., hard disk drive, storage array, etc.). In some embodiments, these other forms of storage may also store program instructions executable by the processor subsystem 880.

[0064] According to various embodiments, the I / O interface 840 may be any type of interface configured to couple to and communicate with other devices. In one embodiment, the I / O interface 840 is a bridge chip (e.g., a south bridge) from a front-end bus to one or more back-end buses. The I / O interface 840 may be coupled to one or more I / O devices 850 via one or more corresponding buses or other interfaces. Examples of I / O devices 850 include storage devices (hard drives, optical drives, removable flash drives, storage arrays, SANs, or their associated controllers), network interface devices (e.g., to a local area network or a wide area network), or other devices (e.g., graphics cards, user interface devices, etc.). In one embodiment, the computer system 800 is coupled to a network via a network interface device 850 (e.g., configured to communicate via WiFi, Bluetooth, Ethernet, etc.).

[0065] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of the present disclosure, even with respect to a single embodiment described with respect to a particular feature. Unless otherwise stated, the examples of features provided in the present disclosure are intended to be illustrative rather than restrictive. The above description is intended to encompass alternatives, modifications, and equivalents that are apparent to those skilled in the art having benefit of the present disclosure.

[0066] The scope of the present disclosure includes any feature or combination of features disclosed herein (explicitly or implicitly), or any generalization thereof, whether or not it mitigates any or all of the problems mentioned herein. Accordingly, during the examination of the present application (or an application claiming priority), new claims may be made for any combination of such features. In particular, with reference to the attached claims, features of the dependent claims may be combined with features of the independent claims, and features of the respective independent claims may be combined in any suitable manner, not just in the specific combinations listed in the attached claims.

Claims

1. A method, comprising: (a) receiving, by a computer system, a dataset of image-derived features for each of a plurality of images, wherein the image-derived features are determined by analyzing the plurality of images using a first machine learning algorithm; (b) generating a reduced dimension dataset from the dataset of image-derived features using a dimensionality reduction algorithm by the computer system; (c) using a clustering algorithm by the computer system to identify a plurality of clusters of data points in the reduced dimensionality data set; (d) generating, by the computer system, a visual representation of the data points as icons grouped by clusters, wherein each icon represents one or more specific data points and is selectable to cause display of one or more of the plurality of images associated with the one or more specific data points; (e) receiving, at the computer system, a selection of one or more of the icons; (f) in response to the selection, causing, by the computer system, display of an image associated with the one or more specific data points represented by the one or more selected icons; (g) receiving, at the computer system, user classification labels for a first set of images from the plurality of images; (h) generating, by the computer system, a second reduced dimension dataset of a dataset of features derived from the images for a second set of images from the plurality of images using the dimensionality reduction algorithm and a second machine learning algorithm distinct from the first machine learning algorithm and using the user classification labels for the first set of images, wherein the second set of images includes unlabeled images; (i) identifying, by the computer system, predicted classification labels for at least some of the unlabeled images in the second image set using the clustering algorithm and the second machine learning algorithm; and (j) receiving, at the computer system, a user response to the predicted classification labels for one or more unlabeled images.

2. The method according to claim 1, characterized in that The user response is to accept the predicted classification label as a user classification label for the one or more specific unlabeled images.

3. The method according to claim 1, characterized in that The user response is a rejection of the predicted classification label for the one or more specific unlabeled images.

4. The method according to claim 1, characterized in that The user response is a command to exclude the one or more specific unlabeled images from the classification.

5. The method according to any one of claims 1 to 4, characterized in that Further comprising repeating steps (h) to (j) until all of the plurality of images are labeled with the user classification label, or are excluded from classification.

6. The method according to any one of claims 1 to 4, further comprising: (k) generating, by the computer system, a visual representation of the data points in the second reduced dimensionality dataset as second icons grouped by the predicted classification labels, wherein each second icon represents one or more specific data points in the second reduced dimensionality dataset and is selectable to cause display of one or more images associated with the one or more specific data points in the second reduced dimensionality dataset; (1) receiving, at the computer system, a selection of one or more of the second icons; (m) in response to the selection, causing, by the computer system, display of an image associated with the one or more specific data points represented by the one or more selected second icons; Wherein a second user response to the predicted classification label is received in response to display of an image associated with the one or more specific data points represented by the one or more selected second icons.

7. The method according to any one of claims 1 to 6, characterized in that The second machine learning algorithm is an iterative optimization algorithm.

8. The method according to any one of claims 1 to 7, characterized in that The plurality of images include images of cells.

9. The method according to claim 8, characterized in that The cell image includes a plurality of multispectral cell images, a plurality of multimodal cell images, or both.

10. The method according to any one of claims 1 to 9, characterized in that The plurality of images are randomly selected from a larger pool of images.

11. The method according to any one of claims 1 to 10, characterized in that The first machine learning algorithm is a convolutional neural network.

12. The method according to any one of claims 1 to 11, characterized in that The dimensionally reduced dataset has three or fewer dimensions.

13. The method according to any one of claims 1 to 12, characterized in that The plurality of clusters is equal to X times Y clusters, where X is the number of groups into which the user wants the image to be classified; and Y is greater than or equal to one.

14. The method according to any one of claims 1 to 13, characterized in that The visual representation of the data points as icons grouped by the clusters of step (d) includes a two-dimensional presentation of the icons grouped by clusters, wherein icons in the same cluster are shaded with the same color and positioned close together.

15. The method according to any one of claims 6 to 14, characterized in that The visual representation of the data points in the second reduced dimensionality data set as second icons grouped by the predicted classification labels of step (k) includes a two-dimensional presentation of the second icons grouped by the predicted classification labels, wherein second icons in the same predicted classification label are shaded with the same color and positioned close together.

16. The method of any one of claims 1 to 15, further comprising: (n) causing, by the computer system, display of a user interface comprising: a visual representation of the data points as icons grouped by clusters; an image gallery of one or more images associated with the one or more specific data points represented by the one or more selected icons; a first list including an indication of each cluster; and The second list includes an indication of a classification label for each user.

17. The method according to claim 16, characterized in that Receiving at least one user classification tag for the displayed image includes: causing a menu to be displayed on the user interface, wherein the menu includes an indication of each user classification tag; and The user classification tag is received by receiving a user selection corresponding to the indication of the user classification tag.

18. The method of any one of claims 6 to 17, further comprising: (o) causing, by the computer system, a user interface to be updated to include: a visual representation of the data points in the second reduced dimensionality dataset as a second icon grouped by the predicted classification label; an image gallery of one or more images associated with the one or more specific data points represented by the one or more selected second icons; a third list including an indication of each predicted classification label; and The second list includes an indication of a classification label for each user.

19. The method of any one of claims 1 to 18, further comprising preparing a labeled training data set for training a third machine learning algorithm to classify images comprising the labeled images and corresponding user classification labels.

20. A method comprising: The computer system causes a user interface to prepare a labeled training data set for training a target machine learning algorithm to classify images to be displayed on a user device, the user interface comprising: a first region comprising a two-dimensional visual representation of a reduced dimension dataset of image data derived from a plurality of images, wherein the two-dimensional visual representation comprises a plurality of icons and indications of clusters in the first reduced dimension dataset, and a second area, the second area including one or more of the plurality of images; and receiving, via the user interface, at the computer system a user input that applies a user classification label to one or more of the images displayed in the second area to generate a first set of tagged images, generating, by the computer system, a second reduced dimension dataset from the labeled training dataset for a second set of images from the plurality of images using the user classification labels of the first set of images using a second machine learning algorithm distinct from the first machine learning algorithm, wherein the second set of images includes unlabeled images; causing, by the computer system, the user interface to update to indicate predicted classification labels for at least some of the unlabeled images in the second image set by the clustering algorithm and the second machine learning algorithm; and At the computer system, a user response is received for the predicted classification labels for one or more unlabeled images.

21. The method of claim 20, wherein: each of the icons representing one or more of the plurality of images and being selectable to cause the represented image to be displayed in the second area; and The one or more of the plurality of images included in the second area are included in response to a user selection of one or more icons.

22. The method of claim 20, wherein: The user interface includes a third area including a list of the clusters in the data set, The one or more of the plurality of images included in the second region are included in response to a user selecting one of the clusters in the list.

23. The method of claim 20, wherein: Preparing the predicted classification label and causing the user interface update are automatically performed in response to receiving user input applying the user classification label.

24. The method according to any one of claims 20 to 23, characterized in that Each second icon in the same given cluster for the predicted classification label is shaded in a common color and positioned close together.

25. The method according to any one of claims 20 to 24, characterized in that The user interface is updated to include a fourth area that includes a list of each predicted classification label.

26. The method according to any one of claims 20 to 25, characterized in that Each second icon is selectable to cause the one or more represented unlabeled images to be displayed in the second area.

27. The method according to any one of claims 25 to 26, characterized in that Each entry of the list of predicted classification labels in the fourth area may be selected to cause one or more unlabeled images having that predicted classification label to be displayed in the second area.

28. The method of any one of claims 20 to 27, further comprising: causing, by the computer system, one or more of the plurality of unlabeled images to be displayed in the second area; as well as User input is received at the computer system through the updated user interface, the user input applying a user classification label to one or more of the unlabeled images displayed in the second area.

29. The method of claim 28, wherein: The user input received through the updated user interface includes: For the one or more displayed unlabeled images, the predicted classification labels are accepted as user input of user classification labels.

30. The method of claim 28, wherein: The user input received through the updated user interface includes: For the one or more displayed unlabeled images, the predicted classification label is rejected as user input of the user classification label.

31. The method of claim 28, wherein: The user input received through the updated user interface includes: User input to exclude one or more displayed unlabeled images from classification.

32. The method according to any one of claims 20 to 31, characterized in that Receiving user input to apply a user classification label to one or more of the images displayed in the image viewer portion includes: causing a menu to be displayed on the user interface, wherein the menu includes an indication of each user classification tag; and The user classification tag is received by receiving a user selection corresponding to the indication of the user classification tag.

33. The method according to any one of claims 20 to 32, characterized in that The image data is derived from the plurality of images by a first machine learning algorithm; wherein the image data derived by the first machine learning algorithm is reduced in dimensionality by a dimensionality reduction algorithm, thereby generating a reduced dimensionality dataset of image data; and The dimension-reduced dataset of image data is clustered by a clustering algorithm.

34. The method according to any one of claims 20 to 33, characterized in that The user interface includes a third area including a list of clusters in the data set including a corresponding indication of a number of images in each cluster.

35. The method according to any one of claims 20 to 34, characterized in that The user interface includes a fifth area including a list of the user classification tags including a corresponding indication of a number of images tagged with each user classification tag.

36. The method according to any one of claims 20 to 35, characterized in that The plurality of images include images of cells.

37. The method of claim 36, wherein: The cell image includes a plurality of multispectral cell images, a plurality of multimodal cell images, or both.

38. The method of any one of claims 20 to 37, further comprising preparing a labeled training data set for training the machine learning algorithm to classify images comprising the labeled images and corresponding user classification labels.

Images