Data diversity visualization and quantification for machine learning models

Abstract

Systems and techniques are provided that facilitate visualization and / or quantification of data diversity with respect to machine learning models. In various implementations, a processor has access to a first dataset and a second dataset, where a machine learning (ML) model is trained on the first dataset. In various cases, the processor can obtain a first set of latent activations generated by the ML model based on the first dataset, and a second set of latent activations generated by the ML model based on the second dataset. In various aspects, via dimension reduction, the processor can generate a first set of compressed data points based on the first set of latent activations, and a second set of compressed data points based on the second set of latent activations. In various cases, a diversity component can compute a diversity score based on the first set of compressed data points and the second set of compressed data points.

Description

[0001] Cross-references to related applications

[0002] This application claims priority to U.S. nonprovisional application serial number 17 / 243,046, filed April 28, 2021, entitled “DATA DIVERSITY VISUALIZATION AND QUANTIFICATION FOR MACHINE LEARNING MODELS,” the entire contents of which are incorporated herein by reference. Technical Field

[0003] This topic discloses, in general, machine learning models, and more specifically, techniques for visualizing and / or quantifying the diversity in data analyzed by machine learning models. Background Technology

[0004] The performance of a machine learning model depends on the data on which it is trained, the data on which it is tested / validated, and / or the data it encounters during field deployment. For example, if a machine learning model is further trained on data significantly different from its original training data, the performance of the trained model can be positively impacted. Conversely, if it is further trained on data not significantly different from its original training data, the performance of the machine learning model may be negatively impacted. Also, when deployed in the field, a trained machine learning model may accurately analyze data substantially similar to its original training data. In contrast, when deployed in the field, a machine learning model may not accurately analyze data substantially dissimilar to its original training data. Therefore, the ability to quickly and / or accurately determine whether two different machine learning datasets are similar to and / or different from each other can be advantageous. Unfortunately, there are no standard tools that facilitate rapid manual and / or automated comparisons of different machine learning datasets.

[0005] Therefore, systems and / or technologies that can solve this technical problem may be desirable. Summary of the Invention

[0006] The following summary is presented to provide a basic understanding of one or more embodiments of the invention. This summary is not intended to identify key or essential elements, nor is it intended to depict any scope of the specific embodiments or any scope of the claims. Its sole purpose is to present concepts in a simplified form as a prelude to the more detailed description that follows. In one or more embodiments described herein, devices, systems, computer-implemented methods, apparatuses, and / or computer program products are described to facilitate the visualization and / or quantification of data diversity for machine learning models.

[0007] According to one or more embodiments, a system is provided. The system may include a computer-readable storage device capable of storing computer-executable components. The system may also include a processor operatively coupled to the computer-readable storage device and capable of executing the computer-executable components stored in the computer-readable storage device. In various embodiments, the computer-executable components may include a receiver component. In various cases, the receiver component has access to a first set of data candidates and a second set of data candidates, on which a machine learning model is trained. In various aspects, the computer-executable components may also include an activation component. In various cases, the activation component can obtain a first set of latent activations generated by the machine learning model based on the first set of data candidates, and can obtain a second set of latent activations generated by the machine learning model based on the second set of data candidates. In various cases, the computer-executable components may also include a compression component. In various cases, the compression component can generate a first set of compressed data points by applying a dimensionality reduction technique to the first set of latent activations, and can generate a second set of compressed data points by applying the dimensionality reduction technique to the second set of latent activations. In various aspects, the computer-executable components may also include a diversity component. In various cases, the diversity component can calculate a diversity score based on the first set of compressed data points and the second set of compressed data points.

[0008] According to one or more implementation schemes, the above system can be implemented as a computer-implemented method and / or computer program product. Attached Figure Description

[0009] This patent or patent application document contains at least one color-drawn drawing. A published copy of this patent or patent application with color drawings will be provided by the Patent Office upon request and payment of the necessary fees.

[0010] Figure 1 A block diagram of an exemplary, non-limiting system for facilitating the visualization and / or quantification of data diversity for machine learning models, according to one or more embodiments described herein, is shown.

[0011] Figure 2 A block diagram of an exemplary, non-limiting system including activation graphs that facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein is shown.

[0012] Figures 3 to 4 Exemplary, non-limiting block diagrams illustrating how various activation graphs can be obtained according to one or more embodiments described herein are shown.

[0013] Figure 5A block diagram of an exemplary, non-limiting system comprising compressed data points that facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein is shown.

[0014] Figure 6 An exemplary, non-limiting block diagram illustrating how various compressed data points can be obtained according to one or more embodiments described herein.

[0015] Figure 7 A block diagram of an exemplary, non-limiting system including diversity scores that facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein is shown.

[0016] Figure 8 A block diagram of an exemplary, non-limiting system including graphs is shown, which facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein.

[0017] Figures 9 to 11 An exemplary, non-limiting diagram of compressed data points according to one or more embodiments described herein is shown.

[0018] Figures 12 to 15 A flowchart is shown of an exemplary, non-limiting computer implementation of a method relating to the quantification of data diversity for a machine learning model, according to one or more embodiments described herein.

[0019] Figure 16 A flowchart is shown of an exemplary, non-limiting computer implementation of a method for facilitating the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein.

[0020] Figure 17 A block diagram is shown illustrating an exemplary, non-limiting operating environment that may facilitate one or more embodiments described herein.

[0021] Figure 18 Exemplary networking environments operable to perform the various specific implementations described herein are shown. Detailed Implementation

[0022] The following specific embodiments are merely exemplary and are not intended to limit the implementation and / or application or use of the embodiments. Furthermore, they are not intended to be construed as being bound by any express or implied information set forth in the foregoing "Background Art" or "Summary of the Invention" or "Detailed Description" sections.

[0023] One or more embodiments will now be described with reference to the accompanying drawings, wherein the same reference numerals are always used to denote the same elements. In the following description, numerous specific details are set forth for illustrative purposes in order to provide a more thorough understanding of one or more embodiments. However, it will be apparent that, in various cases, one or more embodiments may be practiced without these specific details.

[0024] As mentioned above, the performance (e.g., output accuracy, output precision) of a machine learning model (e.g., an artificial neural network) can depend on the data on which it was trained, the data on which it was tested / validated, and / or the data encountered when the model is deployed in the field. For example, further training the machine learning model on data exhibiting patterns / distributions significantly different from those exhibited by the original training data can improve the performance of the trained model. Conversely, further training on data exhibiting patterns / distributions not significantly different from those exhibited by the original training data can negatively impact the model's performance. In other words, additional training on redundant and / or duplicated data can lead to overfitting, which reduces the generalizability of the machine learning model. Similarly, when deployed in the field, a trained machine learning model can accurately analyze data exhibiting patterns / distributions substantially similar to those exhibited by the original training data. In contrast, when deployed in the field, the model may inaccurately analyze data exhibiting patterns / distributions substantially dissimilar to those exhibited by the original training data. In other words, the machine learning model may perform poorly on data different from the data on which it was trained.

[0025] Therefore, the ability to quickly compare patterns / distributions exhibited by two different machine learning datasets (e.g., datasets that can be analyzed by machine learning models and / or otherwise accepted as input) allows model developers to take appropriate actions and / or make decisions. Unfortunately, conventional techniques do not facilitate rapid and / or objective comparisons between patterns / distributions exhibited by two different machine learning datasets.

[0026] For example, suppose a developer decides whether to further train an already trained machine learning model on a particular dataset. As mentioned above, if the patterns / distributions exhibited by the particular dataset are sufficiently different from those exhibited by the dataset on which the machine learning model has already been trained, further training on the particular dataset may be beneficial (e.g., overfitting can be avoided because the two datasets are not redundant and / or duplicated). However, if the patterns / distributions exhibited by the particular dataset are not sufficiently different from those exhibited by the dataset on which the machine learning model has already been trained, further training on the particular dataset may be harmful (e.g., overfitting can occur because the two datasets are redundant and / or duplicated). Conventional techniques do not provide any tools that allow developers to quickly and / or robustly determine whether the patterns / distributions of a particular dataset are sufficiently different from those of the training dataset. Therefore, developers lack a rigorous and / or objective way to determine whether a machine learning model should be further trained on a particular dataset.

[0027] For example, suppose a developer is deciding whether a data augmentation strategy (e.g., modality-based augmentation, biology-based augmentation, mathematical transformation-based augmentation) is appropriate for a machine learning model. In such cases, the developer can create an augmented dataset by applying the augmentation strategy to the original dataset on which the machine learning model has been trained. From a model development perspective, the augmentation strategy is suitable for the machine learning model if the augmented dataset “fills the learning gaps of the machine learning model.” In other words, the augmentation strategy is suitable for the machine learning model if the patterns / distributions exhibited by the augmented dataset are sufficiently different from those exhibited by the original dataset, such that the machine learning model tends to inaccurately analyze the augmented dataset before any training is conducted on it. On the other hand, the augmentation strategy may not be suitable for the machine learning model if the patterns / distributions exhibited by the augmented dataset are not sufficiently different from those exhibited by the original dataset, such that the machine learning model tends to accurately analyze the augmented dataset before any training is conducted on it. Conventional techniques do not provide any tools that allow developers to quickly and / or robustly compare the patterns / distributions of the augmented dataset with those of the original dataset. Therefore, developers lack a rigorous and / or objective way to determine whether an augmentation strategy is appropriate for a machine learning model.

[0028] For example, suppose a developer decides whether to automatically annotate and / or curate a particular dataset via a machine learning model trained on the original dataset, or alternatively, to manually annotate and / or curate the particular dataset. Automatic annotation / curation by a machine learning model may be appropriate if the patterns / distributions exhibited by the particular dataset are sufficiently similar to those exhibited by the original dataset. On the other hand, automatic annotation / curation by a machine learning model may be inappropriate if the patterns / distributions exhibited by the particular dataset are not sufficiently similar to those exhibited by the original dataset, meaning that manual annotation / curation may be necessary. Conventional techniques do not provide any tools that allow developers to quickly and / or robustly determine whether the patterns / distributions of a particular dataset are sufficiently similar to those of the original dataset. Therefore, developers lack a rigorous and / or objective way to determine whether automatic annotation / curation is appropriate.

[0029] Various implementations of this subject-matter innovation can address one or more of these technical problems. One or more implementations described herein include devices, systems, computer-implemented methods, apparatuses, and / or computer program products that facilitate the visualization and / or quantification of data diversity for machine learning models. In various cases, implementations of this subject-matter innovation can be considered as computerized tools (e.g., a combination of computer-executable hardware and / or computer-executable software) that can electronically access a first dataset and a second dataset, on which a machine learning model has been trained, and can electronically quantify, from the perspective of the machine learning model, how different and / or diverse the second dataset is compared to the first dataset. In other words, the computerized tool can electronically calculate a diversity score that indicates how differently the machine learning model tends to interpret the patterns / distributions exhibited by the second dataset compared to how the machine learning model tends to interpret the patterns / distributions exhibited by the first dataset. Furthermore, in various aspects, the computerized tool can electronically present visualizations (e.g., graphs, line graphs) that visually illustrate such differently interpreted patterns / distributions. Therefore, model developers can take any appropriate action and / or make any appropriate decision relative to the machine learning model based on such quantization and / or visualization (e.g., deciding to initiate and / or abandon additional training of the machine learning model on a second dataset based on quantization and / or visualization; deciding to initiate automated and / or manual annotation / curating of the second dataset based on quantization and / or visualization). That is, in various implementations, the computerized tools described herein can be considered as computerized user interfaces that can be utilized by model developers to allow them to make more informed and / or objective design choices relative to the machine learning model.

[0030] In various implementations, such computerized tools may include receiver components, activation components, compression components, diversity components, visualization components, and / or execution components.

[0031] In various implementations, the machine learning model may include any suitable artificial neural network architecture. For example, the machine learning model may include any suitable number of layers, may include any suitable number of neurons in each layer (e.g., different layers may have different numbers of neurons), may implement any suitable activation function (e.g., softmax, hyperbolic tangent, sigmoid), and / or may implement any suitable inter-neuron connection pattern (e.g., forward connection, skip connection, recursive connection).

[0032] In various aspects, a trainable machine learning model (e.g., via supervised training, unsupervised training, and / or reinforcement learning) receives data candidates as input and produces an output based on the data candidates. In various cases, the data candidates can exhibit any suitable dimension and / or format (e.g., they can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof). Similarly, in various cases, the output can exhibit any suitable dimension and / or format (e.g., it can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof). As a non-limiting example, the data candidate can be a medical image of a patient (e.g., an X-ray image of a patient's knee, an MRI image of a patient's brain), and the output can be a label classifying the medical image (e.g., indicating the presence and / or absence of a specific anatomical structure in the medical image).

[0033] In various implementations, a first set of data candidates and a second set of data candidates may exist. In various cases, the first set of data candidates may include any suitable number of data candidates. Similarly, the second set of data candidates may include any suitable number of data candidates (e.g., the same number as and / or a different number from the first set). In various cases, the machine learning model may have already been trained on the first set of data candidates (e.g., via supervised training, unsupervised training, and / or reinforcement learning). In some aspects, the second set of data candidates may represent data candidates obtained from a different source than the first set, and the developer associated with the machine learning model may wish to determine whether the machine learning model can be expected to accurately analyze data candidates from that different source and / or whether the machine learning model should be further trained (e.g., via backpropagation) on data candidates from that different source. In other aspects, the second set of data candidates may have been created by applying reinforcement strategies to the first set of data candidates, and the developer may wish to determine whether such reinforcement strategies are suitable for improving the generality of the machine learning model. In still other aspects, the second set of data candidates may be unannotated, and the developer may wish to determine whether automatic annotation techniques can be accurately applied to the second set of data candidates. As described in this article, computerized tools can analyze the first set of data candidates and the second set of data candidates, and such analysis can be used by developers to make such determinations objectively and / or rigorously.

[0034] In various implementations, the receiver component of the computerized tool can electronically receive and / or otherwise electronically access the machine learning model, the first set of data candidates, and / or the second set of data candidates. In various cases, the receiver component can electronically retrieve the machine learning model, the first set of data candidates, and / or the second set of data candidates from any suitable centralized and / or decentralized data structure (e.g., graph data structure, relational data structure, hybrid data structure) (whether remote from the receiver component and / or local to the receiver component). Therefore, other components of the computerized tool described herein can electronically interact with the machine learning model, the first set of data candidates, and / or the second set of data candidates.

[0035] In various implementations, the activation component of the computerized tool can electronically acquire a first set of latent activations and a second set of latent activations. In various aspects, the first set of latent activations can be generated by a machine learning model based on a first set of data candidates. Similarly, in various cases, the second set of latent activations can be generated by a machine learning model based on a second set of data candidates. For example, in various aspects, the activation component can electronically feed a specific data candidate from the first set of data candidates as input to the machine learning model. In various cases, the hidden layer of the machine learning model can generate a specific latent activation map based on the specific data candidate. Thus, a latent activation map can be obtained for each data candidate in the first set of data candidates, resulting in the first set of latent activations. Similarly, in various aspects, the activation component can electronically feed a certain data candidate from the second set of data candidates as input to the machine learning model. In various cases, the hidden layer of the machine learning model can generate a certain latent activation map based on the specific data candidate. Thus, a latent activation map can be obtained for each data candidate in the second set of data candidates, resulting in the second set of latent activations.

[0036] In various implementations, the compression component of the computerized tool can electronically generate a first set of compressed data points based on a first set of latent activations. Similarly, in various cases, the compression component can electronically generate a second set of compressed data points based on a second set of latent activations. In various cases, the compression component can generate such sets of compressed data points via the application of any suitable dimensionality reduction technique, such as t-distributed random neighbor embedding (t-SNE) and / or uniform manifold approximation and projection (UMAP)). For example, in various aspects, the compression component can electronically apply a dimensionality reduction technique to the first set of latent activations, thereby transforming and / or converting the first set of latent activations into the first set of compressed data points. A given latent activation graph in the first set of latent activations can correspond to a given compressed data point in the first set of compressed data points, such that a given compressed data point represents the given latent activation graph but includes fewer dimensions (e.g., fewer numeric elements) than the given latent activation graph. Similarly, in various aspects, the compression component can electronically apply a dimensionality reduction technique to the second set of latent activations, thereby transforming and / or converting the second set of latent activations into the second set of compressed data points. A given latent activation graph in the second set of latent activations may correspond to a given compressed data point in the second set of compressed data points, such that the given compressed data point represents a given latent activation graph but includes fewer dimensions (e.g., fewer number elements) than the given latent activation graph.

[0037] As described above, in various implementations, data candidates can have any suitable dimension (e.g., one or more scalars, one or more vectors, one or more matrices, one or more tensors, one or more strings, and / or any suitable combination thereof). Similarly, in various cases, the latent activation graph can have any suitable dimension depending on the configuration and / or structure of the machine learning model. In various aspects, the compressed data points can have any suitable dimension smaller than the dimension of the latent activation graph. As a non-limiting example, the latent activation graph can be a matrix of hundreds and / or thousands of activation values ​​output by the hidden layer of a machine learning model, and the compressed data points can be tuples of two elements (e.g., two-element vectors) representing the corresponding latent activation graph. As another non-limiting example, the latent activation graph can be a matrix of hundreds and / or thousands of activation values ​​output by the hidden layer of a machine learning model, and the compressed data points can be tuples of fifty elements (e.g., fifty-element vectors) representing the corresponding latent activation graph. More generally, for any suitable positive integer n, the compressed data points can be tuples of n elements (e.g., n-element vectors), where n is less than the number of elements in the latent activation graph. Because compressed data points can have lower dimensions than the latent activation graph, computations are easier to perform on compressed data points than on the latent activation graph.

[0038] In various aspects, because a machine learning model can be trained on the first set of data candidates, the model can accurately analyze the patterns / distributions exhibited by the first set. This accurate analysis of the patterns / distributions exhibited by the first set of data candidates ultimately allows the first set of compressed data points to be clustered according to the output of the machine learning model, which is configured / trained to produce. For example, suppose the machine learning model is configured / trained to produce binary labels as output, indicating that the input data candidate is a member of one of two categories. In this case, the first set of compressed data points can form two clusters; one cluster of compressed data points represents data candidates that the machine learning model infers belong to the first category, and the other cluster represents data candidates that the machine learning model infers belong to the second category. More generally, for any suitable positive integer m, if the machine learning model is configured / trained to classify data candidates into one of m categories, the first set of compressed data points can be arranged into m clusters. In various cases, clusters of compressed data points can be identified as multiple compressed data points with similar elements (e.g., separated by small and / or short Euclidean distances).

[0039] In various implementations, the diversity component of the computerized tool can electronically calculate a diversity score based on a first set of compressed data points and / or a second set of compressed data points. In other words, the diversity component can implement any suitable statistical techniques and / or statistical calculations (e.g., mean, median, mode, standard deviation, variance) to mathematically quantify the statistical difference between the first and second sets of compressed data points. In various cases, the diversity score can be a scalar (and / or, in other cases, a vector, matrix, and / or tensor) indicating such a statistical difference between the first and second sets of compressed data points.

[0040] In various cases, the first set of latent activations can be considered as an interpretation of the pattern / distribution of the first set of data candidates by the machine learning model, and the first set of compressed data points can be considered as a dimensionality-reduced version of the first set of latent activations. Therefore, the first set of compressed data points can be simplified and thus computed in a conforming format to represent how the machine learning model interprets the pattern / distribution exhibited by the first set of data candidates. Similarly, in various cases, the second set of latent activations can be considered as an interpretation of the pattern / distribution of the second set of data candidates by the machine learning model, and the second set of compressed data points can be considered as a dimensionality-reduced version of the second set of latent activations. Therefore, the second set of compressed data points can be simplified and thus computed in a conforming format to represent how the machine learning model interprets the pattern / distribution exhibited by the second set of data candidates. Therefore, since the first set of compressed data points and the second set of compressed data points can be derived from the first set of data candidates and the second set of data candidates respectively, from the perspective of the machine learning model, the diversity score can therefore indicate the statistical difference between the pattern / distribution exhibited by the first set of data candidates and the pattern / distribution exhibited by the second set of data candidates.

[0041] In each aspect, the diversity score can indicate whether the second set of compressed data points fits and / or otherwise conforms to the cluster of the first set of compressed data points. If the second set of compressed data points fits and / or conforms to the cluster of the first set of compressed data points, the diversity score can be small in magnitude (e.g., below any suitable predetermined threshold). This can indicate that the machine learning model has not detected significant differences between the patterns / distributions exhibited by the first set of data candidates and those patterns / distributions exhibited by the second set of data candidates (e.g., the machine learning model may be unaware of the differences between the first and second sets of data candidates). On the other hand, if the second set of compressed data points does not fit and / or conform to the cluster of the first set of compressed data points, the diversity score can be large in magnitude (e.g., above any suitable predetermined threshold). This can indicate that the machine learning model has detected significant differences between the patterns / distributions exhibited by the first set of data candidates and those patterns / distributions exhibited by the second set of data candidates (e.g., the machine learning model may not be unaware of the differences between the first and second sets of data candidates).

[0042] In all aspects, the diversity component can compute and / or account for diversity scores in any suitable manner (e.g., via any suitable mathematical and / or statistical operations and / or functions). For example, in various cases, the diversity component can compute the center of each cluster for the first set of compressed data points. That is, if each compressed data point is an n-element tuple, the diversity component can compute the center n-element tuple for each cluster for the first set of compressed data points. In various cases, the center n-element tuple of a given cluster can be equal to the average of all n-element tuples in the first set of compressed data points that belong to that cluster (e.g., it is known which cluster each of the first set of compressed data points belongs to, because the first set of data candidates can be a training dataset with a known benchmark true classification, and because a machine learning model can be trained on the first set of data candidates).

[0043] In various cases, for each cluster in the first set of compressed data points, the diversity component can then calculate the Euclidean distance between the center of that cluster and every compressed data point in the first set of compressed data points that belongs to that cluster. In various cases, for each cluster in the first set of compressed data points, the diversity component can therefore calculate the mean and / or standard deviation of such calculated Euclidean distances. For each cluster in the first set of compressed data points, the result can be the mean Euclidean distance value, conveniently denoted as μ, and the standard deviation Euclidean distance value, conveniently denoted as σ. The μ and σ of a given cluster can be considered as statistically describing the density and / or distribution of that given cluster (e.g., μ1 and σ1 can describe the density and / or distribution of cluster 1 in the first set of compressed data points; μ2 and σ2 can describe the density and / or distribution of cluster 2 in the first set of compressed data points).

[0044] In various cases, the diversity component can assign each of the second set of compressed data points to the corresponding cluster of the first set of compressed data points. More specifically, in various cases, if a given compressed data point is closer in Euclidean distance to the center of a given cluster than to the center of any other cluster of the first set of compressed data points, the diversity component can assign a given compressed data point from the second set of compressed data points to the given cluster of the first set of compressed data points. In other words, the diversity component can assign each of the second set of compressed data points to the nearest cluster of the first set of compressed data points. Therefore, for each cluster, the diversity component can calculate the Euclidean distance between the center of the cluster and each compressed data point in the second set of compressed data points that has already been assigned to that cluster.

[0045] In various cases, for each cluster, the diversity component can calculate the percentage and / or proportion of a second set of compressed data points already assigned to that cluster and within and / or outside any suitable threshold Euclidean distance from the center of that cluster. As a non-limiting example, the threshold Euclidean distance for a given cluster can be equal to μ + 2σ (e.g., μ1 + 2σ1 could be the threshold Euclidean distance for cluster 1; μ2 + 2σ2 could be the threshold Euclidean distance for cluster 2). In various cases, if a compressed data point in the second set of compressed data points is within such a threshold Euclidean distance from the center of its assigned cluster, then that compressed data point can be considered a non-outlier. On the other hand, if a compressed data point in the second set of compressed data points is farther than such a threshold Euclidean distance from the center of its assigned cluster, then that compressed data point can be considered an outlier.

[0046] In various aspects, the diversity component can determine the total percentage and / or proportion of the second set of compressed data points considered outliers. In various cases, the diversity score can be any suitable mathematical function of such a total percentage and / or proportion of the second set of compressed data points considered outliers. In some cases, the diversity score can be equal to such a total percentage and / or proportion of the second set of compressed data points considered outliers. In any case, the diversity score can numerically represent how well and / or poorly the second set of compressed data points fits the cluster of the first set of compressed data points. Correspondingly, the diversity score can therefore represent how differently the machine learning model interprets the pattern / distribution exhibited by the second set of data candidates compared to how the machine learning model interprets the pattern / distribution exhibited by the first set of data candidates.

[0047] Therefore, in various cases, a high diversity score may indicate that the majority of the second set of compressed data points are considered outliers (e.g., most of the second set of compressed data points does not neatly fit the clusters of the first set of compressed data points), which could mean that the machine learning model interprets the patterns / distributions of the second set of data candidates differently than those of the first set of data candidates. On the other hand, in various cases, a low diversity score may indicate that a small portion of the second set of compressed data points are considered outliers (e.g., only a small portion of the second set of compressed data points does not neatly fit the clusters of the first set of compressed data points), which could mean that the machine learning model interprets the patterns / distributions of the second set of data candidates similarly to those of the first set of data candidates.

[0048] Those skilled in the art will understand that any other suitable mathematical and / or statistical techniques can be implemented by the diversity component to calculate the diversity score. Furthermore, although the disclosure herein primarily discusses diversity scores, where higher values ​​indicate more diversity and lower values ​​indicate less diversity, those skilled in the art will understand that this is merely a non-limiting example. In various other embodiments, the diversity component may alternatively calculate a similarity score, where higher values ​​indicate more similarity (e.g., less diversity) and lower values ​​indicate less similarity (e.g., more diversity). In any case, the diversity component may quantify and / or measure how well and / or poorly the second set of compressed data points fits and / or conforms to the clusters of the first set of compressed data points, which may correspondingly indicate how similar and / or how different the patterns / distributions exhibited by the second set of data candidates are compared to those exhibited by the first set of data candidates.

[0049] In various implementations, if the compression component implements dimensionality reduction techniques that produce two-dimensional or three-dimensional compressed data points (e.g., n=2 and / or n=3), the visualization component of the computerized tool can electronically present and / or plot graphs depicting and / or illustrating the first set of compressed data points and / or the second set of compressed data points on any suitable computer screen / monitor. In various cases, clusters of the first set of compressed data points can be visually perceived in such graphs. Furthermore, the degree to which the second set of compressed data points fits and / or conforms to clusters of the first set of compressed data points can also be visually perceived in such graphs. In some cases, the visualization component may also present diversity scores on and / or near the graph.

[0050] In various scenarios, the presentation of the visualization components can be viewed and / or is viewable by the developer associated with the machine learning model (e.g., graphs and / or diversity scores can be presented on a computer screen / monitor viewable by the developer). Thus, the developer can be informed of / notified of the diversity scores and / or visually see the graph, allowing them to understand how the second set of compressed data points differs from the first set, and therefore how the patterns / distributions exhibited by the second set of data candidates differ from those exhibited by the first set. In various situations, when making design choices / decision-making regarding the machine learning model (e.g., when determining whether to further train the machine learning model on the second set of data candidates, when determining whether to automatically annotate the second set of data candidates), the developer can refer to the diversity scores and / or graphs.

[0051] In some cases, the graph can be interactive and / or clickable. That is, developers can interact with the graph via any suitable human-computer interface device (e.g., computer mouse, touchscreen, voice commands). In various cases, developers can click on compressed data points plotted on the graph, and visualization components can respond to such clicks by presenting data candidates from a first set of data candidates and / or a second set of data candidates corresponding to the clicked compressed data point. Therefore, developers can manually examine any suitable data candidate as needed by clicking on the corresponding compressed data point of the data candidate.

[0052] In various implementations, the execution component of the computerized tool can electronically take any suitable computerized action and / or make any suitable recommendation electronically based on a diversity score. For example, if a developer wants to determine whether to further train a machine learning model on a second set of data candidates, the execution component can compare the diversity score to any suitable pre-determined threshold. If the diversity score exceeds the pre-determined threshold, the execution component can determine that the patterns / distributions of the second set of data candidates are sufficiently different from those of the first set of data candidates, thus avoiding overfitting. Therefore, the execution component can recommend further training of the machine learning model on the second set of data candidates. On the other hand, if the diversity score does not exceed the pre-determined threshold, the execution component can determine that the patterns / distributions of the second set of data candidates are not sufficiently different from those of the first set of data candidates, thus potentially leading to overfitting. Therefore, the execution component can recommend not to further train the machine learning model on the second set of data candidates.

[0053] For example, if a developer wants to determine whether to deploy a machine learning model in an environment associated with a second set of data candidates, the execution component can compare the diversity score to any suitable pre-defined threshold. If the diversity score exceeds the pre-defined threshold, the execution component can determine that the patterns / distributions of the second set of data candidates are not sufficiently similar to those of the first set of data candidates, resulting in inaccurate performance expectations. Therefore, the execution component can recommend not deploying the machine learning model in the environment associated with the second set of data candidates. On the other hand, if the diversity score does not exceed the pre-defined threshold, the execution component can determine that the patterns / distributions of the second set of data candidates are sufficiently similar to those of the first set of data candidates, resulting in accurate performance expectations. Therefore, the execution component can recommend deploying the machine learning model in the environment associated with the second set of data candidates.

[0054] For example, if a second set of data candidates is created via an augmentation strategy, and if the developer wants to determine whether the augmentation strategy is suitable for the machine learning model, the execution component can compare the diversity score with any suitable pre-determined threshold. If the diversity score exceeds the pre-determined threshold, the execution component can determine that the machine learning model considers the pattern / distribution of the second set of data candidates to be sufficiently different from the pattern / distribution of the first set of data candidates, making the augmentation strategy effective. Therefore, the execution component can recommend that the augmentation strategy is suitable for the machine learning model. On the other hand, if the diversity score does not exceed the pre-determined threshold, the execution component can determine that the machine learning model considers the pattern / distribution exhibited by the second set of data candidates not sufficiently different from the pattern / distribution exhibited by the first set of data candidates, making the augmentation strategy ineffective. Therefore, the execution component can recommend that the augmentation strategy is not suitable for the machine learning model.

[0055] For example, if the second set of data candidates is unannotated, and if the developer wants to determine whether automatic annotation techniques are suitable for the second set of data candidates, the execution unit can compare the diversity score with any suitable pre-determined threshold. If the diversity score exceeds the pre-determined threshold, the execution unit can determine that the machine learning model considers the patterns / distributions of the second set of data candidates to be insufficiently similar to those patterns / distributions of the first set of data candidates, such that automatic annotation will not produce accurate labels. Therefore, the execution unit can recommend applying manual annotation to the second set of data candidates. On the other hand, if the diversity score does not exceed the pre-determined threshold, the execution unit can determine that the machine learning model considers the patterns / distributions of the second set of data candidates to be sufficiently similar to those patterns / distributions of the first set of data candidates, such that automatic annotation will produce accurate labels. Therefore, the execution unit can recommend applying automatic annotation to the second set of data candidates. In some implementations, the execution unit can recommend that only data candidates corresponding to outliers of the second set of compressed data points be manually annotated, and / or can recommend that only data candidates corresponding to non-outliers of the second set of compressed data points be automatically annotated.

[0056] Various innovative implementations of this topic can be used to solve inherently highly technical problems (e.g., to facilitate the visualization and / or quantification of data diversity for machine learning models) using hardware and / or software. These problems are not abstract and cannot be performed as a set of human mental behaviors. Furthermore, some processes within the execution can be performed by a dedicated computer (e.g., a trained machine learning model) specifically designed to perform the defined tasks related to the visualization and / or quantification of data diversity. For example, such defined tasks may include: accessing a first set of data candidates and a second set of data candidates by a device operatively coupled to a processor, wherein a machine learning model is trained on the first set of data candidates; obtaining a first set of latent activations generated by the machine learning model based on the first set of data candidates by the device, and obtaining a second set of latent activations generated by the machine learning model based on the second set of data candidates by the device; generating a first set of compressed data points by the device by applying a dimensionality reduction technique to the first set of latent activations, and generating a second set of compressed data points by the device by applying the dimensionality reduction technique to the second set of latent activations; calculating a diversity score by the device based on the first set of compressed data points and the second set of compressed data points; and / or presenting a graph depicting the first set of compressed data points and the second set of compressed data points by the device. Such defined tasks are generally not performed manually by humans. Furthermore, neither the human mind nor humans with pen and paper can electronically access machine learning models, the first set of data candidates on which to train machine learning models, and the second set of data candidates; the first and second sets of data candidates can be electronically fed into machine learning models to obtain the first and second sets of latent activations; dimensionality reduction can be electronically applied to the first and second sets of latent activations to produce the first and second sets of compressed data points; diversity scores quantifying the differences between the first and second sets of compressed data points can be electronically computed; and / or graphs depicting the first and second sets of compressed data points can be electronically presented. Alternatively, various embodiments of the present invention are inherently and inextricably linked to computer technology and cannot be implemented outside of a computing environment (e.g., embodiments of the present invention constitute a computerized user interface that can automatically quantify and / or visualize the differences between two different machine learning datasets; such computerized tools cannot be implemented in any reasonable way without a computer).

[0057] Furthermore, various innovative implementations of this topic can integrate publicly available teachings on the visualization and / or quantification of data diversity for machine learning models into practical applications. As mentioned above, model developers lack conventional tools that allow them to make various design decisions quickly and / or objectively when developing machine learning models. For example, a model developer might want to determine in advance whether further training on a particular dataset will improve the generality of a particular machine learning model or, alternatively, cause it to overfit. If the model developer is unaware of this information beforehand, there is a risk of performing training and resulting in overfitting, which constitutes a waste of time and / or resources. Similarly, a model developer might want to determine in advance whether an augmentation strategy is suitable for a particular machine learning model. If the model developer is unaware of this information beforehand, there is a risk of applying an augmentation strategy that is unsuitable for a particular machine learning model, which constitutes a waste of time and / or resources. Furthermore, a model developer might want to determine in advance whether a particular unannotated dataset can be accurately labeled via automatic annotation techniques (e.g., transfer learning based on distance to known samples), or whether that particular dataset should instead be manually annotated. If model developers are unaware of this information beforehand, there is a risk that manual annotations may be performed unnecessarily or automatic annotations may be performed regardless of a high level of inaccuracy, both of which constitute a waste of time and / or resources. Conventional tools do not assist model developers in making these determinations, and therefore cannot help them avoid such risks of wasting time and / or resources.

[0058] In stark contrast, the computational tools described herein assist model developers in making such determinations, and thus help avoid the risk of wasting time and / or resources. Specifically, the computational tools described herein can electronically access a first dataset and a second dataset, on which a machine learning model is trained. Furthermore, the computational tools can electronically obtain the first and second sets of latent activations by feeding the first and second datasets to the machine learning model and by extracting the first and second sets of latent activations from the hidden layers of the machine learning model, respectively. In various cases, the computational tools can electronically compress the first and second sets of latent activations into a first set of compressed data points and a second set of compressed data points via any suitable dimensionality reduction technique (e.g., t-SNE and / or UMAP). In various cases, the computational tools can electronically compute a diversity score that quantifies and / or measures how tightly the second set of compressed data points conforms to the clusters exhibited by the first set of compressed data points. A high diversity score indicates that the second set of compressed data points does not tightly conform to the clusters of the first set of compressed data points, which may mean that the machine learning model interprets the second dataset in a different way than the first dataset. On the other hand, a low diversity score can indicate that the second set of compressed data points closely matches the clustering of the first set of compressed data points, which could mean that the machine learning model interprets the second dataset similarly to the first dataset. In various cases, computational tools can visually represent the diversity score and / or visually represent the first and second compressed data points on a graph visible to the model developer. Therefore, the model developer can be informed of the diversity score and / or can manually view the graph. Thus, the model developer can determine whether the first and second datasets are sufficiently similar and / or sufficiently different, and can make corresponding design choices / decisions regarding the machine learning model (e.g., if the diversity score is high enough, the machine learning model can be further trained on the second dataset without the risk of overfitting; if the diversity score is low enough, automatic annotation techniques can be applied to the second dataset). Such computational tools clearly constitute a useful and practical application of computing.

[0059] Furthermore, various implementations of this invention can control real-world tangible devices based on the disclosed teachings. For example, various implementations of this invention can electronically access machine learning models, which can be any suitable combination of tangible computing hardware and / or software. In various cases, implementations of this invention can electronically train and / or recommend training machine learning models on a given dataset. Moreover, in various aspects, implementations of this invention can electronically present various visualizations and / or graphs on any suitable computer screen.

[0060] It should be understood that the accompanying figures and descriptions provide non-limiting examples of the innovations in this subject matter.

[0061] Figure 1 A block diagram of an exemplary, non-limiting system 100, according to one or more embodiments described herein, is shown to facilitate the visualization and / or quantification of data diversity for machine learning models. As shown, the data diversity system 102 can be electronically integrated with a machine learning model 104, a set of training data candidates 106, and / or a set of test data candidates 108 via any suitable wired and / or wireless electronic connection.

[0062] In various cases, the machine learning model 104 can be an artificial neural network that exhibits any suitable artificial neural network architecture. For example, the machine learning model 104 may include any suitable number of layers. In various cases, the machine learning model 104 may include any suitable number of neurons in various layers (e.g., different layers may have different numbers of neurons from each other, and different layers may have the same number of neurons from each other). In various aspects, the various neurons of the machine learning model 104 can implement any suitable activation function (e.g., sigmoid, softmax, hyperbolic tangent, rectified linear unit). In various cases, the machine learning model 104 can implement any suitable connection pattern (e.g., forward connections between neurons, skip connections between neurons, recursive connections between neurons). Although the disclosure herein primarily discusses implementations in which the machine learning model 104 is an artificial neural network, this is merely a non-limiting example. In various implementations, the machine learning model 104 can implement any other suitable machine learning architecture.

[0063] In all aspects, the machine learning model 104 can be configured in any suitable manner to receive data candidates as input and to produce a certain output based on those data candidates. In various cases, the data candidates can have any suitable dimension (e.g., they can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, and / or one or more strings). In various cases, the input can have any suitable dimension (e.g., it can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, and / or one or more strings). For example, the machine learning model 104 can be configured to receive medical images (e.g., X-ray images, MRI images, CT images) as input and produce a label (e.g., a label indicating the anatomical structure depicted in the medical image) as output that classifies the medical image into one of two or more distinct categories.

[0064] In various cases, the set of training data candidates 106 may include any suitable number of training data candidates. In various cases, the training data candidates may be data candidates on which the machine learning model 104 has been trained and / or already trained. For example, the intrinsic parameters of the machine learning model 104 (e.g., weight matrix, bias values) may have been iteratively updated via backpropagation based on the set of training data candidates 106 and / or based on benchmark ground truth annotations corresponding to the set of training data candidates 106. In some cases, the set of training data candidates 106 may be considered as samples and / or subsets obtained from even larger sets of data candidates (not shown) on which the machine learning model 104 has been trained and / or already trained.

[0065] In various aspects, the set of test data candidates 108 may include any suitable number of test data candidates. In various cases, test data candidates may be data candidates on which the machine learning model 104 has not been trained and / or has not yet been trained. In various cases, the operator (not shown) associated with the machine learning model 104 may wish to make design choices and / or design decisions regarding the set of test data candidates 108. For example, in some cases, the set of test data candidates 108 may originate from and / or be sampled from a specific domain and / or environment, and the operator may wish to determine whether the machine learning model 104 can be accurately deployed in that specific domain and / or environment. As another example, the set of test data candidates 108 may be considered potential training data; that is, the operator may wish to determine whether the machine learning model 104 can be trained on the set of test data candidates 108 without overfitting. As yet another example, the set of test data candidates 108 may be created by applying a data augmentation strategy to the set of training data candidates 106, and the operator may wish to determine whether the data augmentation strategy helps generalize the machine learning model 104. For example, the set of test data candidates 108 may be unannotated, and the operator may wish to determine whether the set of test data candidates 108 can be accurately and automatically annotated. As described herein, the data diversity system 102 may electronically generate results and / or visualizations based on the set of training data candidates 106 and / or the set of test data candidates 108, and such results and / or visualizations may be used by the operator to make such determinations quickly and / or objectively.

[0066] In various embodiments, the data diversity system 102 may include a processor 110 (e.g., a computer processing unit, microprocessor) and a computer-readable storage device 112 operatively and / or communicatively connected / coupled to the processor 110. The computer-readable storage device 112 may store computer-executable instructions that, when executed by the processor 110, cause the processor 110 and / or other components of the data diversity system 102 (e.g., receiver component 114, activation component 116, compression component 118, diversity component 120, visualization component 122, and / or execution component 124) to perform one or more actions. In various embodiments, the computer-readable storage device 112 may store computer-executable components (e.g., receiver component 114, activation component 116, compression component 118, diversity component 120, visualization component 122, and / or execution component 124), and the processor 110 may execute the computer-executable components.

[0067] In various embodiments, the data diversity system 102 may include a receiver component 114. In various aspects, the receiver component 114 may electronically retrieve and / or otherwise electronically access the machine learning model 104, the set of training data candidates 106, and / or the set of test data candidates 108 from any suitable centralized and / or distributed data structure (not shown) (whether remote from the receiver component 114 and / or local to the receiver component). Therefore, in various cases, other components of the data diversity system 102 may manipulate the machine learning model 104, the set of training data candidates 106 and / or the set of test data candidates 108, and / or otherwise interact with the machine learning model, the set of training data candidates, and / or the set of test data candidates.

[0068] In various implementations, the data diversity system 102 may include an activation component 116. In various aspects, the activation component 116 may electronically obtain a set of training activation maps based on the set of training data candidates 106, and / or may electronically obtain a set of test activation maps based on the set of test data candidates 108. More specifically, in various cases, the activation component 116 may electronically feed training data candidates (e.g., one of the set of training data candidates 106) to the machine learning model 104, and the activation component 116 may electronically extract activation maps (e.g., hence referred to as "training activation maps") generated by the hidden layers of the machine learning model 104 based on the training data candidates. The activation component 116 may repeat this operation for each (and / or fewer than each) of the set of training data candidates 106, thereby generating the set of training activation maps. Similarly, in various cases, activation component 116 may electronically feed test data candidates (e.g., one of the set of test data candidates 108) to machine learning model 104, and activation component 116 may electronically extract activation maps (e.g., hence referred to as "test activation maps") generated by the hidden layers of machine learning model 104 based on the test data candidates. Activation component 116 may repeat this operation for each (and / or fewer than each) of the set of test data candidates 108, thereby generating the set of test activation maps. In various cases, this set of training activation maps may be considered to show how machine learning model 104 internally interprets the set of training data candidates 106. Similarly, this set of test activation maps may be considered to show how machine learning model 104 internally interprets the set of test data candidates 108.

[0069] In various embodiments, the data diversity system 102 may include a compression component 118. In various aspects, the compression component 118 may electronically reduce the dimensionality of the set of training activation maps and / or the set of test activation maps. More specifically, in various cases, the compression component 118 may electronically generate a set of compressed training data points by performing any suitable dimensionality reduction technique on the set of training activation maps. That is, the compression component 118 may transform training activation maps into compressed training data points, wherein the compressed training data points may be vectors with lower dimensions (e.g., fewer elements) than the training activation maps. The compression component 118 may repeat this operation for each (and / or fewer than each) of the training activation maps in the set, thereby producing the set of compressed training data points. Similarly, in various cases, the compression component 118 may electronically generate the set of compressed test data points by performing a dimensionality reduction technique on the set of test activation maps. That is, compression component 118 can convert a test activation map into compressed test data points, wherein the compressed test data points can be vectors with lower dimensions (e.g., fewer elements) than the test activation map. Compression component 118 can repeat this operation for each (and / or fewer) test activation map in the set of test activation maps to produce the set of compressed test data points.

[0070] In various cases, the compressed set of training data points can be considered to represent and / or summarize how machine learning model 104 interprets the patterns / distributions contained in the set of training data candidates 106 in a reduced feature space. Similarly, in various aspects, the compressed set of test data points can be considered to represent and / or summarize how machine learning model 104 interprets the patterns / distributions contained in the set of test data candidates 108 in a reduced feature space.

[0071] In various embodiments, the data diversity system 102 may include a diversity component 120. In various aspects, the diversity component 120 may electronically calculate a diversity score based on the set of compressed training data points and / or the set of compressed test data points. More specifically, the diversity component 120 may apply any suitable mathematical and / or statistical calculations to statistically compare the set of compressed training data points with the set of compressed test data points.

[0072] As described above, because the machine learning model 104 can be trained on this set of training data candidates 106, and because the set of compressed training data points can be derived from this set of training data candidates 106, the set of compressed training data points can exhibit a clustered arrangement that corresponds to the output that the machine learning model 104 is configured to produce. For example, for any suitable positive integer m, if the machine learning model 104 is configured to produce labels that classify the input data candidates into one of m distinct categories, then the set of compressed training data points can have m distinct clusters (e.g., m distinct subsets of compressed training data points of similar value).

[0073] In various scenarios, the diversity component 120 can calculate a statistical measure (e.g., mean, median, mode, standard deviation) defining the cluster arrangement of the compressed training data points. In each aspect, the diversity component 120 can then determine whether the compressed test data points conform to the statistical measure defining the cluster arrangement of the compressed training data points. If the compressed test data points do conform to such a statistical measure, the diversity component 120 can output a low diversity score, indicating the level of low diversity between the compressed training data points and the compressed test data points. This low diversity score correspondingly indicates how the machine learning model 104 interprets the low level of difference between the patterns / distributions exhibited by the training data candidate 106 and how the machine learning model 104 interprets the patterns / distributions exhibited by the test data candidate 108. In contrast, if the compressed test data points do not conform to such statistical measures, the diversity component 120 can output a high diversity score, which indicates the high level of diversity between the compressed training data points and the compressed test data points. This high diversity score correspondingly indicates how the machine learning model 104 interprets the high level of difference between the patterns / distributions exhibited by the training data candidates 106 and how the machine learning model 104 interprets the patterns / distributions exhibited by the test data candidates 108.

[0074] In various embodiments, the data diversity system 102 may include a visualization component 122. In various aspects, if the compressed data points are binary and / or ternary vectors, the visualization component 122 may electronically present a graph depicting the set of compressed training data points and / or the set of compressed test data points on any suitable computer monitor / screen (not shown) viewable by an operator. In various cases, the visualization component 122 may also electronically present diversity scores on the computer monitor / screen. Thus, the operator is visually informed of the diversity scores and / or the graph. In some cases, the graph may be interactive and / or clickable (e.g., via a computer mouse and / or any other suitable human-machine interface device). In various cases, when an operator clicks on a given compressed data point plotted in the graph, the visualization component may electronically present data candidates corresponding to the clicked compressed data point on the computer monitor / screen, allowing the operator to quickly and manually examine data candidates as needed. In any case, when making various design decisions regarding the set of test data candidates 108 and / or machine learning models 104, the operator can therefore utilize diversity scores and / or graphs.

[0075] In various embodiments, the data diversity system 102 may include an execution component 124. In various aspects, the execution component 124 may electronically transfer diversity scores and / or graphs to any other suitable computing device (not shown). In various cases, the execution component 124 may make various electronic recommendations to an operator based on the diversity scores. For example, if an operator wishes to determine whether a machine learning model 104 can be trained on the set of test data candidates 108 without overfitting, the execution component 124 may compare the diversity scores to any suitable threshold and make a recommendation accordingly. Similarly, if an operator wishes to determine whether the machine learning model 104 can be accurately deployed in the domain and / or environment associated with the set of test data candidates 108, the execution component 124 may compare the diversity scores to any suitable threshold and make a recommendation accordingly. Furthermore, if the set of test data candidates 108 is created by modifying the set of training data candidates 106 via an enhancement strategy, and if an operator wishes to determine whether such enhancement strategy helps improve the generality of the machine learning model 104, the execution component 124 may compare the diversity scores to any suitable threshold and make a recommendation accordingly. For example, if the test data candidate 108 is unannotated, and if the operator wants to determine whether automatic annotation can be accurately applied to the test data candidate 108, the execution unit 124 can compare the diversity score with any suitable threshold and make a recommendation accordingly.

[0076] Figure 2A block diagram of an exemplary, non-limiting system 200 including activation graphs, which facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein, is shown. As shown, in some cases, system 200 may include the same components as system 100, and may also include a set of training activation graphs 202 and / or a set of test activation graphs 204.

[0077] In various implementations, activation component 116 may electronically obtain the set of training activation maps 202 based on the set of training data candidates 106, and / or may electronically obtain the set of test activation maps 204 based on the set of test data candidates 108. More specifically, in various cases, activation component 116 may electronically feed each of the set of training data candidates 106 to machine learning model 104 and may electronically extract activation values ​​generated by the hidden layers of machine learning model 104, thereby producing the set of training activation maps 202. Similarly, in various aspects, activation component 116 may electronically feed each of the set of test data candidates 108 to machine learning model 104 and may electronically extract activation values ​​generated by the hidden layers of machine learning model 104, thereby producing the set of test activation maps 204. Relative to Figures 3 to 4 This was explained in more detail.

[0078] Figures 3 to 4 Exemplary, non-limiting block diagrams illustrating how various activation graphs can be obtained according to one or more embodiments described herein are shown.

[0079] like Figure 3 As shown, in various embodiments, the machine learning model 104 may include a set of layers 302. In various cases, this set of layers 302 may include p layers, for any suitable positive integer p; that is, layers 1 to p. In various aspects, layer 1 may be considered the input layer, and layer p may be considered the output layer. In various cases, intermediate and / or hidden layers in this set of layers 302 may be represented as intermediate and / or hidden layers j, where j may be any suitable positive integer greater than 1 and less than p. Any layer other than layer 1 and layer p may be considered a hidden layer of the machine learning model 104. As described above, each layer in this set of layers 302 may have any suitable number of neurons, implement any suitable type of activation function, and / or be connected to other layers via any suitable connection pattern.

[0080] In various implementations, activation component 116 can feed training data candidate 304 as input to machine learning model 104, where training data candidate 304 can be one of the data candidates from the set of training data candidates 106. Therefore, machine learning model 104 can analyze training data candidate 304 and generate output 306 based on training data candidate 304. As described above, output 306 can be a label for classifying training data candidate 304.

[0081] In further detail, layer 1 may receive training data candidates 304 and may generate various latent activations based on training data candidates 304 (e.g., one or more latent activation maps may be generated). In various cases, the latent activations generated by layer 1 may be received by the next layer 2 (not shown). Based on the latent activations provided by layer 1, layer 2 may generate its own latent activations, which may then be received by the next layer 3 (not shown). Similarly, layer j may receive latent activations generated by the previous layer j-1 and may generate its own latent activations based on the latent activations provided by the previous layer j-1. As shown, the latent activations generated by layer j may be referred to as training activation map 308 (e.g., the term "trained" may be used because the latent activations are based on training data candidates 304). Although Figure 3 Layer j is depicted as generating only one activation map, but this is a non-limiting example. In various cases, layer j can generate any suitable number of activation maps. In various cases, the next layer j+1 (not shown) can then receive the training activation map 308 generated by layer j and can generate its own latent activations based on the training activation map 308. Similarly, layer p can receive the latent activations generated by the previous layer p-1 (not shown) and can generate output 306 based on the latent activations of the previous layer p-1.

[0082] In various aspects, after activation component 116 feeds training data candidates 304 to layer 1, activation component 116 may electronically extract, capture, and / or otherwise record the training activation map 308 generated by layer j. In various cases, training activation map 308 may be considered as a latent activation map generated by the hidden layers (e.g., layer j) of machine learning model 104 based on training data candidates 304. In various cases, activation component 116 may repeat this process for each (and / or fewer than each) of the set of training data candidates 106. The final result of such repetition may be the set of training activation maps 202.

[0083] Figure 4 Similar to Figure 3 .like Figure 4As shown, in various embodiments, activation component 116 can feed test data candidate 402 as input to machine learning model 104, where test data candidate 402 can be one of the data candidates from the set of test data candidates 108. Therefore, machine learning model 104 can analyze test data candidate 402 and generate output 404 based on test data candidate 402. As described above, output 404 can be a label classifying test data candidate 402.

[0084] In further detail, layer 1 may receive test data candidate 402 and may generate various latent activations based on test data candidate 402 (e.g., one or more latent activation graphs may be generated). In various cases, the latent activations generated by layer 1 may be received by the next layer 2 (not shown). Based on the latent activations provided by layer 1, layer 2 may generate its own latent activations, which may then be received by the next layer 3 (not shown). Similarly, layer j may receive latent activations generated by the previous layer j-1 and may generate its own latent activations based on the latent activations provided by the previous layer j-1. As shown, the latent activations generated by layer j may be referred to as test activation graph 406 (e.g., the term "test" may be used because the latent activations are based on test data candidate 402). Although Figure 4 Layer j is depicted as generating only one activation map, but this is a non-limiting example. In various cases, layer j can generate any suitable number of activation maps. In various cases, the next layer j+1 (not shown) can then receive the test activation map 406 generated by layer j and can generate its own latent activations based on the test activation map 406. Similarly, layer p can receive the latent activations generated by the previous layer p-1 (not shown) and can generate output 404 based on the latent activations of the previous layer p-1.

[0085] In various aspects, after activation component 116 feeds test data candidates 402 to layer 1, activation component 116 may electronically extract, capture, and / or otherwise record the test activation map 406 generated by layer j. In various cases, test activation map 406 may be considered as a latent activation map generated by the hidden layers of machine learning model 104 (e.g., layer j) based on test data candidates 402. In various cases, activation component 116 may repeat this process for each (and / or fewer) of the test data candidates in the set of test data candidates 108. The end result of such repetition may be the set of test activation maps 204.

[0086] As described above, in various embodiments, training data candidates 304 and / or test data candidates 402 can exhibit any suitable dimension (e.g., each dimension can be one or more scalars, one or more vectors, one or more matrices, one or more tensors, and / or any suitable combination thereof). Similarly, in various cases, output 306 and / or output 404 can exhibit any suitable dimension. Furthermore, in each aspect, the latent activations generated by the layers in this set of layers 302 can exhibit any suitable dimension. In fact, those skilled in the art will understand that the dimension of the latent activation map (e.g., 308 and / or 406) can depend on the number and / or arrangement of neurons in the layers that generate the latent activation map (e.g., different layers can have different numbers of neurons and thus can generate different latent activations with different dimensions).

[0087] Those skilled in the art will understand that the set of training activation maps 202 may represent and / or otherwise indicate how the machine learning model 104 internally interprets the set of training data candidates 106. Similarly, the set of test activation maps 204 may be considered to represent and / or otherwise indicate how the machine learning model 104 internally interprets the set of test data candidates 108. Therefore, it may be desirable to compare the set of training activation maps 202 with the set of test activation maps 204. However, because each activation map in the set of training activation maps 202 and / or the set of test activation maps 204 can be a high-dimensional matrix and / or tensor (e.g., each activation map may include hundreds and / or thousands of elements), performing computations on the set of training activation maps 202 and / or the set of test activation maps 204 may be difficult and / or suboptimal.

[0088] Figure 5 A block diagram of an exemplary, non-limiting system 500 comprising compressed data points, which facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein, is shown. As shown, in some cases, system 500 may include the same components as system 200, and may also include a set of compressed training data points 502 and / or a set of compressed test data points 504.

[0089] In various embodiments, compression component 118 may electronically generate the set of compressed training data points 502 based on the set of training activation maps 202, and / or may electronically generate the set of compressed test data points 504 based on the set of test activation maps 204. More specifically, in various cases, compression component 118 may electronically reduce the dimension of each of the set of training activation maps 202 via any suitable dimensionality reduction technique (e.g., t-SNE and / or UMAP) to produce the set of compressed training data points 502. Similarly, in various cases, compression component 118 may electronically reduce the dimension of each of the set of test activation maps 204 via dimensionality reduction techniques to produce the set of compressed test data points 504. Relative to Figure 6 This was explained in more detail.

[0090] Figure 6 An exemplary, non-limiting block diagram illustrating how various compressed data points can be obtained according to one or more embodiments described herein.

[0091] As shown in the figure, under various conditions, this set of training activation maps 202 can include any appropriate number of training activation maps. That is, this set of training activation maps 202 can include training activation maps for any suitable positive integer x. Figure 1 To the training activation map x. In various cases, the compression component 118 can electronically apply any suitable dimensionality reduction technique (e.g., t-SNE, UMAP) to the set of training activation maps 202, thereby transforming and / or converting the set of training activation maps 202 into the set of compressed training data points 502. As shown, the set of compressed training data points 502 may correspond to the set of training activation maps 202 respectively. In other words, because the set of training activation maps 202 may include x individual training activation maps, the set of compressed training data points 502 may include x individual compressed training data points (e.g., compressed training data points 1 to compressed training data points x). More specifically, compressed training data point 1 may correspond to the training activation map x. Figure 1 This means that compressed training data point 1 can be considered as a training activation. Figure 1 A compressed and / or dimension-reduced version. For example, for any suitable positive integer n (e.g., n=2, n=50), training activations... Figure 1 It can be a matrix with hundreds and / or thousands of elements, and the compressed training data point 1 can represent the training activation. Figure 1An n-element vector. Similarly, compressed training data points x can correspond to training activation maps x, meaning that compressed training data points x can be considered a compressed and / or reduced-dimensional version of training activation maps x. For example, training activation maps x can be matrices with hundreds and / or thousands of elements, and compressed training data points x can be n-element vectors representing training activation maps x. Because the set of compressed training data points 502 can be derived from the set of training activation maps 202, the set of compressed training data points 502 can exhibit the pattern / distribution of the set of training activation maps 202. However, since the set of compressed training data points 502 can have a lower dimension than the set of training activation maps 202, computations can be performed more easily on the set of compressed training data points 502 than on the set of training activation maps 202.

[0092] As shown in the figure, under various conditions, the set of test activation graphs 204 can include any appropriate number of training activation graphs. That is, for any suitable positive integer y, the set of test activation graphs 204 can include training activation graphs. Figure 1 The training activation map y (e.g., y may be equal to or not equal to x) is then compressed. In various cases, the compression component 118 can electronically apply any suitable dimensionality reduction technique (e.g., t-SNE, UMAP) to the set of test activation maps 204, thereby converting and / or transforming the set of test activation maps 204 into the set of compressed test data points 504. As shown, the set of compressed test data points 504 may each correspond to the set of test activation maps 204. In other words, because the set of test activation maps 204 may include y individual test activation maps, the set of compressed test data points 504 may include y individual compressed test data points (e.g., compressed test data point 1 to compressed test data point y). More specifically, compressed test data point 1 may correspond to the test activation map y. Figure 1 This means that compressed test data point 1 can be considered a test activation. Figure 1 A compressed and / or dimension-reduced version. For example, for any suitable positive integer n (e.g., n=2, n=50), the test activation... Figure 1 It can be a matrix with hundreds and / or thousands of elements, and the compressed test data point 1 can represent the test activation. Figure 1An n-element vector. Similarly, compressed test data points y can correspond to test activation graph y, meaning that compressed test data points y can be considered a compressed and / or reduced-dimensional version of test activation graph y. For example, test activation graph y can be a matrix with hundreds and / or thousands of elements, and compressed test data points y can be an n-element vector representing test activation graph y. Because the set of compressed test data points 504 can be derived from the set of test activation graphs 204, the set of compressed test data points 504 can exhibit the pattern / distribution of the set of test activation graphs 204. However, since the set of compressed test data points 504 can have a lower dimension than the set of test activation graphs 204, computations can be performed more easily on the set of compressed test data points 504 than on the set of test activation graphs 204.

[0093] Figure 7 A block diagram of an exemplary, non-limiting system 700, comprising a diversity score, is shown, which facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein. As shown, in some cases, system 700 may include the same components as system 500 and may also include a diversity score 702.

[0094] In various implementations, the diversity component 120 may electronically calculate a diversity score 702 based on the set of compressed training data points 502 and / or based on the set of compressed test data points 504. More specifically, the diversity component 120 may statistically analyze the patterns / distributions exhibited by the set of compressed training data points 502 and may calculate a measure of how well and / or poorly the set of compressed test data points 504 conforms to those patterns / distributions.

[0095] In various implementations, as described above, because the set of compressed training data points 502 can be derived from the set of training data candidates 106, and because the machine learning model 104 can be trained on the set of training data candidates 106, the set of compressed training data points 502 can exhibit clusters corresponding to the outputs that the machine learning model 104 is configured to generate. For example, for any suitable positive integer m, if the machine learning model 104 is configured to classify the input data candidates into one of m distinct categories, the compressed training data points 502 can be arranged into m clusters. In various aspects, a cluster can be a set and / or subset of compressed training data points with elements similar to each other (e.g., if two training data candidates belong to the same category, the two compressed training data points corresponding to these two training data candidates respectively can have numerically similar elements and / or can otherwise be separated from each other by a relatively small Euclidean distance).

[0096] In various scenarios, for each cluster, the diversity component 120 can calculate the center of that cluster. In various aspects, the center of a given cluster can be equal to the average of all compressed training data points belonging to that given cluster. For example, if each compressed training data point is an n-element vector, then the center of a given cluster can be the average of all n-element vectors in the set of compressed training data points 502 that belong to that given cluster. Therefore, if the set of compressed training data points 502 comprises m clusters, the diversity component 120 can calculate m centers (e.g., one center per cluster, and / or one average n-element vector per cluster).

[0097] In various aspects, for each cluster, the diversity component 120 can calculate the Euclidean distance between the center of the cluster and each (and / or fewer) of the compressed training data points belonging to that cluster. In various cases, the diversity component 120 can then calculate the mean (denoted as μ for convenience) and / or the standard deviation (denoted as σ for convenience) of such Euclidean distances for each cluster. Thus, if the set of compressed training data points 502 comprises m clusters, the diversity component 120 can calculate m distinct μ values ​​and / or m distinct σ values ​​(e.g., a (μ,σ) pair for each cluster, wherein the (μ,σ) pair for a given cluster describes the density and / or spatial distribution of the compressed training data points in the given cluster).

[0098] In various cases, the diversity component 120 may assign each of the set of compressed test data points 504 to the corresponding cluster of the set of compressed training data points 502. More specifically, for each compressed test data point, the diversity component 120 may calculate the Euclidean distance between the compressed test data point and the center of each cluster of the set of compressed training data points 502, and the diversity component 120 may assign the compressed test data point to the nearest and / or closest cluster. Thus, if the set of compressed training data points 502 comprises m clusters, the diversity component 120 may determine which of those m clusters has the closest and / or closest (in terms of Euclidean distance) center to a given compressed test data point, and the diversity component 120 may accordingly assign the given compressed test data point to that closest and / or closest cluster.

[0099] In various cases, for each of the compressed test data points 504 in the set, the diversity component 120 can determine whether the compressed test data point is an outlier or a non-outlier. In various cases, if the Euclidean distance between a given compressed test data point and the center of the cluster to which it is assigned is greater than a threshold Euclidean distance corresponding to that cluster, then the given compressed test data point can be considered an outlier. Conversely, in various aspects, if the Euclidean distance between a given compressed test data point and the center of the cluster to which it is assigned is less than a threshold Euclidean distance corresponding to that cluster, then the given compressed test data point can be considered a non-outlier. In various cases, the threshold Euclidean distance can vary with the cluster. For example, the threshold Euclidean distance for a given cluster can be equal to μ + 2σ (e.g., because μ and σ can vary with the cluster, the threshold Euclidean distance can also vary with the cluster). Therefore, the diversity component 120 can mark each of the compressed test data points 504 as an outlier or a non-outlier.

[0100] In various aspects, the diversity component 120 can then calculate the total proportion and / or percentage of the group of compressed test data points 504 considered outliers. In various cases, such a total proportion and / or percentage can be considered equal to the diversity score 702. In various other cases, the diversity score 702 can be any suitable mathematical function of such a total proportion and / or percentage. In any case, the diversity score 702 can be configured such that a higher value of the diversity score 702 indicates that there are more outliers in the group of compressed test data points 504, which can indicate that the group of compressed test data points 504 does not neatly conform to the clusters of the group of compressed training data points 502. This can mean that the machine learning model 104 interprets the patterns / distributions exhibited by the group of test data candidates 108 in a different way than how the machine learning model 104 interprets the patterns / distributions exhibited by the group of training data candidates 106. Furthermore, in any case, the diversity score 702 can be configured such that a lower value of the diversity score 702 indicates that there are more non-outliers in the group of compressed test data points 504, which could indicate that the group of compressed test data points 504 more neatly conforms to the cluster of the group of compressed training data points 502. This could mean that the machine learning model 104 interprets the patterns / distributions exhibited by the group of test data candidates 108 in a similar way to how the machine learning model 104 interprets the patterns / distributions exhibited by the group of training data candidates 106.

[0101] Figure 8A block diagram of an exemplary, non-limiting system 800, including graphs, is shown, which facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein. As shown, in some cases, system 800 may include the same components as system 700, and may also include Figure 802.

[0102] In various embodiments, visualization component 122 may electronically present Figure 802 on any suitable computer screen and / or monitor (not shown), such that Figure 802 can be viewed by the operator of machine learning model 104. In various cases, if each of the set of compressed training data points 502 and / or each of the set of compressed test data points 504 is two-dimensional and / or three-dimensional (e.g., if n=2 and / or if n=3), then Figure 802 may depict and / or illustrate the set of compressed training data points 502 and / or the set of compressed test data points 504. In various aspects, the degree to which the set of compressed test data points 504 conforms to the clustering of the set of compressed training data points 502 can be easily visually perceived by viewing Figure 802. In some cases, Figure 802 may also include a diversity score 702. Therefore, the operator of machine learning model 104 can visually be informed of the diversity score 702 and / or can visually see how well and / or poorly the set of compressed test data points 504 conforms to the clustering of the set of compressed training data points 502. Thus, when making design decisions regarding machine learning model 104 and / or the set of test data candidates 108, the operator can utilize and / or rely on the diversity score 702 and / or Figure 802.

[0103] In various cases, Figure 802 can be interactive and / or clickable. For example, compressed data points plotted on the graph (e.g., from 502 and / or 504) can be clickable, and an operator can click the compressed data points using any suitable human-machine interface device (e.g., computer mouse, touchscreen, voice command). In various cases, in response to a compressed data point being clicked, visualization component 122 can electronically present data candidates (e.g., from 106 and / or 108) corresponding to the clicked compressed data point. Thus, by selectively clicking different compressed data points plotted on Figure 802, the operator of machine learning model 104 can selectively and manually examine any data candidates as needed. Such functionality can also assist the operator in making design decisions regarding machine learning model 104 and / or the set of test data candidates 108.

[0104] In various embodiments, execution component 124 may electronically transfer the diversity score 702 and / or Figure 802 to any suitable computing device (not shown). In various other embodiments, execution component 124 may make recommendations to an operator based on the diversity score 702. For example, if an operator wishes to determine whether a machine learning model 104 can be trained on the set of test data candidates 108 without overfitting, execution component 124 may compare the diversity score 702 to any suitable threshold and make a recommendation accordingly. If the diversity score 702 is above the threshold, execution component 124 may determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a manner sufficiently different from that of the set of training data candidates 106, such that the machine learning model 104 can be trained on the set of test data candidates 108 without the risk of overfitting. On the other hand, if the diversity score 702 is below the threshold, the execution unit 124 can determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a way that is not sufficiently different from the set of training data candidates 106, so that the machine learning model 104 cannot be trained on the set of test data candidates 108 without the risk of overfitting.

[0105] For example, if an operator wishes to determine whether machine learning model 104 can be accurately deployed in the domain and / or environment associated with the set of test data candidates 108, execution component 124 can compare the diversity score with any suitable threshold and make a recommendation accordingly. If the diversity score 702 is below the threshold, execution component 124 can determine that machine learning model 104 is interpreting the set of test data candidates 108 in a manner sufficiently similar to the set of training data candidates 106, such that machine learning model 104 can be deployed in the domain and / or environment without the risk of frequent inaccurate performance. On the other hand, if the diversity score 702 is above the threshold, execution component 124 can determine that machine learning model 104 is interpreting the set of test data candidates 108 in a manner not sufficiently similar to the set of training data candidates 106, such that machine learning model 104 cannot be deployed in the domain and / or environment without the risk of frequent inaccurate performance.

[0106] For example, if a set of test data candidates 108 is created by modifying the set of training data candidates 106 via an enhancement strategy, and if the operator wishes to determine whether such enhancement strategy helps improve the generality of the machine learning model 104, the execution unit 124 can compare the diversity score with any suitable threshold and make a recommendation accordingly. If the diversity score 702 is higher than the threshold, the execution unit 124 can determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a manner sufficiently different from that of the set of training data candidates 106, such that the enhancement strategy is filling the learning gaps of the machine learning model 104. On the other hand, if the diversity score 702 is lower than the threshold, the execution unit 124 can determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a manner not sufficiently different from that of the set of training data candidates 106, such that the enhancement strategy is not filling the learning gaps of the machine learning model 104.

[0107] For example, if the set of test data candidates 108 is unannotated, and if the operator wishes to determine whether automatic annotation can be accurately applied to the set of test data candidates 108, the execution unit 124 can compare the diversity score with any suitable threshold and make a recommendation accordingly. If the diversity score 702 is below the threshold, the execution unit 124 can determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a manner sufficiently similar to the set of training data candidates 106, so that the automatic annotation technique can be accurately applied to the set of test data candidates 108. On the other hand, if the diversity score 702 is above the threshold, the execution unit 124 can determine that the machine learning model 104 is interpreting the set of test data candidates 108 in a manner not sufficiently similar to the set of training data candidates 106, so that the automatic annotation technique cannot be accurately applied to the set of test data candidates 108. On the other hand, since the diversity component 120 may have already marked each of the compressed test data points 504 as an outlier or non-outlier, the execution component 124 may recommend applying automatic annotation techniques to compressed test data candidates corresponding to non-outliers, and / or may recommend applying manual annotation techniques to compressed test data candidates corresponding to outliers.

[0108] Figures 9 to 11 Exemplary, non-limiting figures 900 to 1100 illustrate compressed data points according to one or more embodiments described herein. In other words, Figures 9 to 11 An exemplary visualization that can be presented by visualization component 122 is shown.

[0109] exist Figures 9 to 11In a non-limiting example, machine learning model 104 is configured to classify X-ray images of a patient's knee into one of five distinct categories: tibia, femur, sagittal correlation, coronal correlation, or no correlation. Furthermore, this set of training data candidates comprises a total of 500 X-ray images: specifically, 100 X-ray images per category, on which machine learning model 104 has already been trained. Additionally, compared to this set of training data candidates 106, this set of test data candidates 108 comprises 500 X-ray images randomly selected from a new data source (e.g., from a different hospital, or from the same hospital but from a different X-ray scanner). This set of compressed training data points 502 and this set of compressed test data points 504 are then generated using t-SNE as a dimensionality reduction technique as described herein, where each compressed data point is a binary vector. Figures 900 to 1100 are then plotted.

[0110] The training data set has 106 candidates. Figures 9 to 11 The same applies in the middle. However, this set of test data has 108 candidate spans. Figures 9 to 11 The variation allows us to see different levels of diversity. Because t-SNE is nonparametric, the absolute position of each compressed training data point is... Figures 9 to 11 The distance between the compressed training data points is not fixed, but it maintains the relative distance between them.

[0111] exist Figure 9 In the figure, the set of compressed training data points 502 is shown in red in the color version of the figure and in black in the black and white version of the figure, and the set of compressed test data points 504 is shown in green in the color version of the figure and in white in the black and white version of the figure. As shown, the set of compressed training data points 502 is grouped into five different clusters 902 to 910, which correspond to five different categories (e.g., tibia, femur, sagittal correlation, coronal correlation, no correlation). As can be readily seen, Figure 900 shows a non-limiting example in which the set of compressed test data points 504 fits well to and / or otherwise conforms to the clusters of the set of compressed training data points 502. In other words, Figure 9 The white (or green) dots in the graph are very close to the clusters of black (or red) dots. Therefore, the diversity score 702 corresponding to Figure 900 can be small in magnitude. Based on Figure 900, the operator can determine that the machine learning model 104 can be accurately deployed on the set of test data candidates 108, that the machine learning model 104 cannot be trained on the set of test data candidates 108 without overfitting, and / or that automatic annotations can be accurately applied to the set of test data candidates 108.

[0112] exist Figure 10In the figure, the compressed training data points 502 are shown in red in the color version of the figure and in black in the black and white version of the figure, and the compressed test data points 504 are shown in green in the color version of the figure and in white in the black and white version of the figure. As shown, the compressed training data points 502 are again grouped into five different clusters 1002 to 1010, which correspond to five different categories (e.g., tibia, femur, sagittal correlation, coronal correlation, no correlation). As can be seen, Figure 1000 shows a non-limiting example in which the compressed test data points 504 fit to and / or otherwise conform to the clusters of the compressed training data points 502 to some extent. In other words, Figure 10 Some white (or green) dots are located near clusters of black (or red) dots, while other white (or green) dots are not located near clusters of black (or red) dots. Therefore, the diversity score 702 corresponding to Figure 1000 can be intermediate in magnitude.

[0113] exist Figure 11 In the figure, the set of compressed training data points 502 is shown in red in the color version of the figure and in black in the black and white version of the figure, and the set of compressed test data points 504 is shown in green in the color version of the figure and in white in the black and white version of the figure. As shown, the set of compressed training data points 502 is grouped into five different clusters 902 to 910, which correspond to five different categories (e.g., tibia, femur, sagittal correlation, coronal correlation, no correlation). As can be readily seen, Figure 1100 shows a non-limiting example in which the set of compressed test data points 504 does not fit well to and / or otherwise conforms to the clusters of the set of compressed training data points 502. In other words, Figure 11 The white (or green) dots in the graph are mostly located far from the clusters of black (or red) dots. Therefore, the diversity score 702 corresponding to curve 1100 can be large in magnitude. Based on Figure 1100, the operator can determine that the machine learning model 104 may not be accurately deployed on the set of test data candidates 108, the machine learning model 104 can be trained on the set of test data candidates 108 without overfitting, and / or automatic annotations may not be accurately applied to the set of test data candidates 108.

[0114] Figures 12 to 15 Flowcharts are shown of exemplary, non-limiting computer-implemented methods 1200 to 1500 relating to the quantification of data diversity for machine learning models according to one or more embodiments described herein. In other words, Figures 12 to 15Exemplary and non-limiting operations / actions that the data diversity system 102 can facilitate in various implementations are collectively illustrated.

[0115] First, consider computer-implemented method 1200, which illustrates how a data diversity system 102 can generate a set of compressed training data points 502 based on a set of training data candidates 106.

[0116] In various implementations, action 1202 may include receiving a set of training data candidates (e.g., 106) of a neural network (e.g., 104) already trained thereon by a device (e.g., 114) operatively coupled to the processor.

[0117] In various aspects, action 1204 may include initializing an empty set of training activation maps (e.g., 202) by a device (e.g., 116).

[0118] In various cases, action 1206 may include the device (e.g., 116) determining whether each training data candidate in the set of training data candidates has been analyzed by the device. If not, the computer-implemented method 1200 may proceed to action 1208. If yes, the computer-implemented method 1200 may proceed to action 1216.

[0119] In various cases, action 1208 may include the selection of training data candidates that have not yet been analyzed by the device (e.g., 116) from the set of training data candidates.

[0120] In various aspects, action 1210 may include feeding selected training data candidates as input to the neural network by a device (e.g., 116).

[0121] In various cases, action 1212 may include the extraction by a device (e.g., 116) of a hidden activation map generated by the hidden layers of the neural network based on and / or in response to selected training data candidates.

[0122] In various cases, action 1214 may include inserting the extracted hidden activation map into the set of training activation maps by a device (e.g., 116). In various aspects, computer-implemented method 1200 may proceed to action 1206.

[0123] As shown in the figure, the computer-implemented method 1200 can iterate through actions 1206 to 1214 until each training data candidate has been analyzed (e.g., until a hidden activation map has been inserted into the set of training activation maps for each training data candidate). At this point, the computer-implemented method 1200 can proceed to action 1216.

[0124] In various cases, action 1216 may include performing a dimensionality reduction technique (e.g., t-SNE, UMAP) on the set of training activation maps by a device (e.g., 118), thereby transforming the set of training activation maps into a set of compressed training data points (e.g., 502). In other words, each training activation map can now be represented by low-dimensional data points that are easier to visualize and / or mathematically manipulate.

[0125] Next, consider computer-implemented method 1300, which illustrates how data diversity system 102 can generate a set of compressed test data points 504 based on the set of test data candidates 108.

[0126] In various specific implementations, action 1302 may include receiving a set of test data candidates (e.g., 108) by a device (e.g., 114).

[0127] In various aspects, action 1304 may include initializing an empty set of test activation graphs (e.g., 204) by a device (e.g., 116).

[0128] In various cases, action 1306 may include the device (e.g., 116) determining whether each test data candidate in the set of test data candidates has been analyzed by the device. If not, the computer-implemented method 1300 may proceed to action 1308. If yes, the computer-implemented method 1300 may proceed to action 1316.

[0129] In various cases, action 1308 may include the device (e.g., 116) selecting test data candidates that have not yet been analyzed from the set of test data candidates.

[0130] In various aspects, action 1310 may include feeding selected test data candidates as input to the neural network by a device (e.g., 116).

[0131] In various cases, action 1312 may include the extraction by a device (e.g., 116) of a hidden activation map generated by the hidden layers of the neural network based on and / or in response to selected test data candidates.

[0132] In various cases, action 1314 may include inserting the extracted hidden activation map into the set of test activation maps by a device (e.g., 116). In various aspects, computer-implemented method 1300 may proceed to action 1306.

[0133] As shown in the figure, the computer-implemented method 1300 can iterate through actions 1306 to 1314 until each test data candidate has been analyzed (e.g., until a hidden activation map has been inserted into the set of test activation maps for each test data candidate). At this point, the computer-implemented method 1300 can proceed to action 1316.

[0134] In various cases, action 1316 may include performing dimensionality reduction techniques (e.g., t-SNE, UMAP) on the set of test activation maps by a device (e.g., 118), thereby transforming the set of test activation maps into a set of compressed test data points (e.g., 504). In other words, each test activation map can now be represented by low-dimensional data points that are easier to visualize and / or mathematically manipulate.

[0135] Now, consider computer-implemented method 1400, which illustrates how the data diversity system 102 can analyze the set of compressed training data points 502.

[0136] In various implementations, action 1402 may include the device (e.g., 120) identifying known clusters exhibited by the set of compressed training data points (e.g., 502). In some cases, because the set of compressed training data points can be derived from the set of training data candidates, and because the neural network can be trained on the set of training data candidates, the set of compressed training data points may exhibit clusters corresponding to the output generated by the neural network. For example, for any suitable positive integer m, if the neural network is configured to classify and / or label the input data candidates as belonging to one of m categories, the set of compressed training data points may exhibit an arrangement of m clusters (e.g., may have m clusters).

[0137] In various aspects, action 1404 may include determining, by the device (e.g., 120), whether each cluster of the set of compressed training data points has been analyzed. If not, computer-implemented method 1400 may proceed to action 1406. If yes, computer-implemented method 1400 may proceed to action 1502 of computer-implemented method 1500.

[0138] In various cases, action 1406 may include the selection by the device (e.g., 120) of a cluster of unanalyzed compressed training data points.

[0139] In various aspects, action 1408 may include the center of the selected cluster being calculated by the device (e.g., 120). For example, the center of a given cluster of compressed training data points may be equal to the average of all compressed training data points belonging to that given cluster.

[0140] In various aspects, for each compressed training data point belonging to the selected cluster, action 1410 may include having a device (e.g., 120) calculate the Euclidean distance between the compressed training data point and the center of the selected cluster. When this operation is performed for each compressed training data point belonging to the selected cluster, this can produce a set of Euclidean distances associated with the selected cluster.

[0141] In various cases, action 1412 may include the device (e.g., 120) calculating an average distance value for the selected cluster based on the set of Euclidean distances associated with the selected cluster, the average distance value being represented as μ.

[0142] In various cases, action 1414 may include having a device (e.g., 120) calculate a standard deviation distance value for the selected cluster based on the set of Euclidean distances associated with the selected cluster, the standard deviation distance value being represented as σ. In various aspects, computer-implemented method 1400 may proceed to action 1404.

[0143] As shown in the figure, the computer-implemented method 1400 can iterate through actions 1404 to 1414 until μ and σ are computed for each cluster of compressed training data points.

[0144] Finally, consider computer-implemented method 1500, which demonstrates how the data diversity system 102 can analyze the set of compressed test data points 504.

[0145] In various implementations, action 1502 may include the device (e.g., 120) determining whether each compressed test data point in the set of compressed test data points (e.g., 504) has been analyzed. If not, computer-implemented method 1500 may proceed to action 1504. If yes, computer-implemented method 1500 may proceed to action 1514.

[0146] In various aspects, action 1504 may include the selection by the device (e.g., 120) of compressed test data points that have not yet been analyzed from the set of compressed test data points.

[0147] In various cases, action 1506 may include the device (e.g., 120) calculating the Euclidean distance between the selected compressed test data points and the nearest cluster of compressed training data points (e.g., the cluster whose center is closest to and / or closest to the selected compressed test data points in terms of Euclidean distance).

[0148] In various cases, action 1508 may include the device (e.g., 120) determining whether the calculated Euclidean distance is greater than a threshold μ+2σ for the nearest cluster (e.g., where μ and σ correspond to the nearest cluster). In other words, the device may determine whether the selected compressed test data points are more than two standard deviations away from the center of the nearest cluster. If yes, the computer-implemented method 1500 may proceed to action 1510. If no, the computer-implemented method 1500 may proceed to action 1512.

[0149] In various aspects, action 1510 may include marking selected compressed test data points as outliers by a device (e.g., 120). In various cases, computer-implemented method 1500 may proceed back to action 1502.

[0150] In various aspects, action 1512 may include having the device (e.g., 120) mark the selected compressed test data points as non-outliers. In various cases, the computer-implemented method 1500 may proceed back to action 1502.

[0151] As shown in the figure, the computer-implemented method 1500 can iterate through actions 1502 to 1512 until each compressed test data point has been marked as an outlier or non-outlier. At such times, the computer-implemented method 1500 can proceed to action 1514.

[0152] In various cases, action 1514 may include the calculation by the device (e.g., 120) of the percentage of that set of compressed test data points marked as outliers. In various cases, this percentage may be considered as a diversity score 702. In other cases, the diversity score 702 may be equal to any suitable mathematical function of this percentage. Although in Figure 15 Not explicitly shown, but the percentage of compressed test data points in the group considered outliers indicates how similar and / or dissimilar the group of test data candidates is to the group of training data candidates. Therefore, as described above, execution unit 124 can take any appropriate computational action based on the value of this percentage.

[0153] Figure 16 A flowchart is shown of an exemplary, non-limiting computer-implemented method 1600 that facilitates the visualization and / or quantification of data diversity for machine learning models according to one or more embodiments described herein. In various cases, the computer-implemented method 1600 may be implemented by a data diversity system 102.

[0154] In various implementations, action 1602 may include access to a first set of data candidates (e.g., 106) and a second set of data candidates (e.g., 108) by a device operatively coupled to the processor (e.g., 114), wherein a machine learning model is trained on the first set of data candidates (e.g., 104).

[0155] In various cases, action 1604 may include obtaining a first set of potential activations generated by a machine learning model based on a first set of data candidates (e.g., 202) by the device (e.g., 116), and obtaining a second set of potential activations generated by a machine learning model based on a second set of data candidates (e.g., 204) by the device (e.g., 116).

[0156] In various aspects, action 1606 may include generating a first set of compressed data points (e.g., 502) by the device (e.g., 118) by applying a dimensionality reduction technique to a first set of potential activations, and generating a second set of compressed data points (e.g., 504) by the device (e.g., 118) by applying a dimensionality reduction technique to a second set of potential activations.

[0157] In various cases, action 1608 may include calculating a diversity score by a device (e.g., 120) based on a first set of compressed data points and a second set of compressed data points (e.g., 702).

[0158] In various aspects, action 1610 may include presenting a graph (e.g., 802) by a device (e.g., 122) depicting a first set of compressed data points and a second set of compressed data points.

[0159] Although not in Figure 16 As explicitly shown, but a given compressed data point plotted in the figure may correspond to a given data candidate from a first set of data candidates or a second set of data candidates, the given compressed data point may be clickable as plotted in the figure, and / or the computer-implemented method 1600 may further include: in response to a given compressed data point being clicked, the device (e.g., 122) presents the given data candidate.

[0160] Although Figure 16 Not explicitly shown, but the second set of data candidates may represent an enhanced version of the first set of data candidates and / or may otherwise represent potential training data for a machine learning model, and the computer-implemented method 1600 may further include: recommending, by the device (e.g., 124), to train a machine learning model on the second set of data candidates in response to the determination that the diversity score meets a predetermined threshold; and recommending, by the device (e.g., 124), not to train a machine learning model on the second set of data candidates in response to the determination that the diversity score fails to meet the predetermined threshold.

[0161] Despite Figure 16 Although not explicitly shown, the second set of data candidates may be unannotated, and the computer-implemented method 1600 may further include: identifying one or more peripheral compressed data points in the second set of compressed data points by the device (e.g., 120) by comparing the second set of compressed data points with the first set of compressed data points; and recommending, by the device (e.g., 124), to manually annotate one or more data candidates from the second set of data candidates corresponding to one or more peripheral compressed data points.

[0162] Despite Figure 16 Although not explicitly shown, the second set of data candidates may be unannotated, and the computer-implemented method 1600 may further include: identifying one or more non-peripheral compressed data points in the second set of compressed data points by the device (e.g., 120) by comparing the second set of compressed data points with the first set of compressed data points; and recommending one or more data candidates from the second set of data candidates corresponding to the one or more non-peripheral compressed data points by the device (e.g., 124).

[0163] Various implementations of this innovative topic involve a computerized tool that can automatically visualize and / or quantify the diversity in input data for artificial intelligence applications. This computerized tool can be used to select new data candidates from a data lake, determine augmentation strategies to be implemented, and / or identify which unannotated data candidates can be automatically annotated / curated. In fact, selecting appropriate data candidates on which machine learning models are trained is important for helping to avoid overfitting and / or helping to improve model generality. Furthermore, selecting appropriate data augmentation strategies is equally important for helping to avoid overfitting and / or helping to improve model generality. Moreover, because manual annotation is very cumbersome and time-consuming, selecting appropriate data candidates for automatic annotation (e.g., semi-supervised annotation via existing models, transfer learning) can be important for helping to save time and / or resources. Because the computerized tool described herein can assist model developers in making these and / or other decisions regarding model design, it certainly constitutes a useful and practical application of computing.

[0164] To provide additional context for the various implementation schemes described herein Figure 17 The following discussion is intended to provide a brief general description of a suitable computing environment 1700 in which various embodiments of the embodiments described herein may be implemented. Although the embodiments have been described above in the general context of computer-executable instructions that can run on one or more computers, those skilled in the art will recognize that these embodiments may also be implemented in combination with other program modules and / or as a combination of hardware and software.

[0165] Typically, program modules include routines, programs, components, data structures, etc., that perform specific tasks or implement specific abstract data types. Furthermore, those skilled in the art will understand that the methods of this invention can be practiced with other computer system configurations, including single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things (IoT) devices, distributed computing systems, and personal computers, handheld computing devices, microprocessor-based or programmable consumer electronics, each operatively coupled to one or more associated devices.

[0166] The embodiments illustrated in this paper can also be practiced in a distributed computing environment where specific tasks are performed by remote processing devices linked via a communication network. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0167] Computing devices typically include a variety of media, which may include computer-readable storage media, machine-readable storage media, and / or communication media, wherein the two terms are used interchangeably herein, as described below. A computer-readable storage medium or a machine-readable storage medium can be any available storage medium accessible by a computer, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, a computer-readable storage medium or a machine-readable storage medium can be implemented in conjunction with any method or technique for storing information such as computer-readable or machine-readable instructions, program modules, structured data, or unstructured data.

[0168] Computer-readable storage media may include, but is not limited to, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CDROM), digital versatile disc (DVD), Blu-ray disc (BD) or other optical disc storage devices, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible and / or non-transitory media that can be used to store desired information. In this regard, the terms “tangible” or “non-transitory” used herein to describe storage devices, memories, or computer-readable media should be understood to exclude only the propagation of transient signals themselves as a modifier, and do not waive the rights of all standard storage devices, memories, or computer-readable media that do not only propagate transient signals themselves.

[0169] Computer-readable storage media can be accessed by one or more local or remote computing devices, for example, via access requests, queries or other data retrieval protocols, to perform various operations on the information stored on the media.

[0170] Communication media typically contain computer-readable instructions, data structures, program modules, or other structured or unstructured data in a data signal. This data signal can be a modulated data signal, such as a carrier wave or other transmission mechanism, and includes any information transmission or delivery medium. The terms "modulated data signal" or "signal" refer to a signal whose one or more characteristics are set or altered to encode information in one or more signals. By way of example and not limitation, communication media include wired media, such as wired networks or direct wired connections, and wireless media, such as acoustic, RF, infrared, and other wireless media.

[0171] Refer again Figure 17 An exemplary environment 1700 for implementing various embodiments of the aspects described herein includes a computer 1702, which includes a processing unit 1704, system memory 1706, and a system bus 1708. The system bus 1708 connects system components, including but not limited to the system memory 1706, to the processing unit 1704. The processing unit 1704 can be any of a variety of commercially available processors. Dual microprocessors and other multiprocessor architectures may also be used as the processing unit 1704.

[0172] System bus 1708 can be any of several types of bus structures that can be further interconnected to a memory bus (with or without a memory controller), a peripheral bus, and a local bus using any of a variety of commercially available bus architectures. System memory 1706 includes ROM 1710 and RAM 1712. The Basic Input / Output System (BIOS) can be stored in non-volatile memory such as ROM, erasable programmable read-only memory (EPROM), or EEPROM, where the BIOS contains basic routines that facilitate, for example, transferring information between components within computer 1702 during startup. RAM 1712 may also include high-speed RAM, such as static RAM for caching data.

[0173] Computer 1702 also includes an internal hard disk drive (HDD) 1714 (e.g., EIDE, SATA), one or more external storage devices 1716 (e.g., floppy disk drive (FDD) 1716, memory stick or flash drive reader, memory card reader, etc.), and a drive 1720, such as a solid-state drive or optical disc drive, which can read from or write to a disk 1722 (e.g., CD-ROM, DVD, BD, etc.). Alternatively, in cases involving solid-state drives, disk 1722 is not included unless separate. Although the internal HDD 1714 is shown as residing within computer 1702, the internal HDD 1714 can also be configured for external use in a suitable infrastructure (not shown). Additionally, although not shown in environment 1700, a solid-state drive (SSD) may be used as a supplement or alternative to the HDD 1714. HDD 1714, external storage device 1716, and drive 1720 can be connected to system bus 1708 via HDD interface 1724, external storage interface 1726, and drive interface 1728, respectively. Interface 1724 for the specific implementation of the external drive may include at least one or both of Universal Serial Bus (USB) and Institute of Electrical and Electronics Engineers (IEEE) 1394 interface technologies. Other external drive connection technologies are contemplated in the embodiments described herein.

[0174] Drives and their associated computer-readable storage media provide non-volatile storage of data, data structures, computer-executable instructions, etc. For computer 1702, drives and storage media accommodate the storage of any data in a suitable digital format. Although the above description of computer-readable storage media refers to a corresponding type of storage device, those skilled in the art will understand that other types of computer-readable storage media (whether currently existing or developed in the future) can also be used in the exemplary operating environment, and furthermore, any such storage media may contain computer-executable instructions for performing the methods described herein.

[0175] Multiple program modules can be stored in the drive and RAM 1712, including an operating system 1730, one or more application programs 1732, other program modules 1734, and program data 1736. All or part of the operating system, application programs, modules, and / or data may also be cached in RAM 1712. The systems and methods described herein can be implemented using various commercially available operating systems or combinations of operating systems.

[0176] Computer 1702 may optionally include emulation technology. For example, a super manager (not shown) or other intermediary may emulate the hardware environment used for operating system 1730, and the emulated hardware may optionally differ from that of the computer. Figure 17The hardware is shown. In such implementations, the operating system 1730 may include one of a plurality of virtual machines (VMs) hosted at the computer 1702. Furthermore, the operating system 1730 may provide a runtime environment for the application 1732, such as the Java Runtime Environment or the .NET Framework. A runtime environment is a compatible execution environment that allows the application 1732 to run on any operating system that includes a runtime environment. Similarly, the operating system 1730 may support containers, and the application 1732 may take the form of a container, which is a lightweight, standalone, executable software package that includes, for example, code, runtime, system tools, system libraries, and settings for the application.

[0177] Additionally, the computer 1702 can utilize security modules such as Trusted Processing Modules (TPMs) for enabling this. For example, in the case of a TPM, the boot part is hashed in the next boot part and waits for the result to be matched with a security value before loading the next boot part. This process can occur at any layer of the computer 1702's code execution stack, such as at the application execution level or the operating system (OS) kernel level, thereby achieving security at any code execution level.

[0178] Users can input commands and information into computer 1702 using one or more wired / wireless input devices (e.g., keyboard 1738, touchscreen 1740, and pointing devices such as mouse 1742). Other input devices (not shown) may include microphones, infrared (IR) remote controls, radio frequency (RF) remote controls or other remote controls, joysticks, virtual reality controllers and / or virtual reality headsets, gamepads, styluses, image input devices (e.g., cameras), gesture sensor input devices, visual motion sensor input devices, emotion or face detection devices, biometric input devices (e.g., fingerprint or iris scanners), etc. These and other input devices are typically connected to processing unit 1704 via input device interface 1744, which may be coupled to system bus 1708, but these and other input devices may be connected via other interfaces, such as parallel ports, IEEE 1394 serial ports, game ports, USB ports, IR interfaces, etc. Interfaces, etc.

[0179] Monitor 1746 or other types of display devices can also be connected to system bus 1708 via an interface (such as video adapter 1748). In addition to monitor 1746, computers typically include other peripheral output devices (not shown), such as speakers, printers, etc.

[0180] Computer 1702 can operate in a networked environment using logical connections to one or more remote computers (such as remote computer 1750) via wired and / or wireless communications. Remote computer 1750 can be a workstation, server computer, router, personal computer, portable computer, microprocessor-based entertainment device, peer-to-peer device, or other common network node, and typically includes many or all of the elements described relative to computer 1702, but for simplicity, only memory / storage device 1752 is shown. The logical connections shown include wired / wireless connections to a local area network (LAN) 1754 and / or a larger network (e.g., a wide area network (WAN) 1756). Such LAN and WAN networking environments are common in offices and corporations and are advantageous for enterprise-wide computer networks, such as intranets, all of which can connect to global communication networks, such as the Internet.

[0181] When used in a LAN networking environment, computer 1702 can connect to local network 1754 via a wired and / or wireless communication network interface or adapter 1758. Adapter 1758 facilitates wired or wireless communication with LAN 1754, which may also include a wireless access point (AP) configured thereon for communicating with adapter 1758 in wireless mode.

[0182] When used in a WAN networking environment, computer 1702 may include modem 1760, or may be connected to a communication server on WAN 1756 via other means (such as via the Internet) for establishing communication over WAN 1756. Modem 1760 (which may be internal or external, wired or wireless) may be connected to system bus 1708 via input device interface 1744. In a networking environment, program modules shown relative to computer 1702 or portions thereof may be stored in remote memory / storage device 1752. It should be understood that the network connection shown is an example, and other means for establishing communication links between computers may be used.

[0183] When used in a LAN or WAN networking environment, in addition to or as an alternative to the external storage device 1716 described above, computer 1702 can access cloud storage systems or other network-based storage systems, such as, but not limited to, network virtual machines that provide one or more aspects of information storage or processing. Generally, the connection between computer 1702 and the cloud storage system can be established, for example, via adapter 1758 or modem 1760 through LAN 1754 or WAN 1756. When computer 1702 is connected to an associated cloud storage system, external storage interface 1726 can manage the storage provided by the cloud storage system by means of adapter 1758 and / or modem 1760, just like other types of external storage devices. For example, external storage interface 1726 can be configured to provide access to cloud storage sources as if these sources were physically connected to computer 1702.

[0184] Computer 1702 may be operable to communicate with any wirelessly positioned device or entity, such as a printer, scanner, desktop and / or laptop computer, portable data assistant, communications satellite, any device or location associated with a wirelessly detectable tag (e.g., a self-service machine, newsstand, store shelf, etc.), and telephone. This may include Wi-Fi and Wireless technology. Therefore, communication can be a predefined structure like a regular network, or simply self-organizing communication between at least two devices.

[0185] Figure 18 This is a schematic block diagram of a sample computing environment 1800 with which the disclosed subject matter can interact. The sample computing environment 1800 includes one or more clients 1810. Clients 1810 can be hardware and / or software (e.g., threads, processes, computing devices). The sample computing environment 1800 also includes one or more servers 1830. Servers 1830 can also be hardware and / or software (e.g., threads, processes, computing devices). For example, server 1830 can accommodate threads to perform transformations by employing one or more embodiments described herein. One possible communication between client 1810 and server 1830 can be in the form of data packets suitable for transmission between two or more computer processes. The sample computing environment 1800 includes a communication framework 1850 that can be used to facilitate communication between client 1810 and server 1830. Client 1810 is operatively connected to one or more client data repositories 1820, which can be used to store information local to client 1810. Similarly, server 1830 is operatively connected to one or more server data repositories 1840, which can be used to store information locally on server 1830.

[0186] This invention can be a system, method, apparatus, and / or computer program product at any possible level of integrated technical detail. A computer program product may include a computer-readable storage medium (or media) having computer-readable program instructions thereon for causing a processor to execute aspects of the invention. The computer-readable storage medium may be a tangible device that can hold and store instructions for use by an instruction execution device. The computer-readable storage medium may be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A less complete list of more specific examples of computer-readable storage media may also include: portable computer floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable optical disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punch cards or protrusions in grooves on which instructions are recorded), and any suitable combination of the foregoing. As used herein, a computer-readable storage medium should not be construed as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0187] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them for storage in a computer-readable storage medium within the corresponding computing / processing device. The computer-readable program instructions used to perform the operations of this invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, integrated circuit configuration data, or source code or object code written in any combination of one or more programming languages ​​(including object-oriented programming languages ​​such as Smalltalk, C++, etc.) and procedural programming languages ​​(such as the "C" programming language or similar programming languages). Computer-readable program instructions can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or can establish a connection with an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) can execute computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of the invention by utilizing state information of the computer-readable program instructions.

[0188] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions. These computer-readable program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, executable via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium that can direct a computer, programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of manufacture comprising instructions implementing aspects of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus or other equipment to cause a series of operations to be performed on the computer, other programmable apparatus or other equipment to produce a computer-implemented process, such that the instructions that execute on the computer, other programmable apparatus or other equipment implement the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0189] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible specific implementations of systems, methods, and computer program products according to various embodiments of the invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of instructions comprising one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than indicated in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or sometimes they may be executed in reverse order, depending on the functionality involved. It will also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified function or action or implements a combination of dedicated hardware and computer instructions.

[0190] Although the subject matter has been described above in the general context of computer-executable instructions of a computer program product running on one or more computers, those skilled in the art will recognize that the present disclosure may also be implemented in combination with other program modules. Generally, program modules include routines, programs, components, data structures, etc., that perform specific tasks and / or implement specific abstract data types. Furthermore, those skilled in the art will understand that other computer system configurations can be used to practice the computer implementation methods of the present invention, including single-processor or multi-processor computer systems, small computing devices, mainframe computers, and computers, handheld computing devices (e.g., PDAs, telephones), microprocessor-based or programmable consumer or industrial electronic devices, etc. The illustrated aspects can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices linked via a communication network. However, some (if not all) aspects of the present disclosure can be practiced on a standalone computer. In a distributed computing environment, program modules can reside in both local and remote memory storage devices.

[0191] As used herein, the terms “component,” “system,” “platform,” “interface,” etc., may refer to and / or include computer-related entities or entities associated with an operator having one or more specific functionalities. Entities disclosed herein may be hardware, a combination of hardware and software, software, or software in execution. For example, a component may be, but is not limited to, a program running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. By way of description, applications running on a server and servers may both be components. One or more components may reside within a program and / or an execution thread, and components may be located on a single computer and / or distributed across two or more computers. Furthermore, a corresponding component may execute on various computer-readable media on which various data structures are stored. Components may communicate via local and / or remote processes, such as based on signals having one or more data packets (e.g., data from one component that interacts with another component in a local system, a distributed system, and / or a network (such as the Internet with other systems). For example, a component can be a device having specific functions provided by mechanical parts operated by an electrical or electronic circuitry system, which is operated by software or firmware applications executed by a processor. In such cases, the processor may be internal or external to the device and may execute at least a portion of the software or firmware application. As yet another example, a component can be a device that provides specific functionality through electronic components rather than mechanical parts, wherein the electronic components may include a processor or other means for executing software or firmware that at least partially endows the electronic components with functionality. In one aspect, a component may be emulated, for example, via a virtual machine within a cloud computing system.

[0192] Furthermore, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X adopts A or B" is intended to mean any natural inclusive substitution. That is, if X adopts A; X adopts B; or X adopts both A and B, then "X adopts A or B" is satisfied in any of the foregoing cases. Additionally, unless otherwise specified or clear from the context to be directed to the singular form, the articles "a" and "an" used in this specification and accompanying drawings should generally be understood to mean "one or more." As used herein, the terms "example" and / or "exemplary" are used to indicate what is intended as an example, instance, or illustration. For the avoidance of doubt, the subject matter disclosed herein is not limited to such examples. Furthermore, any aspect or design described herein as "example" and / or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs, nor does it exclude equivalent exemplary structures and techniques known to those skilled in the art.

[0193] As used herein, the term "processor" can refer substantially to any computing processing unit or device, including but not limited to a single-core processor; a single processor with software multithreading capabilities; a multi-core processor; a multi-core processor with software multithreading capabilities; a multi-core processor with hardware multithreading technology; a parallel platform; and a parallel platform with distributed shared memory. Additionally, a processor can refer to an integrated circuit, an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic controller (PLC), a complex programmable logic device (CPLD), discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. Furthermore, a processor can utilize nanoscale architectures (such as, but not limited to, molecular and quantum dot-based transistors, switches, and gates) to optimize space usage or enhance the performance of user equipment. A processor can also be implemented as a combination of computing processing units. In this disclosure, terms such as "repository," "storage device," "data repository," "data storage device," "database," and substantially any other information storage component related to the operation and functionality of a component are used to refer to a "memory component," an entity specifically embodied in "memory," or a component that includes memory. It should be understood that the memory and / or memory components described herein can be volatile or non-volatile memory, or may include both volatile and non-volatile memory. By way of illustration and not limitation, non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable ROM (EEPROM), flash memory, or non-volatile random access memory (RAM) (e.g., ferroelectric RAM (FeRAM)). For example, volatile memory may include RAM that can act as external cache memory. By way of illustration and not limitation, RAM can be provided in a variety of forms, such as synchronous RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), direct Rambus RAM (DRRAM), direct Rambus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM). Furthermore, the memory components disclosed in the systems or computer-implemented methods described herein are intended to include, but are not limited to, these and any other suitable types of memory.

[0194] The foregoing description includes only examples of systems and computer-implemented methods. Of course, it is impossible to describe every conceivable combination of components or computer-implemented methods for the purposes of describing this disclosure, but those skilled in the art will recognize that many other combinations and substitutions of this disclosure are possible. Furthermore, regarding the extent to which the terms “comprising,” “having,” “possessing,” etc., are used in the detailed description, claims, appendices, and drawings, such terms are intended to be inclusive in a manner similar to the term “comprising,” as interpreted when “comprising” is used as a transitional word in the claims.

[0195] Various embodiments have been described for illustrative purposes, but these descriptions are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best illustrate the principles of the embodiments, the practical application of or improvement of technology found in the market, or to enable others skilled in the art to understand the disclosed embodiments.

[0196] Further aspects of various embodiments of the subject matter innovation protected by the claims are provided in the following subjects:

[0197] 1. A system comprising: a processor executing computer-executable components stored in a computer-readable storage memory, the computer-executable components comprising: a receiver component accessing a first set of data candidates and a second set of data candidates, wherein a machine learning model is trained on the first set of data candidates; an activation component obtaining a first set of latent activations generated by the machine learning model based on the first set of data candidates, and obtaining a second set of latent activations generated by the machine learning model based on the second set of data candidates; a compression component generating a first set of compressed data points by applying a dimensionality reduction technique to the first set of latent activations, and generating a second set of compressed data points by applying the dimensionality reduction technique to the second set of latent activations; and a diversity component calculating a diversity score based on the first set of compressed data points and the second set of compressed data points.

[0198] 2. The system according to any of the foregoing provisions, wherein the computer-executable component further comprises: a visualization component that presents a graph depicting the first set of compressed data points and the second set of compressed data points.

[0199] 3. The system according to any of the foregoing provisions, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and wherein the visualization component presents the given data candidate in response to the given compressed data point being clicked.

[0200] 4. The system according to any of the foregoing provisions, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and wherein the computer-executable component further comprises: an execution component that recommends training the machine learning model on the second set of data candidates in response to determination that the diversity score meets a predetermined threshold, and recommends not training the machine learning model on the second set of data candidates in response to determination that the diversity score fails to meet the predetermined threshold.

[0201] 5. The system according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and wherein the computer-executable component further comprises: an execution component that identifies one or more peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points, and recommends manually annotating one or more data candidates from the second set of data candidates corresponding to the one or more peripheral compressed data points.

[0202] 6. The system according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and wherein the computer-executable component further comprises: an execution component that identifies one or more non-peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points, and recommends automatically annotating one or more data candidates from the second set of data candidates corresponding to the one or more non-peripheral compressed data points.

[0203] 7. The system according to any of the foregoing clauses, wherein the dimensionality reduction technique is t-distributed random neighbor embedding (tSNE) or uniform manifold approximation and projection (UMAP).

[0204] 8. A computer-implemented method comprising: accessing a first set of data candidates and a second set of data candidates by a device operatively coupled to a processor, wherein a machine learning model is trained on the first set of data candidates; obtaining a first set of latent activations generated by the machine learning model based on the first set of data candidates by the device, and obtaining a second set of latent activations generated by the machine learning model based on the second set of data candidates by the device; generating a first set of compressed data points by the device by applying a dimensionality reduction technique to the first set of latent activations, and generating a second set of compressed data points by the device by applying the dimensionality reduction technique to the second set of latent activations; and calculating a diversity score by the device based on the first set of compressed data points and the second set of compressed data points.

[0205] 9. The computer-implemented method according to any of the foregoing clauses further includes: the device presenting a graph depicting the first set of compressed data points and the second set of compressed data points.

[0206] 10. A computer-implemented method according to any of the foregoing provisions, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and further comprising: presenting the given data candidate by the device in response to a click on the given compressed data point.

[0207] 11. A computer-implemented method according to any of the foregoing provisions, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and further comprising: recommending, by the device, to train the machine learning model on the second set of data candidates in response to determination that the diversity score meets a predetermined threshold; and recommending, by the device, not to train the machine learning model on the second set of data candidates in response to determination that the diversity score fails to meet the predetermined threshold.

[0208] 12. A computer-implemented method according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and further comprising: identifying one or more peripheral compressed data points in the second set of compressed data points by the device by comparing the second set of compressed data points with the first set of compressed data points; and recommending, by the device, manually annotating one or more data candidates from the second set of data candidates corresponding to the one or more peripheral compressed data points.

[0209] 13. A computer-implemented method according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and further comprising: identifying one or more non-peripheral compressed data points in the second set of compressed data points by the device by comparing the second set of compressed data points with the first set of compressed data points; and recommending one or more data candidates corresponding to the one or more non-peripheral compressed data points from the second set of data candidates by the device.

[0210] 14. The computer-implemented method according to any of the foregoing clauses, wherein the dimensionality reduction technique is a t-distributed random neighbor embedding (tSNE) technique or a uniform manifold approximation and projection (UMAP) technique.

[0211] 15. A computer program product for facilitating the visualization and quantification of data diversity for a machine learning model, the computer program product comprising a computer-readable storage device having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to: access a first set of data candidates and a second set of data candidates, wherein a machine learning model is trained on the first set of data candidates; obtain a first set of latent activations generated by the machine learning model based on the first set of data candidates, and obtain a second set of latent activations generated by the machine learning model based on the second set of data candidates; generate a first set of compressed data points by applying a dimensionality reduction technique to the first set of latent activations, and generate a second set of compressed data points by applying the dimensionality reduction technique to the second set of latent activations; and calculate a diversity score based on the first set of compressed data points and the second set of compressed data points.

[0212] 16. A computer program product according to any of the foregoing provisions, wherein the program instructions are also executable to cause the processor to: present a graph depicting the first set of compressed data points and the second set of compressed data points.

[0213] 17. A computer program product according to any of the foregoing provisions, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and wherein the processor presents the given data candidate in response to the given compressed data point being clicked.

[0214] 18. A computer program product according to any of the foregoing provisions, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and wherein the program instructions are also executable to cause the processor to: recommend training the machine learning model on the second set of data candidates in response to determination that the diversity score meets a predetermined threshold; and recommend not training the machine learning model on the second set of data candidates in response to determination that the diversity score fails to meet the predetermined threshold.

[0215] 19. A computer program product according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and wherein the program instructions are also executable to cause the processor to: identify one or more peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points; and recommend manually annotating one or more data candidates from the second set of data candidates corresponding to the one or more peripheral compressed data points.

[0216] 20. A computer program product according to any of the foregoing provisions, wherein the second set of data candidates is unannotated, and wherein the program instructions are also executable to cause the processor to: identify one or more non-peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points; and recommend automatically annotating one or more data candidates from the second set of data candidates corresponding to the one or more non-peripheral compressed data points.

Claims

1. A system for facilitating the visualization and quantification of data diversity for machine learning models, the system comprising: A processor that executes computer-executable components stored in a computer-readable storage memory, the computer-executable components comprising: A receiver component accesses a first set of data candidates and a second set of data candidates, wherein a machine learning model is trained on the first set of data candidates, the first set of data candidates and the second set of data candidates are medical images, and the machine learning model is configured to receive the medical images as input and output labels for classifying the medical images; An activation component, wherein the activation component obtains a first set of potential activations generated by the machine learning model based on the first set of data candidates, and obtains a second set of potential activations generated by the machine learning model based on the second set of data candidates; A compression component that generates a first set of compressed data points by applying a dimensionality reduction technique to a first set of latent activations, and generates a second set of compressed data points by applying the dimensionality reduction technique to a second set of latent activations; and A diversity component calculates a diversity score based on a first set of compressed data points and a second set of compressed data points, wherein the diversity score quantifies the degree to which the second set of compressed data points conforms to the clustering exhibited by the first set of compressed data points, and wherein calculating the diversity score includes: The compressed data points in the first group are grouped into clusters; For each cluster, a threshold distance from the center of the cluster is determined based on the density and distribution of the compressed data points in the cluster. Distribute the compressed data points from the second group of compressed data points to the cluster; and The diversity score is determined based on the number of times a corresponding compressed data point in the second set of compressed data points exceeds a corresponding threshold distance from the corresponding center of the assigned cluster.

2. The system of claim 1, wherein the computer-executable component further comprises: A visualization component that presents a graph depicting the first set of compressed data points and the second set of compressed data points.

3. The system of claim 2, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and wherein the visualization component presents the given data candidate in response to the given compressed data point being clicked.

4. The system of claim 1, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and wherein the computer-executable component further comprises: An execution unit, in response to the determination that the diversity score meets a predetermined threshold, recommends training the machine learning model on the second set of data candidates, and in response to the determination that the diversity score fails to meet the predetermined threshold, recommends not training the machine learning model on the second set of data candidates.

5. The system of claim 1, wherein the second set of data candidates is unannotated, and wherein the computer-executable component further comprises: An execution component identifies one or more peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points, and recommends manually annotating one or more data candidates from the second set of data candidates that correspond to the one or more peripheral compressed data points.

6. The system of claim 1, wherein the second set of data candidates is unannotated, and wherein the computer-executable component further comprises: An execution unit identifies one or more non-peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points, and recommends automatically annotating one or more data candidates from the second set of data candidates that correspond to the one or more non-peripheral compressed data points.

7. The system according to claim 1, wherein the dimensionality reduction technique is t-distributed random neighbor embedding (tSNE) technique or uniform manifold approximation and projection (UMAP) technique.

8. A computer-implemented method for facilitating the visualization and quantification of data diversity for machine learning models, the computer-implemented method comprising: A device operatively coupled to a processor accesses a first set of data candidates and a second set of data candidates, wherein a machine learning model is trained on the first set of data candidates, the first set of data candidates and the second set of data candidates are medical images, and the machine learning model is configured to receive the medical images as input and output labels classifying the medical images; The device obtains a first set of potential activations generated by the machine learning model based on the first set of data candidates, and the device obtains a second set of potential activations generated by the machine learning model based on the second set of data candidates; The device generates a first set of compressed data points by applying dimensionality reduction techniques to the first set of potential activations, and the device generates a second set of compressed data points by applying the dimensionality reduction techniques to the second set of potential activations; as well as The device calculates a diversity score based on the first set of compressed data points and the second set of compressed data points, wherein the diversity score quantifies the degree to which the second set of compressed data points conforms to the clustering exhibited by the first set of compressed data points, and wherein calculating the diversity score includes: The compressed data points in the first group are grouped into clusters; For each cluster, a threshold distance from the center of the cluster is determined based on the density and distribution of the compressed data points in the cluster. Distribute the compressed data points from the second group of compressed data points to the cluster; and The diversity score is determined based on the number of times a corresponding compressed data point in the second set of compressed data points exceeds a corresponding threshold distance from the corresponding center of the assigned cluster.

9. The computer-implemented method according to claim 8, further comprising: The device presents a graph depicting the first set of compressed data points and the second set of compressed data points.

10. The computer-implemented method of claim 9, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and further comprising: The device presents the given data candidate in response to a click on the given compressed data point.

11. The computer-implemented method of claim 8, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and further comprises: In response to the determination that the diversity score meets a predetermined threshold, the device recommends training the machine learning model on the second set of data candidates; as well as In response to the determination that the diversity score fails to meet the predetermined threshold, the device recommends not to train the machine learning model on the second set of data candidates.

12. The computer-implemented method of claim 8, wherein the second set of data candidates is unannotated, and further comprises: The device identifies one or more peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points; as well as The device recommends manually annotating one or more data candidates from the second set of data candidates that correspond to the one or more peripheral compressed data points.

13. The computer-implemented method of claim 8, wherein the second set of data candidates is unannotated, and further comprises: The device identifies one or more non-peripheral compressed data points in the second set of compressed data points by comparing the second set of compressed data points with the first set of compressed data points; as well as The device recommends and automatically annotates one or more data candidates from the second set of data candidates that correspond to the one or more non-peripheral compressed data points.

14. The computer-implemented method according to claim 8, wherein the dimensionality reduction technique is t-distributed random neighbor embedding (tSNE) or uniform manifold approximation and projection (UMAP).

15. A computer program product for facilitating the visualization and quantification of data diversity for machine learning models, the computer program product comprising a computer-readable storage medium having program instructions embodied therein, the program instructions being executable by a processor to cause the processor to: Access a first set of data candidates and a second set of data candidates, wherein a machine learning model is trained on the first set of data candidates, the first set of data candidates and the second set of data candidates are medical images, and the machine learning model is configured to receive the medical images as input and output labels for classifying the medical images; A first set of potential activations is obtained by the machine learning model based on the first set of data candidates, and a second set of potential activations is obtained by the machine learning model based on the second set of data candidates; A first set of compressed data points is generated by applying dimensionality reduction techniques to the first set of potential activations, and a second set of compressed data points is generated by applying the dimensionality reduction techniques to the second set of potential activations. as well as A diversity score is calculated based on the first set of compressed data points and the second set of compressed data points, wherein the diversity score quantifies the degree to which the second set of compressed data points conforms to the clustering exhibited by the first set of compressed data points, and wherein calculating the diversity score includes: The compressed data points in the first group are grouped into clusters; For each cluster, a threshold distance from the center of the cluster is determined based on the density and distribution of the compressed data points in the cluster. Distribute the compressed data points from the second group of compressed data points to the cluster; and The diversity score is determined based on the number of times a corresponding compressed data point in the second set of compressed data points exceeds a corresponding threshold distance from the corresponding center of the assigned cluster.

16. The computer program product of claim 15, wherein the program instructions are further executable to cause the processor to: A graph depicting the first set of compressed data points and the second set of compressed data points is presented.

17. The computer program product of claim 16, wherein a given compressed data point plotted on the graph corresponds to a given data candidate from the first set of data candidates or the second set of data candidates, wherein the given compressed data point is clickable as plotted on the graph, and wherein the processor presents the given data candidate in response to the given compressed data point being clicked.

18. The computer program product of claim 15, wherein the second set of data candidates represents an enhanced version of the first set of data candidates or otherwise represents potential training data for the machine learning model, and wherein the program instructions are further executable to cause the processor to: In response to the determination that the diversity score meets a predetermined threshold, it is recommended to train the machine learning model on the second set of data candidates; and In response to the determination that the diversity score fails to meet the predetermined threshold, it is recommended not to train the machine learning model on the second set of data candidates.

19. The computer program product of claim 15, wherein the second set of data candidates is uncommented, and wherein the program instructions are further executable to cause the processor to: One or more peripheral compressed data points in the second set of compressed data points are identified by comparing the second set of compressed data points with the first set of compressed data points; and It is recommended to manually annotate one or more data candidates from the second group of data candidates that correspond to the one or more peripheral compressed data points.

20. The computer program product of claim 15, wherein the second set of data candidates is uncommented, and wherein the program instructions are further executable to cause the processor to: One or more non-peripheral compressed data points in the second set of compressed data points are identified by comparing the second set of compressed data points with the first set of compressed data points; and It is recommended to automatically annotate one or more data candidates from the second group of data candidates that correspond to the one or more non-peripheral compressed data points.

Images