Counting on rare training data for artificial intelligence

Abstract

A system for enhancing classifier predictions for underrepresented categories can be provided. A classifier system trained on training data to build a model is used to classify unknown input data, and an evaluator engine is adapted to determine the underrepresented class. Additionally, the system includes an extractor engine adapted to extract relevant data from another source, and a similarity engine adapted to select a dataset from the relevant data, wherein the similarity engine is also adapted to compare features of the relevant data with features of a representative dataset for the underrepresented category. Finally, the system includes a recursive unit adapted to trigger the evaluator engine, which, along with the similarity engine, takes the selected dataset as input data until the evaluator engine classifies the selected dataset with a confidence level above a confidence threshold.

Description

Background Technology

[0001] This invention generally relates to a system for better machine learning, and more specifically to a system for enhancing classifier predictions about underrepresented categories. The invention further relates to a method for enhancing classifier predictions and a computer program product.

[0002] Developing software and machine learning tools in the field of artificial intelligence (AI) and integrating them into existing applications is a key challenge for the IT industry today. Over the past few years, development has increasingly become an integral part of existing components that will continue to evolve in a paradigm shift in how algorithms are developed—programming becomes labeling, and coding is replaced by enabling computers to solve optimization problems represented through AI, neural networks, and specifically deep learning. This phenomenon can be termed "Software 1.0 vs. Software 2.0 development." Therefore, there is a growing need for IDEs (Integrated Development Environments) to provide the same support for Software 2.0 development (such as concepts and software stacks) as Software 1.0 can be used to develop enterprise-level AI solutions.

[0003] One of the pain points in the industry is labeling images used to train systems or methods: (i) labeling images is extremely time-consuming, making automation necessary; and (ii) rare cases can lead to problems and biases in the system or method that must be addressed. Manual labeling is a huge burden for data scientists because it is primarily a manual process requiring highly skilled personnel who may, on the other hand, not be immune to skewed views.

[0004] A comparison of the time spent on a typical doctoral dissertation with real-world usage reveals the dilemma. Doctoral students spend approximately 5% to 10% of their time on datasets and a massive 90% or more on models and algorithms. In real-world use cases at companies like Tesla, approximately 75% of project time is spent on data preparation and datasets, and only 25% on models and algorithms (source: https: / / vimeo.com / 272696002).

[0005] One of these problems lies in the realm of rare cases, where these are examples of image categories or situations that appear in very small numbers, especially in the training dataset, or are shown very few times compared to other situations. Some examples could be: (i) images of vehicle accidents compared to normal street conditions, (ii) images of traffic lights with amber or even blue colors compared to green or red, (iii) distinctive street signs, and (iv) snowy or foggy conditions on the street.

[0006] The problem with trained machine learning models can be this: if the training set has 9,999 people with brown eyes and 1 person with blue eyes, the algorithm might always predict brown eyes. This results in a very high accuracy of 99.99%, but never predicts "blue eyes." If rare cases are ignored, this limitation can cause major problems in many settings, such as street traffic.

[0007] Several disclosures exist concerning systems or methods for enhancing classifier predictions for underrepresented categories. The document "A Systematic Study of Online Class Imbalance Learning with Concept Drift," IEEE Transactions on Neural Networks and Learning Systems (Vol. 29, No. 10, October 2018), discloses a method for handling concept drift in class-imbalanced data streams by learning underrepresented categories. This reference also discloses a method for preserving categories based on a window with up-to-date examples.

[0008] The document "Class Imbalance and Active Learning" (2013) by Josh Attenberg et al. (https: / / pdfs.semanticscholar.org) discloses a method for enabling models that prevent the resulting model from being biased towards the majority class and reflect the true properties of the minority class. This reference also discloses a method for addressing underrepresented classes by retraining the model using examples.

[0009] Although references have been mentioned, and in order to address the industry challenges in machine learning today, it may be necessary to focus more on rare case management and how to handle them if the available training data does not have a good mix of examples from which to be learned. Summary of the Invention

[0010] According to one aspect of the invention, a system for enhancing classifier predictions for underrepresented categories can be provided. The system may include a classifier system trained with training data to build a model for classifying unknown input data into predicted categories, an evaluator engine adapted to determine underrepresented categories within the predicted categories, wherein the evaluator engine may use a confidence distribution of the predictions, and an extractor engine adapted to extract relevant data from additional sources, wherein the relevant data may be correlated with the determined underrepresented categories.

[0011] Furthermore, the system may include a similarity engine adapted to select datasets from the relevant data. Thus, the similarity engine may also be adapted to compare features of the relevant data with features of a representative dataset of underrepresented categories, and a recursive unit adapted to trigger the evaluator engine, the extractor engine, and the similarity engine, taking the selected dataset as input data, until the evaluator engine classifies the selected dataset with a confidence level higher than a confidence threshold level.

[0012] According to another aspect of the invention, a computer-implemented method for enhancing classifier predictions for underrepresented categories is provided. The method may include: providing a classifier system trained using training data to construct a model that can be used to classify unknown input data into predicted categories; determining underrepresented categories within the predicted categories by determining a confidence level distribution of the predictions; and extracting relevant data from an additional source, said relevant data being correlated with the determined underrepresented categories.

[0013] Furthermore, the method may include selecting a dataset from the relevant data by comparing features of the relevant data with features of a representative dataset for an underrepresented category, and repeating the evaluation, extraction, and selection by using the selected dataset as input data until the selected dataset is classified with a confidence level higher than a confidence threshold level.

[0014] The proposed system for enhancing classifier predictions for underrepresented categories offers several advantages and technical benefits.

[0015] Clearly, better management of rare cases in machine learning can be achieved. This system and related methods are concerned with imbalanced training datasets. If, for example, a classifier's predictions are uncertain about their own outcome, the proposed technique can supervise itself and care about itself, autonomously finding alternatives. An internal evaluator detects such imbalances and can automatically find other examples to augment the training dataset. This can be viewed as an inside-out approach, as the system can determine that the training dataset may not be good enough and decides to go outside the system—that is, outside the available data for training and searching for alternative sources. Therefore, the system can "bypass" the "trainer" who has already provided the training dataset and can determine that "self-experience" might be better.

[0016] For this purpose, external databases—typically available via the Internet—can be used and accessed to find out (determine) whether the discovered data can be used for better training of existing machine learning algorithms.

[0017] Therefore, it is possible to start with a fairly basic training dataset and allow the system to find and augment its own training dataset. Thus, the requirement for labeled training data is reduced, thanks to the automatic labeling process that can be performed as a side effect of the concepts presented here.

[0018] The similarity engine used in this concept can be used to locate images containing numerous similar features among multiple images. The algorithm takes images as input and can give a rating from 0 to 1, where higher values ​​indicate a better match between images. Image processing can be based on image analysis, i.e., finding distinguishable features through image thresholding, image gradients, and convolution. First, features can be computed in the image, then the values ​​and quantities of the images are compared and merged to collect values ​​for all foreseeable image pairs.

[0019] Newly discovered images (which are subsequently predicted with high confidence) can be added to the training set to improve the general multi-class model over time. The label can be derived from a category that may have been underrepresented and may have already triggered a full loop for “going out”.

[0020] In the following sections, additional embodiments of the inventive concept will be described—applicable to both the system and the method.

[0021] According to a useful embodiment of the system, the input data can be an image. Currently, this is perhaps the most prevalent use case; however, the system and related methods can be applied to any other data type. Therefore, the proposed concept is not limited to image data but can be applied to any type of machine learning.

[0022] According to another useful embodiment of the system, image data can be extracted from video data. This can significantly increase the number of available images, since video can be viewed as a simple cascade of still images.

[0023] According to a preferred embodiment of the system, determining an underrepresented class can be based on a function of the difference between the predicted class and other classes. If, for example, 9 out of 10 classes are always predicted in 98+% of all predictions, this could be for two reasons: (i) the input data to the classifier may be biased, i.e., the input data may have no reason to predict the 10th class, or (ii) the trained model on which the classifier is based does not "see" the last class because it is not reflected accordingly in the training dataset.

[0024] According to a preferred embodiment of the system, the supplementary data can be data external to both the training data and the regular input data. Therefore, the system not only relies on the usage data used for training but can also attempt to access data “outside the system” (i.e., “beyond its own nose”). Supplementary data can be found in, for example, a digital library for supplementary images or a dictionary for supplementary text instances. Other external sources are readily conceivable. One way to access external data may also include the option of integrating access to a search engine to search for supplementary external data via the Internet. In another embodiment, the search may be limited to an intranet.

[0025] According to a preferred embodiment, the system may further include a labeling unit adapted to label the selected dataset as an additional training dataset if the evaluator classifies the selected dataset with a confidence level above a confidence threshold level. Thus, an automatic determination can be made to augment the existing training dataset with newly discovered data (i.e., the training dataset used for actual training) outside the scope of the existing experience to date, thereby expanding the training dataset to serve as a broader truth repository.

[0026] According to another preferred embodiment of the system, the labeling unit can also be adapted to annotate the supplementary training dataset with labels associated with underrepresented categories. The supplementary training dataset may include one or more instances of supplementary training data, for which automatic labeling or tagging of data to be newly integrated into the training dataset can be used. Thus, manual interaction with the machine learning system to broaden its experience base, i.e., the training dataset, may not be necessary. Furthermore, if a predefined number or amount of supplementary training data may become available, subsequent retraining of the machine learning system can be automatically triggered.

[0027] According to an alternative embodiment of the system, the recursive unit can also be adapted to stop triggering if a predefined number of recursions have occurred. This prevents the search process from infinitely looping to additional external data. Thus, unnecessary use of system resources can be avoided.

[0028] According to a further optional embodiment of the system, the evaluator engine and / or extractor engine can be connected to the knowledge graph system. The evaluator and / or extractor engines can use the data organization in the knowledge graph system to determine which additional external data might be useful for further investigation into whether the data is useful for the machine learning process.

[0029] Furthermore, embodiments may take the form of a related computer program product accessible from a computer-usable or computer-readable medium that provides program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium may be any device that can contain programs for storing, communicating, propagating, or transmitting programs for use by or in connection with an instruction execution system, device, or apparatus. Attached Figure Description

[0030] It should be noted that embodiments of the invention have been described with reference to different subjects. In particular, some embodiments are described with reference to method type claims, while others are described with reference to apparatus type claims. However, those skilled in the art will understand from the above and the following description that, unless otherwise stated, any combination of features relating to different subjects, in particular any combination of features belonging to one type of subject matter, is considered to be disclosed within this document, in addition to any combination of features belonging to one type of subject matter.

[0031] The foregoing limitations and other aspects of the invention will become clear from the examples of the embodiments to be described below and will be explained with reference to these examples, but the invention is not limited to these embodiments.

[0032] Preferred embodiments of the invention will be described by way of example only and with reference to the following figures:

[0033] Figure 1 A block diagram of one embodiment of the inventive system for augmented classifier prediction of underrepresented categories is shown.

[0034] Figure 2 A block diagram illustrating a simplified embodiment of the proposed concept is shown.

[0035] Figure 3 A block diagram of an embodiment of a more complete system is shown.

[0036] Figure 4 A block diagram illustrating the flowchart of the present invention’s method for augmenting classifier predictions for underrepresented categories is shown.

[0037] Figure 5 It shows including according to Figure 1 An example of a computing system. Detailed Implementation

[0038] In the context of this specification, the following conventions, terms, and / or expressions may be used:

[0039] The term "enhanced classifier prediction" can refer to the process of helping a classifier system make better predictions (i.e., classifying input data more accurately into the predicted class). This can be done particularly for rare cases, i.e., those situations that may only occur infrequently and may underperform on the training dataset.

[0040] The term "underrepresented category" can refer to a category of a classifier that may have been more or less never predicted—or nearly never predicted—because the underlying machine learning model has not been trained sufficiently for that category, since the training dataset has too few examples of underrepresented categories.

[0041] The term "classifier system" can refer to neural networks, deep neural networks, random forest systems, logistic regression systems, Gaussian regression systems, and other forms of machine learning systems and / or models. Any classifier can be used. The concept of a classifier system can refer to a trained classifier that has already been trained with a training dataset.

[0042] The term "evaluator engine" can refer to a unit capable of determining whether cases in input data are likely related to rare cases. For this purpose, the evaluator engine can use different thresholding methods. One could be a relative thresholding method, which compares the confidence levels resulting from predictions across multiple categories to confidence levels that have small, comparable differences from each other. Therefore, the "confidence level distribution" can be used as a basis for determination.

[0043] The term "extractor engine" can refer to a unit of the proposed system and has access to external data sources, which, in a predefined sense, can be comparable to rare case examples. A search engine can be used by the extractor engine—or can be part of it—to identify datasets related to the rare case dataset. Based on this, the "similarity engine" can also determine, based on a threshold, whether the similarity between additional examples identified by the extractor engine is sufficient to be relevant to the rare cases.

[0044] The term "additional source" can refer to a source of potential training data that may not yet have been used to develop machine learning models (i.e., classifiers). An additional source can belong to a pool of data independent of the training data available to date. An additional source can be a collection of libraries and / or repositories accessible via the internet or, in other cases, via the intranet of one or more enterprises.

[0045] The term "recursive unit" can refer to a unit that controls the triggering of other units in a predefined order. In this case, a recursive unit can trigger the evaluator engine, extractor engine, and similarity engine in sequence, at least until an exit condition is met.

[0046] In the following description, embodiments will be illustrated with reference to the accompanying drawings.

[0047] Figure 1 A block diagram of an embodiment of a system 100 for augmenting classifier predictions for underrepresented categories is shown. System 100 includes a classifier system 102 trained with training data to build a model that can be used to classify unknown input data into predicted categories. Examples of classifier systems may include neural networks—specifically deep neural networks—logistic regression systems, Gaussian regression systems, and random forest systems.

[0048] Furthermore, system 100 may include an evaluator engine 104 adapted to identify underrepresented categories within the predicted categories. For this purpose, evaluator engine 104 uses a distribution of confidence levels for the predictions. This can be implemented as a relative thresholding scheme. If, for a category, the confidence level for the prediction is relatively low and other categories can show relatively high values, this can be said to indicate uncertainty about a particular prediction. On the other hand, if the confidence level for the prediction can show 90% confidence and for other categories it can be in the single-digit range, then the confidence level is much higher than the confidence levels for other predicted categories, making this category potentially underrepresented.

[0049] Furthermore, system 100 may include an extractor engine 106 adapted to extract relevant data from additional sources (specifically external sources). Thus, the relevant data relates to identified underrepresented categories, i.e., to specific input data that is only vaguely classified. This is a remarkable aspect of machine learning systems, where the "experience" of the system can be extended, i.e., new training data can be pre-selected for further investigation.

[0050] To this end, method 100 also includes a similarity engine 108 adapted to select multiple datasets from the relevant data, wherein the similarity engine is further adapted to compare features of the relevant data with multiple features of a representative dataset that does not adequately represent the categories. Thus, the similarity engine searches for "things it already knows" and attempts to identify examples of categories available in only a limited number of training data sets in the machine learning training dataset. Specifically, a targeted search is performed to find training data that is initially unavailable but absolutely necessary for a reliable and realistic classifier system.

[0051] Last but not least, method 100 includes a recursive unit 110 adapted to at least trigger the evaluator engine, the extractor engine, and the similarity engine to treat the selected dataset as input data until the evaluator engine classifies the selected dataset with a confidence level above a confidence threshold. Therefore, the proposed concept does not rely on a “first shot” to expand the training dataset of the machine learning model (i.e., the classifier), but rather on a target-oriented search of a set of new data from which to be learned. For this purpose, the recursive unit 110 is connected to the evaluator engine 104, the extractor engine 106, and the similarity engine 108.

[0052] Figure 2 A block diagram illustrating a simplified embodiment of the proposed concept is shown. A trained classifier 202 can be regularly used to evaluate input data to perform classification. System 200 uses an evaluation process 204 to control its own predictive output, regardless of whether its own prediction is “good enough.” If the result is case “Y,” the classifier continues with its prediction unchanged. However, if the result is that evaluation 204 may not be good enough (case “N”), the system searches 206 for additional external data (“better data”), which can be used in new training of the underlying machine learning model of classifier 202. Whether a prediction is “good enough” can be determined by using a metric that compares the relative differences in class probabilities. So, for example, if a significant distance exists from other classes, then the prediction is accepted. The significant distance can be determined, for example, by a predefined threshold (percentage value or absolute value).

[0053] This simplified embodiment makes the concepts presented herein understandable. Thus, a self-optimizing machine learning system is proposed that attempts to expand its empirical range when the results are determined to be insufficient. The term "attempt to expand its empirical range" can be considered synonymous with expanding the data base used for prediction (through a new search of additional data) if a predefined threshold of difference between the primary prediction class and, for example, a second, other class is not reached.

[0054] Figure 3A block diagram of a more complete embodiment of system 300 is shown. The initial conditions can be described as follows: Classifier 302 can be trained using a multi-class model for n distinct categories. The q categories (q < n) are underrepresented in the training dataset, i.e., (number of samples in each category < total number of samples / (n * (|mean - median|))), also known as rare cases. Rare cases can be defined using different thresholds, and the |mean - median| factor is one example of an implementation. An inherent problem can be described as follows: during training, the classifier may learn the distribution in the training dataset, rather than the features that distinguish the q categories. Evaluator engine 304, rare case extractor 310, and similarity engine 316 can help address this "mistraining".

[0055] As an initial step 301, the image, along with any other available metadata, is fed into the classifier 302. It should be noted that, for simplicity, image data can be used to describe the instance. However, the proposed concept is applicable to any kind of classification (sound, text, video, health data, stock market data, to name just a few application areas).

[0056] The classifier's output (i.e., predictions) is fed to an evaluator engine 304, which determines whether a given case is a rare case. To do this, it uses a knowledge graph 314 to provide a good interpretation of the predicted class's confidence level within the context. Therefore—and bridging back to the term "good enough"—a rare case is identified if the evaluator engine determines that the difference between the main predicted class and the next predicted class in the ranked sequence of predicted classes is below a predefined difference threshold.

[0057] If the case is not rare, i.e., the confidence value is good enough (above the predefined confidence level threshold or the maximum number of iterations has been reached), then the classifier's output is valid (306) and is sent out (307). Additionally, the data related to this case can be used to periodically augment the training set, i.e., become part of the training data 308.

[0058] If the evaluator engine 306 determines that a case is a rare case, the input data is forwarded to the rare case extractor 310. This module is used to potentially amplify the corpus of training data with potentially related or similar images. These can be extracted from external data 312—particularly additional databases identified via search engines on the Internet or entirely external data—and can be sent to the similarity engine 316 to select the most similar images and feed them back to the classifier 302 for the next iteration (back to the beginning 301).

[0059] It can be noted that the rare case extractor can also access the knowledge graph to identify relevant images in the additional data 312.

[0060] In the next iteration, traffic lights can be used as an example input image for classifier 302. However, the image of a traffic light may be ambiguous and of an unclear kind (due to unusual perspective, fog, mist, light defects, etc.). As a result of classifier 302, it returns "stop light" with a confidence level of 0.712, "light bulb" with a confidence level of 0.709, "train stop light" with a confidence level of 0.698, and so on. Therefore, the system is "uncertain" about its predictions because there are similar but unequal predictions with fairly close confidence levels. Therefore, a relative measurement of confidence levels is used.

[0061] The evaluator engine 304 determines that this is a rare case. Then, as an example, the extractor 310 can—for example, by using a search engine to access the Internet—return 230 additional images by searching for stop lights, light bulbs, and train stop lights, i.e., a predefined number or percentage of the classifier 302's predictions whose confidence level difference is below a predetermined threshold.

[0062] Next, the similarity engine 316 can select, for example, five images from 230 that show relative proximity to the input image, and then send these images back to the classifier 302 for further processing as input data in the same way that normal input data for classification is fed into the classifier 302.

[0063] In further iterations, classifier 302 determines the categories of five additional images sent from similarity engine 316 to classifier 302. The result of the classifier can be:

[0064] - Image 1: "Stop Light", confidence level 0.812; "Light Bulb", confidence level 0.809; "Train Stop Light", confidence level 0.498;

[0065] - Image 2: "Stop light," confidence level 0.612; "Light bulb," confidence level 0.609; "Train stop light," confidence level 0.598; and so on for all identified and appended images. It can also be noted that the classifier is a trained classifier that has been trained with annotated training data to automatically determine the internal model to be used for classification. Typically, the number of categories is also given as input to the classifier (in addition to the images) for the remainder of the session.

[0066] Next, the evaluator engine 304 evaluates the confidence level of all five additional images. At least one image is not classified as a rare case and, because its relative confidence level is above a predetermined threshold, is sent as output to the right, i.e., to the valid output box 306. This image can then be used to augment the training dataset 308, accumulating more instances of rare cases in the training dataset 308. Retraining—which can occur whenever a predefined number of additional training data (absolute or relative) has been added to the training dataset 308—will now modify the underlying machine learning model of the classifier 302, thereby further enhancing the identification of rare cases in the future.

[0067] However, for the remaining images not added to the training dataset 306, the loop continues. For these four remaining images, the rare case extractor 310 can find an additional 350 images by accessing external data sources to search for "stop lights," etc. In this case, 7 images out of the 350 are selected as close to the input image and sent to another iterative loop of the system for further processing. This iterative process can continue until a predefined number of iterations have been performed.

[0068] Now, let's examine each unit in more detail: Evaluator engine 304 is used to determine whether the output of classifier 302 is a rare case (as compared above), in order to determine whether iterations in the main process are needed. The conditions used by evaluator engine 304 are based on the distribution of the predicted confidence levels. Specifically, a prediction is accepted if the most important class has a significant distance from other classes (specifically, the confidence level). This distance is controlled by a predetermined threshold. If no further iterations are performed in the main process and the output of classifier 302 is considered valid, a further exit condition is that a certain number of iterations have been performed in the main process.

[0069] Knowledge graphs can be used as a reference to better account for different prediction confidence levels. The closer two predictions are, the more likely they are to be true, and vice versa.

[0070] Extractor 310 (or rare case extractor) is used to access external data. The amount of available video material is increasing. This can come from, for example, "smart glasses" (goalkeepers), video cars (for supporting automated map generation of geographic locations), autonomous vehicles equipped with cameras, etc. Images constituting the video can be analyzed and classified. These classified images can then be compared to a database of "rare cases," i.e., images that are relevant to some machine learning categories but have only a few instances. This comparison can use techniques such as similarity learning and principal component analysis. Any images identified as part of a rare case category are then extracted and fed into the rare case database for use in training new and improved classifiers. Rare case categories can change over time; for example, once a sufficient number of cases have accumulated for a category, that category will no longer be considered rare. It can also be noted that the process will shift over time regarding which categories are considered to have rare cases. Therefore, it may be necessary to periodically update the "rare case" labels for categories. Depending on the hardware requirements of the system response, all images are used independently, one after another, or one every n frames. A natural consequence of using the process proposed in this paper is that as the training set becomes larger and processes more images, fewer and fewer rare cases will be identified in chronological order.

[0071] The similarity engine 316 is used to discover images that include digitally similar features between multiple images. Algorithms (e.g., SSIM (Structural Similarity) or PSNR (Peak Signal-to-Noise Ratio) algorithms) can take images as input and assign them a rating from 0 to 1, where higher values ​​indicate a better match between images. Image processing is based on image analysis, i.e., finding distinguishable features through image thresholding, image gradients, and convolution. First features are computed in the images, and then the values ​​of these images and the number of features are compared with and combined with the values ​​of all promising couplers.

[0072] New font images predicted with high confidence will be added to the training dataset to improve the general multi-class model over time.

[0073] Figure 4 A block diagram illustrating an embodiment of a computer-implemented method 100 for predicting underrepresented categories using an augmented classifier is shown. Method 100 includes: 402 providing a classifier system, trained using training data to construct a model that can be used to classify unknown input data into predicted categories, based on the relevant machine learning system explained above; 404 determining underrepresented categories within the predicted categories by determining a confidence level distribution of the predictions; and 406 extracting relevant data from additional sources. Thus, the relevant data is correlated with the determined underrepresented categories.

[0074] Additionally, method 400 includes 408 selecting a dataset from the relevant data by comparing features of the relevant data with features of a representative dataset of underrepresented categories, and 410 repeating determination 404, extraction 406, and selection 408 by using the selected dataset as input data until the selected dataset is classified with a confidence level above a confidence threshold. Therefore, steps 104, 106, and 108 are repeated until a stopping condition is met. The stopping condition can be a threshold for a specific prediction confidence level, or it can be seen in a predefined number of loops where searches for additional data have already been performed. If this is the case, the system can determine that the set "may not make sense" and search for additional data to augment the training dataset.

[0075] Embodiments of the present invention can be implemented with virtually any type of computer, regardless of the platform's suitability for storing and / or executing program code. For completeness reasons, Figure 5 As an example, a computing system 500 suitable for executing program code related to the proposed method is shown.

[0076] The computing system 500 is merely one example of a suitable computer system and is not intended to impose any limitation on the scope of use or functionality of the embodiments of the invention described herein, regardless of whether the computer system 500 is capable of implementing and / or performing any of the functions set forth above. Within the computer system 500, there are components that can operate with many other general-purpose or special-purpose computing system environments or configurations. Examples of well-known computing systems, environments, and / or configurations suitable for use with the computer system / server 500 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computers, and distributed cloud computing environments that include any of the aforementioned systems or devices, etc. The computer system / server 500 can be described in the general context of computer system executable instructions (such as program modules) executed by the computer system 500. Generally, program modules may include routines, programs, objects, components, logic, data structures, etc., that perform a specific task or implement a specific abstract data type. The computer system / server 500 can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can reside on both local and remote computer system storage media (including memory storage devices).

[0077] As shown in the figure, the computer system / server 500 is illustrated as a general-purpose computing device. Components of the computer system / server 500 may include, but are not limited to, one or more processors or processing units 502, system memory 504, and a bus 506 that couples the various system components, including system memory 504, to the processor 502. Bus 506 represents one or more of several types of bus architectures, including memory buses or memory controllers, peripheral buses, accelerated graphics ports, and processor or local buses using any of a variety of bus architectures. By way of example and not limitation, such architectures include Industry Standard Architecture (ISA) buses, Micro Channel Architecture (MCA) buses, Enhanced ISA (EISA) buses, Video Electronics Standards Association (VESA) local buses, and Peripheral Component Interconnect (PCI) buses. The computer system / server 500 typically includes a variety of computer system readable media. Such media can be any available media accessible by the computer system / server 500, and it includes both volatile and non-volatile media, removable and non-removable media.

[0078] System memory 504 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 508 and / or cache memory 510. Computer system / server 500 may also include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 512 may be provided for reading from and writing to a non-removable, non-volatile magnetic medium (not shown, and generally referred to as a "hard disk drive"). Although not shown, disk drives for reading from and writing to removable non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable non-volatile optical disks (such as CD-ROMs, DVD-ROMs, or other optical media) may be provided. In such instances, each may be connected to bus 506 via one or more data media interfaces. As will be further described and depicted below, memory 504 may include at least one program product having a set (e.g., at least one) of program modules configured to perform the functions of embodiments of the invention.

[0079] A program / utility having a set (at least one) of program modules 516 may be stored (as an example and not a limitation) in memory 504, as well as in an operating system, one or more applications, other program modules, and program data. Each of the operating system, one or more applications, other program modules, and program data, or a combination thereof, may comprise an embodiment of a networked environment. As described herein, program modules 516 typically perform the functions and / or methods of embodiments of the present invention.

[0080] The computer system / server 500 can also communicate with one or more external devices 518, such as a keyboard, pointing device, display 520, etc.; one or more devices that enable a user to interact with the computer system / server 500; and / or any device that enables the computer system / server 500 to communicate with one or more other computing devices (e.g., network interface card, modem, etc.). Such communication can occur via input / output (I / O) interface 514. Furthermore, the computer system / server 500 can communicate with one or more networks, such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet), via network adapter 522. As depicted, network adapter 522 can communicate with other components of the computer system / server 500 via bus 506. It should be understood that, although not shown, other hardware and / or software components can be used in conjunction with the computer system / server 500. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archiving storage systems.

[0081] Additionally, the system 100 used to enhance classifier predictions can be attached to the bus system 506.

[0082] Various embodiments of the invention have been described for illustrative purposes, but are not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein has been chosen to best explain the principles of the embodiments, their practical application, or technical improvements to existing technologies on the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

[0083] This invention can be embodied as a system, method, and / or computer program product. The computer program product may comprise a computer-readable storage medium (or medium) having computer-readable program instructions thereon for causing a processor to perform aspects of the invention.

[0084] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system used for propagation. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), rigid disks, and optical discs. Current examples of optical discs include compact disc read-only memory (CD-ROM), compact disc read / write (CD-R / W), DVDs, and Blu-ray discs.

[0085] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes the following: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital universal disc (DVD), memory sticks, floppy disks, mechanical encoding devices (such as punched cards or raised structures in grooves with instructions recorded thereon), and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0086] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device or via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network) to an external computer or external storage device. The network may include copper transmission cables, optical fiber transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.

[0087] Computer-readable program instructions used to perform the operations of this invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk and C++, as well as conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially as a standalone software package on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet through an Internet service provider). In some embodiments, electronic circuitry (including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs)) may execute the computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of the invention.

[0088] This document describes aspects of the invention with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, which executes on the processor of the computer or other programmable data processing apparatus, creating means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer-readable storage medium having the instructions stored therein includes an article of writing comprising instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0090] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or another device that causes a series of operational steps to be performed on a computer, other programmable device or other device to produce a computer-implemented process, such that the instructions executed on the computer, other programmable device or other device implement the functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0091] The flowcharts and / or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in the flowchart or block diagram may represent a module, segment, or part of an instruction, which includes one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It will also be noted that each block in the block diagram and / or flowchart, and combinations of blocks in the block diagram and / or flowchart, may be implemented by a system based on dedicated hardware that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.

[0092] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that when the terms “comprises” and / or “comprising” are used in this specification, they specify the presence of the stated features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0093] All means or steps in the following claims, plus corresponding structures, materials, actions, and equivalents of the functional elements, are intended to include any structure, material, or action for performing a function in conjunction with other claimed elements (such as those specifically claimed). The description of the invention is presented for purposes of illustration and description, and is not intended to be exhaustive or limited to the forms of the invention disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. These embodiments were chosen and described in order to best explain the principles and practical application of the invention, and to enable those skilled in the art to understand the invention with respect to different embodiments with different modifications suitable for the particular application considered.

Claims

1. A computer system, comprising: One or more computer processors; One or more computer-readable storage media; as well as Program instructions stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors: A classifier system that is trained using training data to build a model for classifying unknown input data into a predicted category; An evaluator engine adapted to determine categories that are not adequately represented in the predicted categories, the evaluator engine using a confidence level distribution of the predictions, wherein the difference between the main predicted category of the input data belonging to the underrepresented category and the next predicted category in the ranked sequence of predicted categories is less than a predefined difference threshold, and wherein the confidence level represents the uncertainty of the predicted category; An extractor engine adapted to extract relevant data from an additional source, the additional source comprising multiple features associated with the identified underrepresented category; A similarity engine, adapted to identify, from training data, categories available for a limited number of training data sets, wherein the training data has not been used in previous training and includes the plurality of features associated with the identified underrepresented categories, and selecting from the relevant data a dataset including the identified plurality of features representing the identified underrepresented categories, wherein: The similarity engine is also adapted to: use the relevant data as a basis for searching additional datasets. When the number of examples in the supplementary dataset exceeds the relative confidence value for the feature associated with the determined underrepresented category, examples in the supplementary dataset are selected from the results of the search query as candidates for training. as well as A recursive unit, adapted to trigger the evaluator engine, the extractor engine, and the similarity engine to take the selected dataset as input data until the evaluator engine classifies the selected dataset with a confidence level higher than a confidence threshold level.

2. The computer system as described in claim 1, wherein, The input data is image data.

3. The computer system as described in claim 2, wherein, Extract the image data from the video data.

4. The computer system as described in claim 1 or 2, wherein, The determination of underrepresented categories is a function based on the difference between the predicted category and other categories.

5. The computer system as described in any one of claims 1 to 3, wherein, The additional data refers to data outside of the training data and regular input data.

6. The computer system as described in any one of claims 1 to 3, further comprising: A labeling unit, adapted to label the selected dataset as an additional training dataset if the evaluator classifies the selected dataset with a confidence level higher than a confidence threshold level.

7. The computer system as claimed in claim 6, wherein, The labeling unit is also adapted to annotate the additional training dataset using the labels associated with categories that are not adequately represented.

8. The computer system as described in any one of claims 1 to 3 and 7, wherein, The recursive unit is adapted to stop the triggering if a predefined number of recursions have already occurred.

9. The computer system as described in any one of claims 1 to 3 and 7, wherein, The evaluator engine is connected to the knowledge graph system.

10. A computer-implemented method, comprising: Provides a classifier system that uses training data to train and build a model for classifying unknown input data into predicted categories; The confidence level distribution of the prediction is determined by comparing the relative differences in the corresponding class probabilities, thereby identifying the underrepresented classes in the prediction classes, wherein the difference between the main predicted class of the input data belonging to the underrepresented classes and the next predicted class in the ranked prediction class sequence is less than a predefined difference threshold, and wherein the confidence level represents the uncertainty of the prediction class. The classifier system is enhanced by extracting relevant data from an additional source that includes multiple features associated with the identified underrepresented categories, wherein the extraction of relevant data includes: Examples of categories available for a finite number of training data points are identified from the training data, wherein the training data available for the examples is finite, the training data has not been used in previous training, and represents the plurality of features associated with the identified underrepresented category; Select a dataset from the relevant data that includes identified features representing the determined, underrepresented categories; use the relevant data as the basis for a search query to search the additional dataset. When the number of examples in the supplementary dataset exceeds the relative confidence value for the feature associated with the determined underrepresented category, examples from the supplementary dataset are selected from the results of the search query as candidates for training; and The determination, extraction, and selection are repeated by using the selected example as input data until the selected dataset is classified with a confidence level higher than the confidence threshold level.

11. The computer-implemented method as described in claim 10, wherein, The input data is image data.

12. The computer-implemented method as described in claim 11, wherein, Extract the image data from the video data.

13. The computer-implemented method as described in any one of claims 10 to 12, wherein, The determination of underrepresented categories is a function based on the difference between the predicted category and other categories.

14. The computer-implemented method as described in any one of claims 10 to 12, wherein, The additional data refers to data outside of the training data and regular input data.

15. The computer-implemented method as described in any one of claims 10 to 12, further comprising: If the selected dataset is classified as having a confidence level higher than the confidence threshold, then the selected dataset is labeled as an additional training dataset.

16. The computer-implemented method as described in claim 15, wherein, The marker further includes: The additional training dataset is annotated with the labels associated with categories that are not adequately represented.

17. The computer-implemented method as described in any one of claims 10 to 12, 16, wherein, If a predefined number of recursions have already occurred, then the repetition stops.

18. The computer-implemented method as described in any one of claims 10 to 12, 16, wherein, The determination refers to accessing a knowledge graph system.

19. The computer-implemented method as described in claim 18, wherein, The extraction refers to accessing the knowledge graph system.

20. A computer program product comprising: A computer-readable storage medium having program instructions contained therein, the program instructions being executable by one or more computing systems or controllers to cause the one or more computing systems to: Provides a classifier system that utilizes training data to build a model that can be used to classify unknown input data into predicted categories; The underrepresented categories in the prediction categories are determined by determining the confidence level distribution of the predictions, wherein the difference between the main predicted category of the input data belonging to the underrepresented category and the next predicted category in the ranked prediction category sequence is less than a predefined difference threshold, and wherein the confidence level represents the uncertainty of the prediction category; The classifier system is enhanced by extracting relevant data from additional sources, the relevant data including multiple additional features associated with identified underrepresented categories, wherein extracting the relevant data includes: Program instructions for identifying examples of categories available for a finite number of training data points from training data, wherein the training data available for the examples is finite, the training data has not been used in previous training, and represents the plurality of features associated with the identified underrepresented category; Select a dataset from the relevant data that includes identified features representing the determined, underrepresented categories; Using the relevant data as the basis for a search query, a search is performed on an additional dataset. When the number of examples in the additional dataset exceeds the relative confidence value for features associated with the identified underrepresented category, examples from the additional dataset are selected as candidates for training from the search query results. The determination step, the extraction step, and the selection step are repeated by using the selected dataset as input data until the selected dataset is classified with a confidence level higher than the confidence threshold level.

Images