Artificial intelligence transparency

Abstract

A computer-implemented method for generating a representative set of model cases for a trained machine learning model can be provided. The method includes determining an input space (102), determining an initial plurality of model cases (104), and expanding the initial plurality of model cases by progressively modifying field values of records representing the initial plurality of model cases, thereby producing an exploration set of model cases (106). Additionally, the method includes obtaining model score values for each record of the exploration set of model cases (108), continuing expansion of the exploration set of model cases, thereby generating an improved set of model cases (110), and selecting records in the improved set of model cases based on relative record distance values and associated model score values between pairs of records (112), thereby generating the representative set of model cases.

Description

BACKGROUND

[0001] The present invention relates generally to the field of explainable AI (Artificial Intelligence) and more specifically to a computer-implemented method for generating a representative set of model cases for a trained machine learning model. The invention further relates to a related representative case generation system for generating a representative set of model cases for a trained machine learning model and a related computer program product.

[0002] The field of machine learning is a subfield of artificial intelligence that allows certain tasks to be done automatically without explicitly stating how to do them when general research techniques and algorithms are concerned. Machine learning methods are generally divided into three categories: supervised machine learning, unsupervised machine learning and reinforcement learning.

[0003] Supervised machine learning techniques are used to build models that can be generalized from a set of labeled input data. This input data can be in various forms, e.g. images, audio sequences or sequences of actions, while the labels order the data points into different categories: pictures of cats or dogs, sequences leading to success or failure, etc.

[0004] Unsupervised machine learning techniques are used to find patterns, relationships and structures in data. In contrast to supervised machine learning techniques, the data points provided to the learner, i.e. the learning system, are not labeled or categorized.

[0005] Reinforcement learning involves the behavior of a software agent acting in an environment in order to maximize some notion of reward. The agent acts based on its state and knowledge about potential rewards / punishments. This technique is improved, for example, by introducing a "discount factor" that determines how far into the future the agent should look. SUMMARY

[0006] According to an aspect of the invention, a computer-implemented method for generating a representative set of model cases for a trained machine learning model can be provided. The method can comprise determining an input space for the trained machine learning model; determining an initial plurality of model cases based on the determined input space, wherein each model case can be represented by a record comprising a plurality of record fields having respective initial input values; and expanding the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases.

[0007] The method can further comprise obtaining a model score value for each record of the exploration set of model cases, and continuing the expansion of the exploration set of model cases in the direction of those modified model cases that provide one of a relatively high model score value and a relatively low model score value compared to the records of the exploration set. Thereby, a set of modified model cases can be resulted.

[0008] Finally, but not least, the method can comprise selecting records in the improved model case set based on the relative record distance values and the related model score values between pairs of records. Thereby, a representative model case group can be generated.

[0009] According to another aspect of the present invention, a representative case generation system for generating a representative model case group for a trained machine learning model can be provided. The system can comprise a determining unit adapted to determine an input space of the trained machine learning model. Thereby, the determining unit can further be adapted to determine an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields having respective initial input values. Further, the method can comprise an expanding module adapted to expand the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby generating an exploration set of model cases; and a model score obtaining unit adapted to obtain a model score value for each record of the exploration set of model cases.

[0010] Additionally, the method can comprise a triggering unit adapted to continue the expansion of the exploration set of model cases in the direction of those modified model cases providing one of a relatively high model score value and a relatively low model score value compared to the records of the exploration set, thereby generating an improved model case set; and a selecting unit adapted to select records in the improved model case set based on the relative record distance values and the related model score values between pairs of records, thereby generating a representative model case group.

[0011] Further, embodiments can take the form of an appropriately programmed computer program product which can be accessed from a computer-usable or computer-readable medium providing program code for use by or in connection with a computer or any instruction execution system. For the purposes of this description, a computer-usable or computer-readable medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. BRIEF DESCRIPTION OF DRAWINGS

[0012] It should be noted that embodiments of the present invention are described with reference to different subject matters. In particular, some embodiments are described with reference to method type claims whereas other embodiments are described with reference to device type claims. However, a person skilled in the art will gather from the above and the following description that, unless otherwise indicated, in addition to any combination of features belonging to one type of subject matter, also any combination between features according to different subject matters, in particular between features according to method type claims and features according to device type claims, is considered to be disclosed within this document.

[0013] The above-defined aspects and further aspects of the present application are apparent from the examples of embodiments to be described hereinafter and are explained with reference to examples of embodiments, which are described hereinbelow, but the application is by no means limited thereto.

[0014] The preferred embodiments of the present application will be described by way of example only with reference to the following drawings in which:

[0015] Figure 1 A block diagram illustrating an embodiment of the inventive computer-implemented method for generating a representative set of model cases for a trained machine learning model is shown;

[0016] Figure 2 A block diagram illustrating the flow of the different stages of the proposed method is shown;

[0017] Figure 3 A diagram showing the amount of model cases generated and selected during the execution of the proposed method is shown;

[0018] Figure 4 A block diagram illustrating an embodiment of the representative case generation system is shown;

[0019] Figure 5 An embodiment of a computing system comprising a representative case generation system according to Figure 4 is shown. DETAILED DESCRIPTION

[0020] In the context of the present specification, the following conventions, terms and / or expressions can be used:

[0021] The term ‘representative model case’ can denote at least two types of model cases: model cases representing the average value of the input cases - which can also be considered as the most representative - in order to generate the result of the trained machine learning model. In contrast to the representative model cases, there are extreme value model cases, which have extreme values close to the edges / boundaries of the input space of the trained machine learning model and / or abnormal values that can cause problems during the classification or regression process of the trained machine learning model. For these cases, the confidence value can be extremely low compared to the average value cases. Furthermore, the number of false negative or false positive results for the trained machine learning model can also significantly increase, i.e. wrong prediction results can be generated.

[0022] The term‘trained machine learning model’ can be described as a set of hyperparameters that describe the internal functioning of a machine learning“black box”. In the proposed concept, it can be assumed that these hyperparameters are unknown. The hyperparameters are developed inside the machine learning model during the training phase. During the training phase, in the case of supervised machine learning, a larger amount of labeled dataset (records) can be used to learn the expected outcome from the labels. This can apply to classifiers, which are usually implemented as neural networks. However, regression models can also be used as trained machine learning models in order to apply the proposed concept. Alternatively, regression models can be reformulated as clustering models.

[0023] The term‘input space’ can represent the range of input record fields of a machine learning model, such that the meaning of the generated outcome, i.e. a classification with a relatively high confidence value, is generated. In simple words: the input space can define the range of attribute fields for input records, where the trained machine learning model works as expected.

[0024] The term‘initial plurality of model cases’ can represent an initial set of records that represent model cases used as an initial dataset to start the concept / method proposed here. These initial model cases / records can be generated randomly or according to another predefined algorithm. The training dataset of the validation dataset can also be used as a seed value.

[0025] The term‘exploration set of model cases’ can represent an expanded set of records generated from the initial plurality of model cases.

[0026] The term‘model score value’ can generally represent a value that is suitable to distinguish between good and bad models. One example of a model score value can be a confidence value for a given input record.

[0027] The term‘improved model case set’ can represent a collection of model cases / records that are used as a basis to select the most representative model cases and / or extreme model cases by a set of predefined selection methods.

[0028] The term“selection records” can represent an activity performed in the selection phase of the proposed concept. The selection can be equivalent to grouping certain records, or even clustering certain records, in order to establish a group of model cases that are representative of the most representative model cases for a given trained machine learning model, and / or to establish sample model cases at the boundaries or average values of the input space for a given trained machine learning model.

[0029] The term‘relative record distance value’ can represent a floating point number that describes the level of similarity or the level of difference between two records. A conceivable distance measure is the cosine distance. However, other distance measure concepts are also possible and can equally support the proposed concept.

[0030] The term ‘constraining’ can mean a boundary condition under which new records representing additional model cases can be generated.

[0031] The term ‘better support records’ can mean newly generated records starting from existing records that respect and reflect the constraints during the generation process.

[0032] The term ‘most relevant records’ can mean input records to a trained machine learning model that represent the average input record (or neighboring records), i.e. having values for the record fields that are each average for the input space of the machine learning model.

[0033] With the proliferation of AI applications, more and more questions are raised about transparency, explainability, trustworthiness, and testability of AI applications. One of the reasons is that more complex (and possibly more accurate) machine learning models tend to have more black box character. Recently, the question of controllability of AI systems has become more and more prevalent. This has led to the field of explainable AI (XAI) that tries to make the black box behavior of AI systems more understandable.

[0034] The proposed computer-implemented method for generating a representative set of model cases for a trained machine learning model can provide a number of advantages, contributions, and technical effects:

[0035] In general, the proposed method can allow for a better understanding of existing trained machine learning models in terms of clarity, transparency, testability, and explainability. To achieve this, the most relevant cases and how they map to their outcomes are isolated and presented to the user.

[0036] This clearly goes beyond such models that have been analyzed as a white box (i.e. the inner workings of the model are available for analysis) or that merely focus on the importance and coefficients of the input variables. Instead, the concept presented here focuses on machine learning models that can only be accessed as a black box, i.e. the internal structure (e.g. architecture and algorithm type) is considered unknown. So, no information about the used algorithm or the training data can be given. However, the trained model can be used to score examples and the output can be used for analysis. In addition to looking at the relevant features with assigned importance, coefficients, and other relevant metrics, a novel approach is presented here to look at the most relevant cases (e.g. typical cases vs. outliers) for a given trained machine learning model.

[0037] It is believed that this approach can add another very relevant angle for analyzing such machine learning models, especially in the new era where more and more cases are organizations trading such IP (intellectual property) but at the same time need to make sure it is ethical and complies with rules such as GDPR (General Data Protection Regulation in Europe).

[0038] The proposed method can be used for evaluating classification systems, like neural networks, but is not limited to such machine learning models. Furthermore, the proposed method can also be used for regression models. The application of regression models can be performed in multiple ways: (i) The regression problem can be transformed into a classification problem by creating buckets. Then, the algorithm can be applied to these buckets as proposed above. (ii) Alternatively, a number of representative cases (i.e. records) can be defined using random initialization. Then, the initialized representative cases should be moved in the input space, where the optimization criterion is to increase the distance to each other in the output space; i.e. one should also focus on maximizing the distance in the input space in order to find different representative cases that are mapped to the same output.

[0039] In addition, the proposed concept can be applied to both supervised as well as unsupervised machine learning models, as the proposed method can treat a machine learning model as a black box with defined input and output spaces. The type of model is irrelevant to the concept of the proposed work.

[0040] Furthermore, it is worth noting that the proposed method can also be adapted to be independent of the confidence values. For this, only the assigned classes are considered. By exploring the input space (i.e. the movement of the initial data points), each class can be mapped in the input space. Then, a centrality measure can be used to find the most representative data point of the input cluster of a given class (e.g. consider cross-entropy).

[0041] An additional benefit of this variation can be that the algorithm is not dependent on the amount of available training data. In the above case, a high data availability of certain input space regions would lead to a higher confidence of this region, as the algorithm has more data to learn from.

[0042] In the following, further embodiments of the proposed method (also in relation to the related system) will be described:

[0043] According to one embodiment of the method, determining the input space can comprise using pre-defined values for the fields of the records, or using the training data set of the trained machine learning model as seed records. The second method can be supported by unsupervised learning to find central groups in the training data or another (typical) set of data related to the machine learning model. Furthermore, the determination of the input space can also be achieved by using outliers of the records of the training data (or another typical data set) of the trained machine learning model as seed records. Thereby, multiple different methods can be used to start the proposed method here with an initial set of model cases (i.e. typical records). Thus, it can be assumed that each model case can be represented by a respective record. Furthermore, it should be mentioned that also a combination of the above proposed ways for determining the input space can be used.

[0044] According to an advantageous embodiment of the method, the stepwise modification of the field values of the records can comprise (i) using records of a group comprising a confidence higher than a predefined high confidence value as a basis for the modification, or (ii) using records of a group comprising a confidence higher than a predefined low confidence value as a basis for the modification, or (iii) using records of a group having a predefined high range of confidence values at the boundaries of the input space and / or a predefined low range of confidence values as a basis for the modification. Thus, there are multiple options available for modifying the field values or features of the records describing the model cases.

[0045] According to an embodiment of the method, the initial plurality of model cases can be generated randomly or by using a subset of the training or validation data of the trained machine learning model. Thus, here too a choice can be made from different options during the initialization phase of the proposed method.

[0046] According to an optional embodiment of the method, the stepwise modification of the field values of the records representing the initial plurality of model cases can comprise modifying one field value of a record at a time randomly or modifying multiple field values of a related plurality of fields at a time randomly. The second option can be the basis for even more randomness of the model cases to be tested.

[0047] According to a further embodiment of the method, the stepwise modification of the field values of the records representing the initial plurality of model cases can comprise modifying one or more field values of a record at a time or modifying multiple field values of a related plurality of fields under the influence of constraints supporting a given objective. Thus, the direction of the changes (i.e. the direction of the modifications of the available model cases) can be directed into certain (i.e. influenced by constraints) directions. This can improve the simple trial and error method and can be more efficient during the exploration phase to generate new relevant model cases. For this, different shooting methods can also be used: (i) potentially, the problem at hand can already provide some information that can be advantageously used here. (ii) a sensitivity analysis can provide the “deepest descent” which can give the direction in which the features (i.e. the fields of the model case records) can need to be adjusted. (iii) last but not least, heuristic optimization methods can be applied which can comprise reinforcement learning, simulated annealing, genetic algorithms, tabu search, etc.

[0048] According to an advantageous embodiment, the method can further comprise interrupting the continuation of the expansion of the exploration set of model cases if (a) the newly generated - i.e. by modification, see above - records for expanding the exploration set of model cases do not show a better support of the target - i.e. they remain more or less constant compared to the records already available in the exploration set of model cases, (b) a predefined number of records can be reached within the exploration set, (c) a predefined number of expansion loops can have been performed, or (d) a preset time period ends. Thus, different approaches are also possible here in order to limit the time and computational effort of generating a useful set of model cases for further processing.

[0049] According to another embodiment, the method can further comprise reducing the number of records in the improved set of model cases. This can be performed according to predefined rules in order to reach a useful number of model cases for further processing. For example, more similar record model cases (i.e. records) that only slightly differ on one feature (i.e. one field of the record) can be eliminated in the improved set of model cases.

[0050] According to another advanced embodiment, the method can further comprise using the distance measure representing the most relevant records of a cluster for selecting those records during the reduction. Thus, different approaches can also be used here for reducing the improved set of model cases.

[0051] According to a preferred embodiment of the method, the distance measure can be the cosine similarity. However, other distance measure algorithms can also be used. It can also be mentioned that the cosine similarity is known as a measure of similarity between two non-zero vectors in an inner product space, which measures the cosine of the angle between them.

[0052] In the following, a detailed description of the figures will be given. All instructions in the figures are schematic. First, a block diagram of an embodiment of the inventive computer-implemented method for generating a representative set of model cases for a trained machine learning model will be given. Thereafter, further embodiments will be described, as well as embodiments of a representative case generation system for generating a set of representative model cases for a trained machine learning model.

[0053] Figure 1 A block diagram of a preferred embodiment of a computer-implemented method 100 for generating a representative set of model cases for a trained machine learning model is shown. The method 100 comprises determining (102) an input space for the trained machine learning model in an initialization phase. The method 100 comprises further determining (104) an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields (i.e. features of the record) with respective initial input values.

[0054] The method 100 further comprises - in an exploration phase - expanding (106) the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases. Thus, more records are artificially created - of course within the input space - in order to increase the number of model cases to be evaluated. So, the amount of model cases thus created is of course larger than the initial plurality. Another part of the exploration phase is obtaining (108) a model score value for each record of the exploration set of model cases. As model score, for example, the confidence of each record can be used. Further, this phase of the method 100 comprises continuing (110) expanding the exploration set of model cases in the direction of those modified model cases providing one of a relatively high model score value and a relatively low model score value compared to the records of the exploration set. Thereby, an improved set of model cases is created.

[0055] The term'relatively high' means that the model score value of a given model case can be compared within a predetermined distance to the surrounding model cases (i.e. their related records), or alternatively, with respect to all model cases in the improved set of model cases. Other orientation points (e.g. predefined or dynamic thresholds) or regions for comparison can also be chosen. Equivalent methods can be used for the term'relatively low'.

[0056] In the last phase (i.e. the selection phase), the method 100 comprises selecting (112) - which can also be interpreted as grouping or clustering - the records of the improved set of model cases based on the relative record distance values and the related model score values between the record pairs. Thereby, groups of representative model cases are created and can be presented to a user for reasons of understandability and interpretation. Any data format can be useful; in the form of listed numbers or in any suitable graphical form.

[0057] It can also be mentioned that besides determining the distance between record pairs, alternative (or combined with the distance metric) options can be used as selection criteria. The selection process can be based on regions in the point clouds of the records and their special characteristics or the centrality of the point clouds of the records in the multi-dimensional space.

[0058] Figure 2 A block diagram showing the flow of the different phases 200 of the proposed method 100 is shown. Figure 2 Further insights can result due to their more practical way of describing the proposed method. During the initiation phase 202, the first step is that the method "understands" how the input space is defined, i.e. the list of possible ranges of each feature of each record / model case and / or where it is limited. Basically, this step deals with the problem that the "black box" of the trained machine learning model provides a score value of the machine learning model to a specific model case. The range can be given explicitly.

[0059] If not provided, a brute force approach (with binary search and other optimizers extensions) can be used to find out which algorithms they use that can provide useful scores. A set of seeds (i.e. initial model cases to start from) needs to be generated. In the simplest approach, n random model cases can be generated that are consistent with the input space under examination. Additional approaches have been described above and can be summarized here again: seeds can be given in order to cover the input space. The general approach works with random seeds. However, for certain cases, it can be optimized to use seeds that are more tailored to the given problem. Alternative approaches can be based on already existing data sets that can be extended later on: (i) a subset of the available data set can be used; (ii) unsupervised machine learning methods can be applied in order to e.g. cluster, such as a K-means-algorithm, in order to find the "centers" of the clusters and use these "centers" as seed records. Or (iii), an outlier detection of the data can be performed. Model cases identified this way - especially model cases that describe the most extreme model cases that are still within the input space of the algorithm - can be useful. Of course, a combination of the different approaches (i) to (iii) can be used.

[0060] Thus, the start-up phase 202 comprises the main activity 204 of defining the input space and finding the starting positions.

[0061] The exploration phase 206 can also comprise two main activities 208: the generation of new model cases for finding more relevant model cases, and the measurements made in order to deal with the width and depth of the concepts presented here.

[0062] The first step in this phase is to define different objectives and to perform different sets for each objective. Model cases can be found that have the highest confidence value for a given classification label. Alternatively, cases can be found that have the lowest confidence value for a given classification label, but this confidence is still the highest among all classes. Furthermore, cases can be found that have a high or low confidence value and are at the "border" of the input space.

[0063] Then, the scoring of the existing cases is performed. This is done for the initial start seed records and during the later iterations to the newly generated model case sets.

[0064] In the third sub-step of this phase, each candidate model case is modified within the input space. So, the actual values of one or more features of the record representing the model case are modified in order to produce additional model cases. This can be done randomly or according to predefined constraints in order to see whether adjacent cases are even more extreme / typical than the already provided cases.

[0065] In a fourth sub-step, an iterative approach is executed, which can stop when no further optimization of different model cases is possible (i.e. modifications do not improve the above defined objectives and / or can remain constant over different iterations).

[0066] To improve the trial-and-error approach just described and to be more efficient with the exploration of new relevant cases, different tuning approaches can be executed: (i) potentially, the current problem already provides some information that can be exploited; (ii) a sensitivity analysis can provide the direction of the “deepest descent”, which can give the direction in which the characteristics (recorded field / attribute values) of the current model case can need to be adjusted; (iii) in addition or alternatively, heuristic optimization methods like reinforcement learning, simulated and annealing, and genetic algorithms, tabu search, etc. can also be applied.

[0067] Several of these exploration methods generate several different neighboring cases for a given model case. Both neighboring cases (discovered by modifying existing cases) and cases with optimized objective functions are kept in the set of model cases that continue in the next iteration.

[0068] Finally, the “memory” of analyzed model cases is updated, so that only new cases will be further explored and unnecessary cycles and optimizations can be avoided. Thus, unused generated model cases can be stored in an intermediate storage and can be compared with newly generated model cases to decide whether the case has been generated in the past. Thus, a de-duplication process is inherent in the generation or new record or exploration method part.

[0069] The final phase (i.e. the selection phase 210) represents the activity 212 of finding a balanced representative set of model cases by exploiting the measured distances and providing the identified representative model cases. These can also be described as personas.

[0070] At this point, it can be useful to explain the background aspects of the concepts presented here. To build trust into existing trained machine learning models, the requirements of high transparency (i.e. explainable AI (XAI)) and the concept of design thinking have been put together in order to show how the concept of representative cases (a well-known and rewarding concept in design thinking) can improve the transparency of AI models. Design thinking generally refers to a set of processes and methods used to generate new designs and innovative ideas. In particular, this includes the concept of personas, which are fictional characters that act / interact with a given product in a specific setting to bring the user to the center of the process or procedure. In this sense, personas are representative (model) cases.

[0071] Furthermore, the selection phase 220 can be described in more detail in the following way: After the exploration phase is finished, each objective (like a constraint) has a set of model cases (i.e. potential records), the selection step ensures that this number is reduced to a number that is desired by the user or a subsequent process in order to interpret the behavior of the trained machine learning model.

[0072] The most relevant model cases are selected using one or more distance measures. Typically, the globally optimal values should be used and considered as the final representative cases. Furthermore, local optima can also be relevant, but since the underlying solution space can consist of many of them, not all of them are equally relevant. A further selection process can be needed.

[0073] It can be very useful that the final set has a good spread of model cases within the input space (e.g. for a medical algorithm, they should include all age groups, genders, and a good balance between healthy and unhealthy people).

[0074] The distance between model cases can be used to find more isolated and possibly relevant model cases. The calculation of the distance happens in the embedding space of the model cases. By this, a case can be represented in the embedding space by encoding each feature as a 0 / 1 value, resulting in a vector of length N for each case, where N is the dimension of the input space. A preferred but finally exemplary measure for the distance measure is the cosine similarity. To identify isolated and possibly relevant cases, the relative distance between sets of model cases is calculated and in a hierarchical clustering fashion, central cases are identified (e.g. by merging cases from the bottom). This will result in the most relevant “islands” or clusters and the model cases closest to the centroid of this cluster will be one of the highest relevance. Therefore, it can be marked as highlighted as an output of the concept presented here.

[0075] The distance measure values can be adjusted / expanded to favor different features based on criteria like model confidence, importance of features, personal interests, etc. The final set of representative model cases is then considered as the output of the method presented here. Depending on the configuration decided by the user, after the selection step, a new exploration step can start with the model cases that reside from the last selection step.

[0076] It is also worth noting that the concept presented here can be used to generate new validation data sets for the analyzed algorithm. The set of relevant cases should be amplified to have a test or confirmation data set that makes sense in terms of its size. This can be done by adjusting the parameters of the concept presented here, e.g. making sure that the distance measure does not filter too many local optima.

[0077] Other possible fields of utilization of the presented concept are as follows: it can be used to support and / or be integrated into active learning concepts in order to improve performance. It can also be used to provide a limited and meaningful set of model cases. The scores / predictions of the model can be reviewed and checked for correctness and can well indicate whether the model is useful or not. Such a check can be done by official government agencies to approve specific trained machine learning models.

[0078] Furthermore, a visualization can be performed on how to score specific model cases; they can be highlighted as areas of weak or strong model cases. Additionally, if the labels of the discovered representative cases are available, it can be highlighted which parts of the input space have problems and should be excluded from the allowed input space in order to achieve good classification or regression results.

[0079] In order to make the representative model cases more understandable, the presented concept can be combined with information on feature importance and provide information on representative model cases based on filtering or ranking based on the feature importance information.

[0080] Figure 3 A diagram 300 is shown that illustrates the amount of model instances generated and selected during the execution of the presented method. During the launch phase 202, only a small amount of records 302 exist, e.g. seed records. Then, during the exploration phase 206 (see Figure 2 ), the number of records 304 is extended. Together with the scores of the records (and possibly constraints), additional areas 308, 310, 312 of records can be added depending on the direction of the extension (depending on the used constraint model), resulting in a combined record space 306 (comprising the arrays of records 308, 310, 312). Finally, during the selection phase 210 (see Figure 2 ), the trained machine learning model running in a typical way or model cases that are outliers on the boundary of the input space are shown as a relatively small amount of records - i.e. model cases - 316, 314, 318.

[0081] For completeness reasons, Figure 4 A block diagram of an embodiment of a representative case generation system 400 for a trained machine learning model is shown. The system 400 comprises a determination unit 402 adapted to determine an input space of the trained machine learning model. Furthermore, here the determination unit is also adapted to determine an initial plurality of model cases based on the determined input space. Thereby, each model case is represented by a record comprising a plurality of record fields with respective initial input values.

[0082] The system 400 further comprises an expansion module 404 adapted to expand the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby producing an exploration set of model cases, and a model score obtaining unit 406 adapted to obtain a model score value for each record of the exploration set of model cases. Last but not least, the system 400 comprises a triggering unit 408 adapted to continue the expansion of the exploration set of model cases in the direction of those modified model cases providing one of a relatively high model score value and a relatively low model score value compared to the records of the exploration set, thereby generating an improved set of model cases, and a selection unit 410 adapted to select a record of the improved set of model cases based on the relative record distance value and the relevant model score value between the pair of records, thereby generating a representative set of model cases.

[0083] Embodiments of the present application can be implemented together with virtually any type of computer, regardless of the platform suitable for storing and / or executing program code. Figure 5 As an example, a computing system 500 adapted to execute program code relating to the proposed method is shown.

[0084] The computing system 500 is only one example of a suitable computer system and is not intended to suggest any limitation as to the scope of use or functionality of embodiments of the present application described herein, whether or not implementation is actual or future. In computing system 500, there are components, which are operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with computer system / server 500 include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, set top boxes, programmable consumer electronics, network PCs, minicomputer systems, mainframe computer systems, and distributed cloud computing environments that include any of the above systems or devices, and the like. Computer system / server 500 can be described in the general context of computer system executable instructions, such as program modules being executed by computer system 500. Generally, program modules can include routines, programs, objects, components, logic, data structures, and the like that perform particular tasks or implement particular abstract data types. Computer system / server 500 can be practiced in distributed cloud computing environments with remote processing devices that are linked through a communications network. In a distributed cloud computing environment, program modules can be located in both local and remote computer system storage media including memory storage devices.

[0085] As shown, computer system / server 500, which is an example of computing device, is operational with numerous other general purpose or special purpose computing system environments or configurations. Examples of well- known computing systems, environments, and / or configurations that can be suitable for use with computer system / server 500 include but are not limited to one or more personal computers, server computers, handheld or laptop devices, multiprocessor systems, microprocessor-based systems, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, and the like.

[0086] System memory 504 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 508 and / or cache memory 510. Computer system / server 500 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 512 can be provided for reading from and writing to non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Although not specifically shown, a magnetic disk drive can also be used for reading from and writing to a removable, non-volatile magnetic disk (e.g., a "floppy disk"), and an optical disk drive can be used for reading from or writing to a removable, non-volatile optical disk such as a CD-ROM, DVD-ROM or other optical media. Each of these disk drives and their associated computer system media, including the storage of the present application, can be connected to bus 506 by one or more data media interfaces. As will be further depicted and described below, memory 504 can include at least one program product having a set (e.g., at least one) of program modules that are configured to carry out the functions of embodiments of the application.

[0087] Program / utility 514, having a set (at least one) of program modules 516, can be stored in memory 504 by way of example, and not limitation, as well as an operating system, one or more application programs, other program modules, and program data. Each of the operating system, one or more application programs, other program modules, and program data or some combination thereof, can include an implementation of a network environment. Program modules 516, when executed, typically carry out the functions and / or methodologies of embodiments of the application, as described herein.

[0088] Computer system / server 500 can also communicate with one or more external devices 518 such as a keyboard, a pointing device, a display 520, etc.; one or more devices that enable a user to interact with computer system / server 500; and / or any devices (e.g., network card, modem, etc.) that enable computer system / server 500 to communicate with one or more other computing devices. Such communication can occur via Input / Output (I / O) interfaces 514. Still yet, computer system / server 500 can communicate with one or more networks such as a local area network (LAN), a general wide area network (WAN), and / or a public network (e.g., the Internet) via network adapter 522. As depicted, network adapter 522 can communicate with the other components of computer system / server 500 via bus 506. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer system / server 500. Examples include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0089] Additionally, the representative case generation system 400 for generating a representative set of model cases for a trained machine learning model can be attached to the bus system 506.

[0090] The description of the various embodiments of the present application have been presented for purposes of illustration, but is not intended to be exhaustive or limited to the embodiments disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0091] The present application can be embodied as a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions stored thereon for causing a processor to perform aspects of the present application.

[0092] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or apparatus) for propagating or transmitting a computer readable program. Examples of a computer readable medium include a semiconductor or solid state memory, magnetic tape, a removable computer diskette, a random access memory (RAM), a read-only memory (ROM), a rigid disk, and an optical disk. Current examples of optical disks include compact disk - read only memory (CD-ROM), compact disk - read / write (CD-R / W), DVD, and Blu-ray® disks.

[0093] The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non- exhaustive list of more specific examples of the computer readable storage medium include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0094] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0095] Computer readable program instructions for carrying out operations of the present application can be assembly instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present application.

[0096] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0097] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0098] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0099] The computer program product of the different embodiments of the present application can be implemented on one computer or on a plurality of computers in or coupled to the same communication network. The computer program code can be written in any suitable languages for the intended application. The program code can be stored on a computer readable medium, which can be a non-transitory medium, until ready to be used to implement the methods according to the embodiments of the present application. Examples of non-transitory media include a floppy disk, a ZIP® disk, a hard disk, a magnetic tape, an optical storage medium, a memory chip, a memory card, a software storage, and the like. The program code can be distributed over the media, stored thereon, and / or performed, by one or more computers or processors.

[0100] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present application. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0101] All methods and steps plus their corresponding structures, materials, acts and equivalents of the claims below are intended to include any structure, material, or act for performing the function of other claim elements in combination with other claim elements. The description of the present application has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the present application in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the present application. The embodiment was chosen and described in order to best explain the principles of the present application and the practical application, and to enable others skilled in the art to understand the present application with various modifications as are suited to the particular use contemplated.

Claims

1. A computer-implemented method for generating a representative set of model cases for a trained machine learning model, the method comprising: determining an input space for the trained machine learning model, the input data of the input space comprising: image data, audio data, medical data; determining an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields with respective initial input values; extending the initial plurality of model cases by stepwise modifying field values of records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases; obtaining model score values for each record of the exploration set of model cases; continuing the extension of the exploration set of model cases in a direction of those modified model cases providing one of relatively high and relatively low model score values compared to records of the exploration set, thereby generating an improved cluster of model cases; and selecting records of the improved cluster of model cases based on relative record distance values and associated model score values between pairs of records, thereby resulting in the representative set of model cases.

2. The method of claim 1, wherein, determining the input space comprises: using predefined values of record fields, or using a training data set of the trained machine learning model as seed records, or using outlier values of records of the training data of the trained machine learning model as seed records.

3. The method according to any of the preceding claims, wherein, the stepwise modifying of field values of records comprises: using records of a group comprising a confidence higher than a predefined high confidence value as basis for modification, or using records of a group comprising a confidence higher than a predefined low confidence value as basis for modification, or using records of a group having a predefined high range of confidence values at a boundary of the input space and / or a predefined low range of confidence values as basis for modification.

4. The method according to any one of claims 1-2, wherein the initial plurality of the model cases is generated randomly or using a subset of training or validation data of a trained machine learning model.

5. The method of any one of claims 1-2, wherein, the stepwise modifying of the field values of records representing the initial plurality of model cases comprises modifying one field value of a record at a time randomly or modifying a plurality of field values of a related plurality of fields at a time randomly.

6. The method of any one of claims 1-2, wherein, the stepwise modifying of the field values of records representing the initial plurality of model cases comprises modifying one or more field values of a record at a time or modifying a plurality of field values of a related plurality of fields under influence of constraints supporting a given goal.

7. The method according to claim 6, further comprising interrupting the continuation of the extension of the exploration set of model cases if: the newly generated records for extending the exploration set of model cases do not show a better support of the goal than already available records in the exploration set of model cases, a predefined number of records is reached within the exploration set, a predefined number of extension loops has been performed, or a preset time is over.

8. The method of any of claims 1-2, further comprising reducing the number of records in the improved model case clusters.

9. The method of claim 8, further comprising: using distance measures representing the most relevant records of each cluster in the model case clusters to select those records during the reducing.

10. The method of claim 9, wherein, The distance measures are cosine similarities.

11. A representative case generation system for generating a representative set of model cases for a trained machine learning model, the system comprising at least one processor and a memory having stored thereon program instructions executable by the at least one processor to cause the system to perform a method, the method comprising: determining an input space for the trained machine learning model, the input data of the input space comprising: image data, audio data, medical data; determining an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields having respective initial input values; extending the initial plurality of model cases by stepwise modifying field values of records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases; obtaining model score values for each record of the exploration set of model cases; continuing the extension of the exploration set of model cases in a direction providing those modified model cases having one of relatively high and relatively low model score values compared to the records of the exploration set, thereby generating an improved set of model cases; and selecting records in the improved set of model cases based on relative record distance values and related model score values between pairs of records, thereby resulting in the representative set of model cases.

12. The system of claim 11, wherein, determining the input space comprises: using predefined values of record fields, or using a training data set of the trained machine learning model as seed records, or using outlier values of records of the training data of the trained machine learning model as seed records.

13. The system of any one of claims 11-12, wherein, stepwise modifying the field values of the records comprises: using records of a group comprising a confidence higher than a predefined high confidence value as a basis for modification, or using records of a group comprising a confidence higher than a predefined low confidence value as a basis for modification, or using records of a group having a predefined high range of confidence values at a boundary of the input space and / or a predefined low range of confidence values as a basis for modification.

14. The system of any of claims 11-12, wherein the initial plurality of the model cases are generated randomly or using a subset of training or validation data of the trained machine learning model.

15. The system of any one of claims 11-12, wherein, The stepwise modifying the field values of the records representing the initial plurality of model cases comprises modifying one field value of a record at a time randomly or modifying a plurality of field values of a related plurality of fields at a time randomly.

16. The system of any one of claims 11 to 12, wherein, The stepwise modifying the field values of the records representing the initial plurality of model cases comprises modifying one or more field values of a record at a time or modifying a plurality of field values of a related plurality of fields under influence of constraints supporting a given goal.

17. The system of claim 16, further comprising interrupting the continuation of the expansion of the exploration set of model cases if: the newly generated records for expanding the exploration set of model cases do not show better support for the target than the already available records in the exploration set of model cases, a predefined number of records within the exploration set is reached, a predefined number of expansion loops has been performed, or a preset time is over.

18. The system of any one of claims 11 to 12, wherein the method further comprises reducing the number of records in the improved model case set.

19. The system of claim 18, wherein the method further comprises using distance measures representing the most relevant records of each cluster in the model case cluster set to select those records during the reduction.

20. A computer program product for generating a representative model case group for a trained machine learning model, the computer program product comprising program instructions executable by at least one processor to cause the at least one processor to perform a method, the method comprising: determining an input space for the trained machine learning model, the input data of the input space comprising: image data, audio data, medical data; determining an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields having respective initial input values; expanding the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases; obtaining a model score value for each record of the exploration set of model cases; continuing the expansion of the exploration set of model cases in a direction providing ones of those modified model cases having one of relatively high and relatively low model score values compared to the records of the exploration set, thereby generating an improved model case cluster set; and selecting records in the improved model case cluster set based on relative record distance values and related model score values between pairs of records, thereby resulting in the representative model case group.

21. A computer program product for generating a representative model case group for a trained machine learning model, the computer program product comprising program instructions executable by at least one processor to cause the at least one processor to perform a method, the method comprising: determining an input space for the trained machine learning model, the input data of the input space comprising: image data, audio data, medical data; determining an initial plurality of model cases based on the determined input space, wherein each model case is represented by a record comprising a plurality of record fields having respective initial input values; expanding the initial plurality of model cases by stepwise modifying the field values of the records representing the initial plurality of model cases, thereby resulting in an exploration set of model cases; obtaining a model score value for each record of the exploration set of model cases; continuing the expansion of the exploration set of model cases in a direction providing ones of those modified model cases having one of relatively high and relatively low model score values compared to the records of the exploration set, thereby generating an improved model case cluster set; and selecting records in the improved model case cluster set based on relative record distance values and related model score values between pairs of records, thereby resulting in the representative model case group.

Images