Introspection of AI Model Learning

Abstract

Techniques related to AI model introspection are provided. For example, one or more embodiments described herein may have a system that may include a memory that may store computer-executable components. The system may also include a processor operatively coupled to the memory and capable of executing the computer-executable components stored in the memory. The computer-executable components may include a model introspection component that may analyze the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the artificial intelligence model for multiple test data subsets having different code complexity distributions.

Description

Technical Field

[0001] This disclosure relates to introspection of the learning behavior of an artificial intelligence model, and more specifically, to analyzing the learning behavior of an artificial intelligence (「AI」) model by comparing the output of a trained AI model with test data having samples of different code complexities.

[0002] AI models are used in various source code understanding tasks, such as defect detection, code summarization, code completion, bug fixing, naming of functions and variables, code recommendations, software development workflows, etc. However, training an AI model for source code understanding tasks may face various reliability concerns, including data duplication bias, labeling quality, low robustness and / or generalization, overfitting of the model (e.g., due to limitations in the dataset size), class imbalance (e.g., due to mismatches in sample counts belonging to different classes), etc. For example, the performance quality of an AI model may be affected by whether the model is learning task-related aspects of the source code. Further, understanding the learning behavior of an AI model typically focuses on analyzing the software architecture of the model. Thus, typical approaches may require access to the internal operation of the model and / or deep subject knowledge of the internal operation of the model. Additionally, typical introspection techniques may ignore the analysis of signal recognition in the learning behavior of an AI model. Also, existing model performance metrics may ignore considering aspects of the dataset on which the AI model is trained.

Summary of the Invention

[0003] The following is an overview to provide a basic understanding of one or more embodiments of the present invention. This overview is not intended to identify any important or essential elements or to define any scope of any particular embodiment or any scope of any claim. Its sole purpose is to present the concepts in a simplified form as a prelude to the more detailed description presented later. One or more embodiments described herein describe a system, computer implementation method, apparatus, and / or computer program product that can infer one or more insights regarding the learning behavior of an AI model.

[0004] According to one embodiment, a system is provided. The system may include memory capable of storing computer executable components. The system may also include a processor operably coupled to the memory and capable of executing the computer executable components stored in memory. The computer executable components may include a model introspection component capable of analyzing the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the artificial intelligence model with respect to multiple test data subsets having different code complexity distributions. An advantage of such a system may be the ability to identify how the code complexity of the samples analyzed by the AI ​​model may affect the performance of the AI ​​model.

[0005] In some examples, the system may also include an extraction component capable of extracting code complexity metrics from multiple code samples contained in a test dataset. The advantage of such a system may be that it can use various code complexity metrics to characterize one or more aspects of code complexity in the samples.

[0006] According to another embodiment, a system is provided. The system may include memory capable of storing computer executable components. The system may also include a processor operably coupled to the memory and capable of executing the computer executable components stored in memory. The computer executable components may include a model introspection component capable of inferring the learning behavior of an artificial intelligence model for a code comprehension task based on the performance of the artificial intelligence model on multiple test data subsets having different code complexity distributions. An advantage of such a system may be that it can introspect AI models from a dataset-driven perspective.

[0007] In some examples, the system may also include a test data subset component capable of generating multiple test data subsets by grouping multiple code samples based on performance metrics used to evaluate the performance of the artificial intelligence model. The advantage of such a system can be the tuning of AI model introspection based on one or more target performance metrics.

[0008] According to one embodiment, a computer implementation method is provided. A computer implementation method may include a step in which a system operablely coupled to a processor analyzes the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the AI ​​model with respect to multiple test data subsets having different code complexity distributions. An advantage of such a computer implementation method is that it can analyze whether the AI ​​model learned task-related signals during training.

[0009] In some examples, a computer implementation method may include a step in which the system determines the distribution of multiple code samples within multiple test data subsets based on a code complexity metric. The advantage of such a computer implementation method is that it allows for the analysis of the sample population with respect to code complexity across multiple performance results of the AI ​​model.

[0010] According to another embodiment, a computer implementation method is provided. The computer implementation method may include a step in which a system operably coupled to a processor infers the learning behavior of an artificial intelligence model for a code comprehension task based on the performance of the artificial intelligence model on multiple test data subsets having different code complexity distributions. The advantages of such a computer implementation method may be, for example, the deciphering of trained model logic, the acceleration of model evolution, the evaluation of the model design space, and / or the development of insights regarding dataset segmentation.

[0011] In some examples, the computer implementation method may include a step in which the system compares a first distribution of code samples associated with a first test data subset from multiple test data subsets with a second distribution of code samples associated with a second test data subset from multiple test data subsets. The advantage of such a computer implementation method is that it can identify sample aspects that may influence the distribution of samples among the test data subsets, where each test data subset can be associated with a defined AI model performance condition.

[0012] According to one embodiment, a computer program product for code complexity-driven model introspection is provided. The computer program product may include a computer-readable storage medium having program instructions to be implemented. The program instructions are transmitted to a processor The processor may be able to perform the analysis of the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the AI ​​model with respect to multiple test data subsets having different code complexity distributions. The advantage of such a computer program product is that it can be an introspection method that is independent of the type of AI model and / or the source code comprehension task.

[0013] In some examples, the program instructions may cause the processor to extract a code complexity metric for a plurality of code samples included in the test dataset. The program instructions may also cause the processor to generate a plurality of test data subsets by grouping the plurality of code samples based on a performance metric for evaluating the output of the artificial intelligence model. Furthermore, the program instructions may cause the processor to determine the distribution of the plurality of code samples within the plurality of test data subsets based on the code complexity metric. The program instructions may also cause the processor to compare a first distribution of code samples associated with a first test data subset from the plurality of test data subsets with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets. The advantage of such a computer program product is that it can group samples by prediction accuracy and use a code complexity metric to determine the learning behavior of the AI ​​model with respect to aspects of the code that are easily understood by the AI ​​model. [Brief explanation of the drawing]

[0014] [Figure 1]A block diagram of an exemplary, non-limiting system is shown, which allows for introspection of the learning behavior of one or more AI models according to one or more embodiments described herein.

[0015] [Figure 2] A block diagram of an exemplary, non-limiting system capable of generating one or more test datasets with different code complexity levels, according to one or more embodiments described herein, is shown.

[0016] [Figure 3] A block diagram of an exemplary, non-limiting system capable of determining the distribution of source code samples within a test data subset with respect to one or more code complexity metrics, according to one or more embodiments described herein, is shown.

[0017] [Figure 4] The following are illustrative, non-limiting graphs that demonstrate one or more relationships between the code complexity of test data and the accuracy of AI model predictions, as described in one or more embodiments of this specification.

[0018] [Figure 5] A block diagram of an exemplary, non-limiting system is shown, which allows a subset of test data having a distribution of source code samples to be compared and / or contrasted with different amounts and / or types of code complexity, according to one or more embodiments described herein.

[0019] [Figure 6] The diagram shows an exemplary, non-limiting introspection scheme that can be implemented to infer one or more insights about the learning behavior of one or more trained AI models according to one or more embodiments described herein.

[0020] [Figure 7]A diagram of an exemplary non - limiting graph chain is shown that can demonstrate how one or more insights regarding the learning behavior of a trained AI model can be inferred from multiple test data subsets, according to one or more embodiments described herein.

[0021] [Figure 8] A flowchart of an exemplary non - limiting computer - implemented method is shown that enables introspecting a trained AI model to analyze the learning behavior and / or the evolution of training of the trained AI model, according to one or more embodiments described herein.

[0022] [Figure 9] A flowchart of an exemplary non - limiting computer - implemented method is shown that enables introspecting a trained AI model to analyze the learning behavior and / or the evolution of training of the trained AI model, according to one or more embodiments described herein.

[0023] [Figure 10] A cloud computing environment is shown, according to one or more embodiments described herein.

[0024] [Figure 11] An abstraction model layer is shown, according to one or more embodiments described herein.

[0025] [Figure 12] A block diagram of an exemplary non - limiting operating environment is shown that facilitates one or more embodiments described herein.

Best Mode for Carrying Out the Invention

[0026] The following detailed description is illustrative and not intended to limit any embodiments, applications, uses, or combinations thereof. Furthermore, it is not intended to be limited by any representations or implied information presented in the background art section, the summary section of the invention, or the section on embodiments for carrying out the invention.

[0027] Here, one or more embodiments are described with reference to the drawings, where similar reference numerals are used throughout to indicate similar elements. In the following description, many specific details are included for illustrative purposes to provide a more complete understanding of one or more embodiments. However, it will be apparent that in various cases one or more embodiments can be carried out without these specific details.

[0028] Considering the problems associated with other implementations of introspection of trained AI models, this disclosure can be implemented to create solutions to one or more of these problems by enabling introspection of the learning behavior of one or more AI models through a code complexity-based approach. Advantageously, one or more embodiments described herein can introspect the learning behavior of a trained AI model from the perspective of a dataset to provide insights superior to statistical quality measures. Furthermore, the various introspection techniques described herein can be independent of the type of AI model and / or the source understanding task.

[0029] Various embodiments of the present invention may relate to computer processing systems, computer implementations, apparatus, and / or computer program products that facilitate efficient, effective, and autonomous (e.g., without direct human guidance) introspection of one or more trained AI models. For example, one or more embodiments described herein may determine one or more insights regarding the evolution of a trained AI model based on one or more code metrics, e.g., one or more code complexity metrics. For example, one or more embodiments may infer learning behavior insights of one or more trained AI models by comparing code metric distributions across test data subsets grouped by performance metrics such as prediction accuracy. Insights that can be determined through the various embodiments described herein may relate to, for example, dataset segmentation, deciphering the learning model logic, signal recognition of the AI ​​model, evolution of the AI ​​model during training, and / or evaluation of the AI ​​model design space. For example, one or more embodiments described herein can compare common characteristics of test data samples correctly predicted by the trained AI model with characteristics of test data samples incorrectly predicted by the AI ​​model to determine aspects of a source code sample that the AI ​​model has learned well from and aspects that could be targeted for further improvement learning.

[0030] The various embodiments described herein are illustrated with respect to source code comprehension tasks; however, the architecture is not limited in this respect. For example, one or more embodiments described herein can introspect an AI model used to perform machine learning tasks, including code comprehension. Furthermore, the various embodiments described herein are illustrated with respect to the use of source code samples as test data; however, the architecture is not limited in this respect. For example, one or more embodiments described herein can utilize binary code samples and / or assembly code samples as test data in introspection of a trained AI model.

[0031] Computer processing systems, computer implementation methods, devices, and / or computer program products utilize hardware and / or software to solve problems of a highly technical nature that cannot be performed as a set of human mental activities rather than abstract ones (e.g., introspection of one or more AI models). For example, one or more individuals cannot infer insights into the evolution of an AI model through training based on a comparison of test data subsets having sample distributions of different code complexity. Furthermore, one or more embodiments described herein can constitute a technical improvement over conventional AI model introspection by inferring one or more insights into whether a trained AI model has learned signals relevant to a machine learning task to be performed. Moreover, various embodiments described herein can demonstrate a technical improvement over conventional AI model introspection by identifying how one or more training variations may affect the learning behavior of a trained AI model.

[0032] Furthermore, one or more embodiments described herein may have practical applications by identifying one or more characteristics of training data samples that make a sample easier or harder for one or more AI models to accurately predict. In another example, the various embodiments described herein can decipher which aspects of a code sample are learned by the trained AI model or ignored. Furthermore, one or more embodiments described herein may have practical applications by targeting one or more sample characteristics (e.g., aspects of code) that may be common to the mispredictions of one or more AI models. One or more embodiments described herein may control multiple hyperparameter tuning iterations to track improvements in AI model understanding and / or derive one or more of the exemplary insights described herein.

[0033] Figure 1 shows a block diagram of an exemplary, non-limiting system 100 capable of analyzing the learning behavior of one or more trained AI models 124. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. The systems (e.g., system 100), apparatus, or process embodiments in various embodiments of the present invention may constitute one or more machine-executable components, embodied in one or more machines, for example, in one or more computer-readable media associated with one or more machines. When executed by one or more machines (e.g., computers, computing devices, virtual machines, or combinations thereof), such components can cause the machines to perform the operations described.

[0034] As shown in Figure 1, the system 100 may comprise one or more servers 102, one or more networks 104, and / or one or more input devices 106. The server 102 may comprise a model introspection component 110. The model introspection component 110 may further comprise a communication component 112 and / or an extraction component 114. The server 102 may also comprise at least one memory 116, or may be associated with it in a different way. The server 102 may further comprise a system bus 118 that can be coupled to various components, such as, but not limited to, the model introspection component 110, and associated components, memory 116, and / or a processor 120. Although the server 102 is shown in Figure 1, in other embodiments, various types of multiple devices may be associated with or include the features shown in Figure 1. Furthermore, the server 102 may communicate with one or more cloud computing environments.

[0035] One or more networks 104 may comprise wired and wireless networks, including, but are not limited to, cellular networks, wide area networks (WANs) (e.g., the Internet), or local area networks (LANs). For example, server 102 may communicate with one or more input devices 106 (and vice versa) using virtually any desired wired or wireless technology, including, but are not limited to, cellular, WAN, Wireless Fidelity (Wi-Fi®), Wi-Max, WLAN, Bluetooth® technology, and / or combinations thereof. Furthermore, while the model introspection component 110 may be provided on one or more servers 102 in the embodiments shown, it should be understood that the architecture of system 100 is not so limited. For example, model introspection component 110, or one or more components of model introspection component 110, may be located on other computer devices, such as other server devices and / or client devices.

[0036] One or more input devices 106 may include, but are not limited to, one or more computerized devices such as personal computers, desktop computers, laptop computers, mobile phones (e.g., smartphones), computerized tablets (e.g., including a processor), smartwatches, keyboards, touchscreens, mice, and / or combinations thereof. One or more input devices 106 may be used to input one or more test datasets 122 and / or trained AI models 124 into the system 100, thereby allowing the data to be shared with the server 102 (e.g., via a direct connection and / or via one or more networks 104). For example, one or more input devices 106 may transmit data to a communication component 112 (e.g., via a direct connection and / or via one or more networks 104). Furthermore, one or more input devices 106 may have one or more displays capable of presenting one or more outputs generated by the system 100 to the user. For example, one or more displays may include, but are not limited to, cathode ray tube displays ("CRT"), light-emitting diode displays ("LED"), electron-emitting diode displays ("ELD"), plasma display panels ("PDP"), liquid crystal displays ("LCD"), organic light-emitting diode displays ("OLED"), and / or combinations thereof.

[0037] In various embodiments, one or more input devices 106 and / or one or more networks 104 can be used to input one or more settings and / or commands to the system 100. For example, in various embodiments described herein, one or more input devices 106 can be used to operate and / or control the server 102 and / or related components. Furthermore, one or more input devices 106 can be used to display one or more outputs (e.g., displays, data, visualizations, etc.) generated by the server 102 and / or related components. Furthermore, in one or more embodiments, one or more input devices 106 can be provided within a cloud computing environment and / or operably coupled to that cloud computing environment.

[0038] For example, in one or more embodiments, one or more input devices 106 may be used to input one or more test datasets 122 into the system 100, and the one or more test datasets 122 may be stored in one or more memories 116. Furthermore, the model introspection component 110 may use one or more test datasets 122 to determine one or more insights about the learning behavior of one or more trained AI models 124 (for example, for one or more code comprehension tasks). In one or more embodiments, one or more test datasets 122 may comprise one or more code samples, such as source code samples 126. Furthermore, one or more test datasets 122 may comprise other types of code samples, such as binary code samples and / or assembly code samples. Furthermore, one or more samples of the test dataset 122 (e.g., source code samples 126) may include, for example, samples extended from the original code dataset, ranked samples, randomized samples, and / or combinations thereof.

[0039] In one or more embodiments, one or more input devices 106 can also be used to input one or more trained AI models 124 into the system 100, which are stored in one or more memories 116 and / or can be analyzed by the model introspection component 110. Furthermore, in various embodiments, one or more input devices 106 can be used to define one or more settings and / or parameters of the introspection of the trained AI models 124. For example, one or more input devices 106 can be used to define one or more code complexity metrics according to one or more embodiments described herein. In various embodiments, the communication component 112 can share one or more inputs (e.g., input to the system 100 via one or more input devices 106) with the relevant components of the model introspection component 110 according to one or more embodiments described herein.

[0040] In various embodiments, one or more trained AI models 124 can be used to facilitate one or more machine learning tasks (e.g., code comprehension tasks such as source code comprehension tasks). In one or more embodiments, the trained AI model 124 can simulate a number of interconnected processing units, which may resemble an abstract version of neurons. For example, the processing units can be arranged in multiple layers (e.g., one or more input layers, one or more hidden layers, and / or one or more output layers) connected with different connection strengths (e.g., which may commonly be called "weights" in the art). The trained AI model 124 can be trained through one or more training schemes, where data with known outcomes is input to the computer model, the output with respect to that data is compared to the known outcomes, and / or the weights of the computer model are autonomously adjusted to replicate the known outcomes based on that comparison. As the model is trained, the computer model can become increasingly accurate; thus, the trained AI model 124 can accurately analyze data with unknown outcomes based on the lessons learned from the training data to facilitate one or more machine learning tasks. Exemplary trained AI models 124 include, but are not limited to, perceptrons ("perceptron: P"), feedforward ("feed forward: FF"), radial basis networks ("radial basis network: RBF"), deep feedforward ("deep feedforward: DFF"), recurrent neural networks ("recurrent neural network: RNN"), long / short term memory ("long / short term memory: LSTM"), gated recurrent units ("gated recurrent unit: GRU"), autoencoders ("autoencoder: AE"), variational AEs ("variational AE: VAE"), denoising AEs ("denoising AE: DAE"), sparse AEs ("sparse AE: SAE"), and Markov chains ("markov").Chain (MC), Hopfield network (HN), Boltzmann machine (BM), deep belief network (DBN), deep convolutional network (DCN), deconvolutional network (DN), deep convolutional inverse graphics network (DCIGN), generative adversarial network (GAN), liquid state machine (LSM), extreme learning machine (ELM), echo state network (ESN), deep residual network (DRN), Kohonen network (KN), support vector machine (SVM), and / or neural Turing machine (neural Turing machine:NTM) may be included. Furthermore, the various embodiments described herein are also applicable to classical machine learning models.

[0041] In one or more embodiments, the model introspection component 110 can infer one or more insights into the learning behavior of one or more trained AI models 124 by utilizing one or more test datasets 122 to compare code characteristics with respect to one or more performance metrics (e.g., across correctly and incorrectly predicted source code samples 126). Exemplary metrics that can characterize one or more code characteristics may include code complexity metrics 128. As used herein, the term “code complexity” can refer to the complexity inherent in a given computer application source code. Code complexity can affect the efficiency of computer software. For example, as code complexity increases, the probability that a source code sample contains defects may also increase. Furthermore, code complexity can be characterized and / or quantified by one or more code complexity metrics 128.

[0042] In various embodiments, the extraction component 114 can extract one or more code complexity metrics 128 from one or more training datasets 122 of one or more test datasets 122 (e.g., from source code samples 126). In one or more embodiments, code complexity can be measured in terms of cyclomatic complexity and / or Halsted complexity via one or more code complexity metrics 128 and / or as a function of the multiple code complexity metrics 128. For example, one or more code complexity metrics 128 may be, but are not limited to, cyclomatic complexity metrics, source lines of code (SLOC) (e.g., number of lines of source code and / or executable code), Halsted volume, maintainability index, coupling / inheritance depth, cognitive complexity, rework rate, program volume, program difficulty calculation (e.g., difficulty in writing or understanding the source code), effort calculation (e.g., degree of coding time), degree of program length, degree of program vocabulary, number of classes, number of comment lines, comment line density, number of directories, number of files, number of functions, number of projects in a view, number of statements, and / or combinations thereof. In one or more embodiments, one or more code complexity metrics 128 may be a function of one or more semantic artifacts contained in one or more test data samples (e.g., source code sample 126). Examples of artifacts, though not limited to them, may include decision points, if conditions, iterative loops, the number of linearly independent paths, the number of commits modifying a given source code, the number of developers contributing to a given source code, the number of conversations associated with a given source code, and / or combinations thereof. For example, code complexity can be calculated based on the number of paths through source code sample 126. If the control flow of a function splits, code complexity can increase (for example, the complexity counter may increase incrementally with each split).

[0043] In various embodiments, the extraction component 114 can analyze each sample (e.g., source code sample 126) contained in one or more test datasets 122 and extract one or more code complexity metrics 128 from the samples (e.g., source code sample 126) to determine the code complexity of the samples (e.g., source code sample 126). For example, the extraction component 114 can characterize the code complexity of a source code sample 126 by extracting one type of code complexity metric 128 from each of the source code samples 126. In another example, the extraction component 114 can characterize the code complexity of a source code sample 126 by extracting multiple types of code complexity metrics 128 from each source code sample 126 (e.g., extracting two or more code complexity metrics 128 per source code sample 126). In various embodiments, the extraction component 114 can characterize the code complexity of the source code sample 126 as a function of multiple code complexity metrics 128 extracted from the source code sample 126.

[0044] The extraction method used by the extraction component 114 to extract one or more code complexity metrics 128 may vary depending on the code complexity metrics 128 to be extracted. For example, the extraction component 114 may utilize Frama-C to extract one or more code complexity metrics 128. For example, one or more code complexity metrics 128 may be extracted by counting one or more relevant artifacts in the source code sample 126 (e.g., counting the number of lines of code). In another case, the extraction component 114 may extract one or more code complexity metrics 128 using one or more analyzer tools, including, but not limited to, cyclomatic complexity analyzers (e.g., Lizard analyzer tool), McCabe complexity analyzers (e.g., mccabe analyzer tool), pycodequal analyzer tool, xenon analyzer tool, cyclocomp analyzer tool, saikuro analyzer tool, cmetrics analyzer, cscout analyzer tool, and / or combinations thereof.

[0045] In one or more embodiments, the extraction component 114 may store one or more extracted code complexity metrics 128 together with identification information of the associated source code samples 126 in one or more complexity databases 130 in one or more memories 116. For example, one or more complexity databases 130 may include one or more charts, tables, and / or logs, etc., that enumerate the analyzed source code samples 126 and the extracted code complexity metrics 128 associated with each analyzed source code sample 126.

[0046] Figure 2 shows an exemplary, non-limiting system 100 further comprising a test data subset component 202 according to one or more embodiments described herein. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. In one or more embodiments, the model introspection component 110 can perform one or more source code understanding tasks (defined, for example, via one or more input devices 106) on one or more trained AI models 124. Thereafter, one or more trained AI models 124 can generate one or more outputs (e.g., predictions) when completing the source code understanding tasks. In one or more embodiments, the outputs of the trained AI models 124 can be input to the system 100 via one or more input devices 106.

[0047] In various embodiments, the test data subset component 202 can analyze one or more outputs of the trained AI model 124 with respect to one or more performance metrics, such as accuracy, precision, and / or recall. For example, the test data subset component 202 can determine the accuracy of one or more predictions generated by the trained AI model 124 when performing a defined source code comprehension task on one or more test datasets 122. For example, the correct output (e.g., prediction) associated with each source code sample 126 may be known to the test data subset component 202 (e.g., it may be input to the system 100 via one or more input devices 106), and the test data subset component 202 can test the accuracy of the trained AI model 124 by comparing the known correct outputs with the outputs of the trained AI model 124.

[0048] Furthermore, the test data subset component 202 can group test samples (e.g., one or more source code samples 126) into one or more test data subsets 204 based on the performance of one or more trained AI models 124 (e.g., based on accuracy, precision, recall, signal recognition recall, and / or a combination thereof). For example, the test data subset component 202 can group one or more source code samples 126 to generate one or more test data subsets 204 that can be stored in one or more memories 116. Exemplary test data subsets 204 generated when evaluating the performance of the trained AI model 124 with respect to accuracy may include, but are not limited to, true positive ("TP") subsets, false negative ("FN") subsets, true negative ("TN"), false positive ("FP"), and / or combinations thereof. For example, the test data subset component 202 can group source code samples 126 that have been correctly analyzed as positive by one or more trained AI models 124 into a TP test data subset 204. Alternatively, the test data subset component 202 can group source code samples 126 that have been incorrectly analyzed as negative by one or more trained AI models 124 into an FN test data subset 204. In various embodiments, the test data subset component 202 can group a test sample (e.g., source code sample 126) into two, three, or more test data subsets 204. In one or more embodiments, the performance metrics used by the test data subset component 202 to evaluate one or more trained AI models 124 can be defined via one or more input devices 106 and / or based on a source code comprehension task.Furthermore, in one or more embodiments, the number and / or type of test data subsets 204 used by the test data subset component 202 to group source code samples 126 can be defined via one or more input devices 106 and / or based on a source code comprehension task.

[0049] Figure 3 shows an exemplary, non-limiting system 100 further comprising a distribution component 302 according to one or more embodiments described herein. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. In one or more embodiments, the distribution component 302 can determine the distribution of test samples (e.g., source code samples 126) contained within a test data subset 204 with respect to one or more of the extracted code complexity metrics 128.

[0050] For example, the distribution component 302 can identify an extracted code complexity metric 128 associated with each of the source code samples 126 contained within the test data subset 204. For example, the distribution component 302 can determine how many of the source code samples 126 in a given test data subset 204 (e.g., as a percentage of the total population of the test dataset 122 and / or the population of the test data subset 204) have a given value of the code complexity metric 128. In another example, the distribution component 302 can determine the range of values ​​for the code complexity metric 128 that characterize the source code samples 126 in a given test data subset 204. In yet another example, the distribution component 302 can determine how often the conditions of a given test data subset 204 are met by source code samples 126 having a given value of the code complexity metric 128.

[0051] In one or more embodiments, the distribution component 302 can generate one or more displays to represent the distribution of the source code sample 126. For example, one or more displays may include, but are not limited to, one or more graphs, charts, tables, text, images, videos, and / or combinations thereof. Furthermore, the distribution component 302 can share one or more displays with one or more input devices 106 (for example, via one or more networks 104).

[0052] Figure 4 shows exemplary, non-limiting graphs 402 and / or 404 that can be generated by the distribution component 302 to represent the distribution of code complexity within each source code sample 126 of the test data subset 204 generated by the test data subset component 202. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. Figure 4 shows one or more bar graphs for representing the distribution of source code samples 126 with respect to one or more code complexity metrics 128; however, the architecture of the model introspection component 110 is not limited in this way. For example, the form of the bar graphs shown in Figure 4 is exemplary, and the distribution component 302 can represent the distribution of code complexity metrics 128 through various display formats.

[0053] To demonstrate the effectiveness of the various embodiments described herein, System 100 was used for a vulnerability detection use case on multiple datasets; however, the various embodiments are applicable to other source code understanding tasks. An exemplary test dataset 122 used to demonstrate the effectiveness of System 100 includes an s-bAbI synthetic dataset ("s-bAbI"). The s-bAbI test dataset 122 can be used to demonstrate the effectiveness of the various embodiments described herein; on the other hand, the features of the model introspection component 110 are not limited to the exemplary s-bAbI test dataset 122, but rather other source code datasets can be analyzed. The s-bAbI test dataset 122 contains a syntactically valid C program with a non-trivial control flow and focuses on buffer overflow vulnerabilities. For the purpose of vulnerability detection tasks, an s-bAbI generator can be used to create a dataset, where samples with the "unsafe" tag can be labeled 1 and samples with the "safe" tag can be labeled 0. Furthermore, in one or more embodiments, a trained graph neural network ("GNN") AI model 124 can be used to demonstrate the effectiveness of system 100. However, the various features of the model introspection component 110 are not limited to the type of trained GNN AI model 124.

[0054] Graphs 402 and / or 404 relate to the distribution of source code samples 126 across two test data subsets 204 generated from running a trained GNN AI model 124 on the s-bAbI test dataset 122. For example, source code samples 126 that yield TP output from the trained AI model 124 can be grouped into the first test data subset 204a. Similarly, source code samples 126 that yield FN output from the trained AI model 124 can be grouped into the second test data subset 204b. Graph 402 shows the distribution of source code samples 126 in the first test data subset 204a with respect to one or more of the code complexity metrics 128. Graph 404 shows the distribution of source code samples 126 in the second test data subset 204b with respect to the same one or more code complexity metrics 128. For example, Graph 402 shows that 58.2 percent of the TP occurrences achieved by the trained AI model 124 were associated with source code samples 126 that were judged to have a difficulty code complexity metric value of 12. In another example, Graph 404 shows that 54.7 percent of the FN occurrences achieved by the trained AI model 124 were associated with source code samples 126 that were judged to have a difficulty code complexity metric value of 16.

[0055] Figure 5 shows an exemplary, non-limiting system 100 further comprising a comparison component 502 according to one or more embodiments described herein. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. In various embodiments, the comparison component 502 can compare and / or contrast the distribution of test samples (e.g., source code samples 126) generated by the distribution component 302 across multiple test data subsets 204.

[0056] In one or more embodiments, the comparison component 502 can compare the distribution of source code samples 126 in one test data subset 204 with the distribution of source code samples 126 in one or more other test data subsets 204 to facilitate the identification of one or more similarities and / or differences between the distributions. For example, the comparison component 502 can compare the distribution of source code samples 126 in a first test data subset 204a represented by graph 402 with the distribution of source code samples 126 in a second test data subset 204b represented by graph 404. For example, the comparison component 502 can compare the range of values ​​of a code complexity metric 128 associated with the distribution of source code samples 126 across the test data subsets 204. In another example, the comparison component 502 can compare a population of source code samples 126 with a predefined code complexity metric 128 across the test data subsets 204. In a further instance, the comparison component 502 can compare the population of source code samples 126 across a subset of test data 204. In one or more embodiments, the comparison and / or contrast performed by the comparison component 502 may depend on one or more parameters and / or purposes defined by one or more input devices 106.

[0057] In various embodiments, the model introspection component 110 can generate one or more insights regarding the learning behavior and / or training evolution of one or more trained AI models 124 based on comparisons and / or contrasts performed by the comparison component 502. For example, with respect to the exemplary source code sample distribution 126 shown in Figure 4, the comparison component 502 can compare the range of values ​​of the code complexity metric 128 embodied by the distribution to determine that the majority of the source code samples 126 included in the first test data subset 204a (e.g., TP results) have a code complexity metric 128 value equal to or less than 14, and the majority of the source code samples 126 included in the second test data subset 204b (e.g., FN results) have a code complexity metric 128 value equal to or greater than 16. Based on this comparison, the model introspection component 110 can determine the insight that the accuracy of the trained AI model 124 may depend on the code complexity of the source code sample 126; here, the trained AI model 124 can make accurate predictions for source code samples 126 with a code complexity metric 128 value of 14 or less, but its accuracy may drop significantly for source code samples 126 with a code complexity metric 128 value of 16 or higher.

[0058] Thus, the evaluation metrics used to classify the source code samples 126 into the test data subset 204, combined with the distribution of the source code samples 126 in the test data subset 204, can indicate one or more decisions made by the model introspection component 110. In one or more embodiments, the model introspection component 110 can analyze the distribution of the test data subset 204 (e.g., generated by the distribution component 302) and / or comparisons (e.g., generated by the comparison component 502) to determine a number of insights, such as dataset segmentation and introspection, deciphering trained model logic, deriving signal recognition, evolving code-centric models, evaluating design space from the perspective of datasets, comparing code-centric models, and / or combinations thereof.

[0059] For example, if the test data subset 204 is generated based on the accuracy of the output of the trained AI model 124, the model introspection component 110 can determine, based on the population of the test data subset 204, which source code samples 126 are more or less predictable for the trained AI model 124. Furthermore, the model introspection component 110 can identify one or more characteristics that may make a source code sample 126 more or less predictable, based on the distribution of the source code samples 126 within each test data subset 204. For example, the introspection component 110 can identify one or more commonalities between the distributions of the source code samples 126 based on comparisons performed by the comparison component 502.

[0060] In another example, the model introspection component 110 can determine whether the signals learned by the trained AI model 124 are relevant to a given machine learning task, or whether the signals learned by the trained AI model 124 contain noise and / or dataset bias. For example, the model introspection component 110 can determine that the trained AI model 124 has learned about signals unrelated to a given task based on the fact that the population of source code samples 126 is small in the test data subset 204 associated with accurate results, and / or that the test data subset 204 associated with inaccurate results contains source code samples 126 with greater code complexity. In yet another example, the model introspection component 110 can identify common code characteristics in the mispredictions made by the trained AI model 124.

[0061] In a further example, the trained AI model 124 can be analyzed across multiple iterations of its execution with different hyperparameter settings. This allows the model introspection component 110 to analyze the distribution of source code samples 126 across test data subsets 204 and execution iterations to track model understanding related to hyperparameter settings. In a further example, the trained AI model 124 can be analyzed by the model introspection component 110 across multiple test datasets 122, each containing different operations and / or extensions to the composition of source code samples 126. Furthermore, the trained AI model 124 can be analyzed by the model introspection component 110 across multiple test datasets 122, with different order in which the source code samples 126 are introduced into the trained AI model 124. In a further example, the model introspection component 110 can analyze multiple trained AI models 124 with respect to the same test dataset 122, and each trained AI model 124 can be trained in different ways. This allows the model introspection component 110 to analyze the distribution of source code samples 126 across test data subsets 204, test dataset 122, various model configurations, and / or various model training schemes to track model improvement techniques such as data augmentation, curriculum learning, active learning, adversarial training, and / or combinations thereof, and / or determine the task suitability across models.

[0062] Figure 6 shows an exemplary, non-limiting introspection scheme 600 that can be implemented by the model introspection component 110 according to one or more embodiments described herein. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. Figure 6 shows an exemplary introspection scheme 600; however, the architecture of system 100 is not limited to the order of events or number of features shown in Figure 6. For example, one or more features of the introspection scheme 600 can be performed in an alternative order (e.g., the code complexity metric 128 can be extracted from source code sample 126 before or after generating test data subset 204). In another example, although two test data subsets 204 are clearly shown in Figure 6, it is possible to generate additional test data subsets 204 (e.g., as indicated by "..." in Figure 6).

[0063] As shown in Figure 6, one or more trained AI models 124 can perform one or more machine learning tasks (e.g., source code comprehension tasks) using one or more test datasets 122. According to various embodiments, the execution of the trained AI models 124 can be performed by a model introspection component 110 and / or independently, where the results of the execution can be supplied to the model introspection component 110 via one or more input devices 106. Based on the output of one or more trained AI models 124, a test data subset component 202 can group the source code samples 126 of the test dataset 122 into multiple test data subsets 204 based on one or more conditions associated with each test data subset 204. For example, the condition associated with a first test data subset 204a may be that the first test data subset 204 includes source code samples 126 associated with TP results from one or more trained AI models 124. In another example, the condition associated with the second test data subset 204b may be that the second test data subset 204 includes source code samples 126 associated with FN results from one or more trained AI models 124. In one or more embodiments, the conditions associated with multiple test data subsets 204 can be predefined based on a given task performed by one or more trained AI models 124 and / or defined by the test data subset component 202. In one or more embodiments, one or more conditions associated with multiple test data subsets 204 can be defined via one or more input devices 106.

[0064] Furthermore, the extraction component 114 can extract one or more code complexity metrics 128 from each of the source code samples 126 contained in multiple test data subsets 204. The distribution component 302 can then determine one or more distributions of the source code samples 126 within the test data subsets 204 with respect to one or more of the extracted code complexity metrics 128. According to various embodiments described herein, the distribution component 302 can represent the distribution of the source code samples 126 by generating one or more displays, such as graphs (illustrated, for example, in Figure 4). Furthermore, the comparison component 502 can compare and / or contrast the distributions of the source code samples 126 across multiple test data subsets 204. According to various embodiments described herein, the model introspection component 110 can determine one or more insights into the learning behavior of one or more trained AI models 124 based on one or more comparisons generated by the comparison component 502.

[0065] Figure 7 shows an exemplary non-limiting graph chain 702, 704 that can show one or more insights that can be determined by the model introspection component 110 by utilizing multiple iterations of the exemplary introspection scheme 600 according to one or more embodiments described herein. For the sake of brevity, a description of the repetition of similar elements used in other embodiments described herein is omitted. In one or more embodiments, the model introspection component 110 can perform multiple iterations of the introspection scheme 600 across different parameter settings, trained AI models 124, and / or test datasets 122. For example, Figure 7 shows one or more comparisons and / or insights that can be generated by the model introspection component 110 when multiple iterations of the exemplary introspection scheme 600 are performed across multiple trained AI models 124, each trained on training datasets with different amounts of augmentation.

[0066] For example, the model introspection component 110 can infer the learning behavior of a trained AI model 124 by comparing the distribution of source code samples 126 from a test data subset 204 (defined, for example, based on prediction accuracy) with respect to code complexity across multiple trained AI models 124 trained on datasets having the same architecture and / or configuration but with different amounts of augmentation compared to the base dataset. As shown in Figure 7, in the first introspection iteration 706a, the model introspection component 110 can analyze a trained AI model 124 trained on the base dataset (e.g., with 0% augmentation). Subsequently, the model introspection component 110 can perform multiple iterations of the introspection scheme 600. For example, the model introspection component 110 can perform a second introspection iteration 706b, a third introspection iteration 706c, and / or a fourth introspection iteration 706d with respect to a trained AI model 124 trained on datasets containing 1% augmentation, 2% augmentation, and 20% augmentation, respectively. Figure 7 shows four iterations of the introspection scheme 600, but the architecture of system 100 is not limited in this way. For example, the model introspection component 110 can be executed more or fewer than four iterations.

[0067] For each introspection iteration 706 of the introspection scheme 600 (e.g., the first introspection iteration 706a, the second introspection iteration 706b, the third introspection iteration 706c, and / or the fourth introspection iteration 706d), the model introspection component 110 can analyze the trained AI model 124 with respect to the first test data subset 204a, which includes the source code sample 126, based on the recall (signal recognition) TP results (e.g., represented in the graph chain 702). Furthermore, for each introspection iteration 706 of the introspection scheme 600 (e.g., the first introspection iteration 706a, the second introspection iteration 706b, the third introspection iteration 706c, and / or the fourth introspection iteration 706d), the model introspection component 110 can analyze the trained AI model 124 with respect to a second test data subset 204b containing source code samples 126, based on the FN results of recall (signal recognition) (e.g., represented in graph chain 704).

[0068] The signal recognition recall metric can be used to demonstrate improvements in model performance. The signal recognition recall metric can be used to measure the signal recognition of an AI model for code. For example, in the context of a vulnerability detection task, recall can measure the percentage of vulnerable samples that a model correctly predicts, while the signal recognition recall metric can measure how many of these instances the model captures the correct signal to arrive at a prediction. For example, in the context of vulnerability detection, the signal recognition recall metric can measure the percentage of vulnerable samples that a trained AI model 124 correctly predicts (e.g., using task-relevant signals).

[0069] Furthermore, the model introspection component 110 can analyze the distribution of source code samples 126 within the test data subset 204 with respect to the SLOC code complexity metric 128. However, the architecture of the model introspection component 110 is not limited to the exemplary test data subset 204 and / or code complexity metric 128 shown in Figure 7. For example, the model introspection component 110 can perform exemplary introspection schemes 600 with respect to other test data subsets 204 (e.g., based on evaluation metrics other than accuracy) and / or other code complexity metrics 128.

[0070] Based on multiple introspection iterations, the model introspection component 110 can determine that the trained AI model 124, trained on the base dataset, may face the greatest challenge in understanding large source code samples 126 (e.g., source code samples 126 with large SLOC values). However, the model introspection component 110 can determine that a given augmentation scheme may be effective in improving the learning behavior of the architecture and / or configuration of the trained AI model 124. For example, as shown in the comparison of dotted lines 708 and 710, as the amount of augmentation used to train the architecture and / or configuration of the trained AI model 124 increases, the number of large source code samples 126 correctly predicted by the trained AI model 124 (e.g., as defined by the SLOC code complexity metric 128) increases, and the number of large source code samples 126 mispredicted by the trained AI model 124 (e.g., as defined by the SLOC code complexity metric 128) decreases. By identifying trends represented by dotted lines 708 and / or 710 across multiple introspection iterations 706, the model introspection component 110 can determine the insight that an extension scheme tested across multiple replicas of the architecture and / or configuration of the trained AI model 124 may result in improved model understanding of a large (e.g., SLOC-defined) source code sample 126.

[0071] Figure 8 shows an exemplary, non-limiting computer implementation method 800 that facilitates introspection of one or more trained AI models 124 according to one or more embodiments described herein. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity.

[0072] In 802, the computer implementation method 800 may include a step of extracting one or more code complexity metrics 128 for a plurality of code samples (e.g., source code samples 126) contained in one or more test datasets 122 by a system 100 operably coupled to a processor 120 (e.g., via an extraction component 114). According to various embodiments described herein, the code complexity metrics 128 can characterize the code complexity of the plurality of code samples (e.g., source code samples 126) and / or can be stored in one or more memories 116 for subsequent analysis by the system 100.

[0073] In 804, the computer implementation method 800 may include a step of generating multiple test data subsets 204 by the system 100 (e.g., via a test data subset component 202) by grouping multiple code samples (e.g., source code samples 126) based on one or more performance metrics (e.g., accuracy metrics) that evaluate the output of a trained AI model 124. For example, the trained AI model 124 can complete one or more source code comprehension tasks using one or more test datasets 122, where the grouping in 804 can be based on the performance of the trained AI model 124 in performing those tasks. In one or more embodiments, the task execution can be performed by the system 100 and / or the trained AI model 124 can operate independently, and the resulting output and / or performance metrics can be shared with the system 100 (e.g., via one or more networks 104 and / or input devices 106). In 806, the computer implementation method 800 may include a step in which the system 100 (for example, via the distribution component 302) determines the distribution of multiple code samples (e.g., source code samples 126) within multiple test data subsets 204 based on the code complexity metric 128 extracted in 802. For example, Figures 4 and / or 6-7 show exemplary distributions of source code samples 126 that can be generated in 806.

[0074] In 808, the computer implementation method 800 may include a step in which the system 100 compares multiple distributions of code samples (e.g., source code samples 126) generated in 806 (e.g., via a comparison component 502). For example, the comparison in 808 may be between distributions associated with different test data subsets 204, different iterations of the computer implementation method 800 (e.g., different introspection iterations 706), different trained AI models 124, and / or different test datasets 122. In 810, the computer implementation method 800 may include a step in which the system 100 infers one or more insights into the learning behavior of one or more trained AI models 124 (e.g., via a model introspection component 110) based on the comparison in 810.

[0075] Figure 9 shows a flowchart of an exemplary, non-limiting computer implementation method 900 that facilitates the introspection of one or more trained AI models 124 in one or more embodiments described herein, from the perspective of a dataset. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity.

[0076] In 902, the computer implementation method 900 may include a step of extracting one or more code complexity metrics 128 for a plurality of code samples (e.g., source code samples 126) contained in one or more test datasets 122 by a system 100 operably coupled to a processor 120 (e.g., via an extraction component 114). According to various embodiments described herein, the code complexity metrics 128 can characterize the code complexity of the plurality of code samples (e.g., source code samples 126) and / or can be stored in one or more memories 116 for subsequent analysis by the system 100.

[0077] In 904, the computer implementation method 900 may include a step of the system collecting output data relating to the execution of one or more code understanding tasks by a trained AI model 124 on one or more test datasets 122, by the system (e.g., via a model introspection component 110 and / or a communication component 112). In one or more embodiments, the execution of tasks can be performed by the system 100 and / or the trained AI model 124 can operate independently, and the resulting output and / or performance metrics can be shared with the system 100 (e.g., via one or more networks 104 and / or input devices 106). In 906, the computer implementation method 900 may include a step of the system 100 generating multiple test data subsets 204 by grouping multiple code samples (e.g., source code samples 126) based on the output data collected in 904 (e.g., via a test data subset component 202). For example, grouping in 906 can be based on the performance of the trained AI model 124 when performing the task.

[0078] In 908, the computer implementation method 900 may include a step in which the system 100 (for example, via a distribution component 302) determines the distribution of multiple code samples (e.g., source code samples 126) within multiple test data subsets 204 based on the code complexity metric 128 extracted in 902. For example, Figures 4 and / or 6-7 show exemplary distributions of source code samples 126 that can be generated in 906. In 910, the computer implementation method 900 may include a step in which the system 100 (for example, via a comparison component 502) compares a first distribution of code samples (e.g., source code samples 126) associated with a first test data subset 204a with a second distribution of code samples (e.g., source code samples 126) associated with a second test data subset 204b.

[0079] In step 912, the computer implementation method 900 may include a step in which the system 100 determines (for example, via the model introspection component 110) whether an additional trained AI model 124 is available for introspection. If an additional trained AI model 124 is available, the computer implementation method 900 may proceed to step 914. If an additional trained AI model 124 is not available, the computer implementation method 900 may proceed to step 916. In step 914, the computer implementation method 900 may include a step in which the system 100 collects (for example, via the model introspection component 110 and / or the communication component 112) output data relating to the execution of a code comprehension task by another trained AI model 124 on one or more test datasets 122. The computer implementation method 900 may then proceed to step 906. As a result, the computer implementation method 900 can perform multiple introspection iterations 706 with respect to multiple trained AI models 124, each of which may include one or more structural variations, training variations, and / or configuration variations (for example, illustrated in Figure 7). In 916, the computer implementation method 900 may include a step in which the system 100 (for example, via the model introspection component 110) generates one or more insights about model learning based on comparisons made in at least 910. For example, the generation in 916 may include a step in which one or more characteristics of the distribution of code samples (e.g., source code samples 126) among test data subsets 204 and / or the trends of code samples (e.g., source code samples 126) among introspection iterations 706 are identified.

[0080] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in conjunction with any other type of computing environment that is currently known or may be developed in the future.

[0081] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0082] The characteristics are as follows:

[0083] On-demand self-service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically as needed, without requiring human interaction with service providers.

[0084] Broad network access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client or thick-client platforms (e.g., mobile phones, laptops, and PDAs®).

[0085] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, with various physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have no control or knowledge of the exact location of the resources provided, but there is a certain degree of location independence in that they may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).

[0086] Rapid resilience: Capabilities are provisioned quickly and flexibly, sometimes automatically, allowing for rapid scaling out or rapid release and rapid scaling in. To consumers, the available capacity for provisioning often appears unlimited and can be purchased in any quantity at any time.

[0087] Measurement Services: Cloud systems automatically control and optimize resource usage by leveraging measurement capabilities appropriate to the type of service (e.g., storage, processing, bandwidth, and active user accounts) at a certain level of abstraction. Resource utilization is monitored, controlled, and reported, thereby providing transparency to both the service providers and consumers using the services.

[0088] The service model is as follows:

[0089] Software as a Service (SaaS): The capability offered to consumers is the use of a provider's applications running on cloud infrastructure. These applications are accessible from various client devices via thin client interfaces, such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or even individual application capabilities, with the exception of limited user-specific application configuration settings.

[0090] Platform as a Service (PaaS): The capability offered to consumers is the ability to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the configuration of the deployed applications and, in some cases, the application hosting environment.

[0091] Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources, allowing consumers to deploy and run any software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they do control the operating system, storage, and deployed applications, and, in some cases, have limited control over selected networking components (e.g., host firewalls).

[0092] The deployment model is as follows:

[0093] Private Cloud: Cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may reside on-premises or off-premises.

[0094] Community Cloud: A cloud infrastructure is shared by multiple organizations to support a specific community that shares common interests (e.g., mission, security requirements, policies, and compliance considerations). The community cloud may be managed by those organizations or a third party and may reside on-premises or off-premises.

[0095] Public cloud: Cloud infrastructure is made available to the general public or large industry groups and is owned by organizations that sell cloud services.

[0096] Hybrid Cloud: This cloud infrastructure is a complex of two or more clouds (private, community, or public) that remain unique entities but are joined together by standardized or proprietary technologies that enable data and application portability (e.g., cloud bursting for load distribution across clouds).

[0097] Cloud computing environments are service-oriented, focusing on statelessness, low coupling, modularity, and semantic interoperability. At the core of cloud computing lies an infrastructure that includes a network of interconnected nodes.

[0098] Referring now to Figure 10, an exemplary cloud computing environment 1000 is shown. As illustrated, the cloud computing environment 1000 comprises one or more cloud computing nodes 1002, and local computing devices used by cloud consumers, such as a personal digital assistant (PDA) or mobile phone 1004, a desktop computer 1006, a laptop computer 1008, and / or an automotive computer system 1010, may communicate using the cloud computing nodes 1002. The nodes 1002 can communicate with each other. They may be grouped physically or virtually within one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above in this specification (not illustrated). This makes it possible for the cloud computing environment 1000 to provide infrastructure, platform, or software and / or as a service that does not require cloud consumers to maintain resources on their local computing devices for that purpose. The types of computing devices 1004-1010 shown in Figure 10 are for illustrative purposes only, and it should be understood that computing node 1002 and cloud computing environment 1000 can communicate with any type of computerized device via any type of network and / or network addressable connection (e.g., using a web browser).

[0099] Referring here to Figure 11, a set of functional abstraction layers provided by the cloud computing environment 1000 (Figure 10) is shown. Repeated descriptions of similar elements used in other embodiments described herein are omitted for brevity. It should be understood in advance that the components, layers, and functionalities shown in Figure 11 are for illustrative purposes only and that embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functionalities are provided:

[0100] The hardware and software layer 1102 comprises hardware components and software components. Examples of hardware components include a mainframe 1104; a RISC (Reduced Instruction Set Computer) architecture-based server 1106; a server 1108; a blade server 1110; a storage device 1112; and network and networking components 1114. In some embodiments, the software components include network application server software 1116 and database software 1118.

[0101] The virtualization layer 1120 provides an abstraction layer from which the following examples of virtual entities may be provided: a virtual server 1122, virtual storage 1124, a virtual network 1126 including a virtual private network, a virtual application and operating system 1128, and a virtual client 1130.

[0102] In one example, the management layer 1132 may provide the functions described below. Resource provisioning 1134 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Measurement and pricing 1136 provides cost tracking as resources are used within the cloud computing environment and billing or invoicing for the consumption of these resources. In one example, these resources may include application software licenses. Security provides identity verification for cloud consumers and tasks, as well as protection of data and other resources. User portal 1138 provides access to the cloud computing environment for consumers and system administrators. Service level management 1140 provides cloud computing resource allocation and management to ensure that required service levels are met. Service level agreement (SLA) planning and execution 1142 provides pre-arrangement and procurement of cloud computing resources where future requirements are anticipated in accordance with the SLA.

[0103] The workload layer 1144 provides examples of functions for which a cloud computing environment may be used. Examples of workloads and functions that may be provided from this layer include mapping and navigation 1146, software development and lifecycle management 1148, virtual classroom education delivery 1150, data analysis processing 1152, transaction processing 1154, and AI model introspection 1156. Various embodiments of the present invention can use a cloud computing environment described with reference to Figures 10 and 11 to introspect one or more trained AI models 124 according to one or more embodiments described herein, and / or infer insights into the learning behavior of one or more trained AI models 124.

[0104] The present invention may also be a system, method, and / or computer program product integrated at any possible level of technical detail. The computer program product may include a computer-readable storage medium (or a plurality of computer-readable storage mediums) having computer-readable program instructions for causing a processor to perform aspects of the present invention. The computer-readable storage medium may be a tangible device capable of holding and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any preferred combination thereof. A non-exclusive list of more specific examples of computer-readable storage media includes: portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved ridges on which instructions are recorded, and any suitable combination thereof. Computer-readable storage media, as used herein, should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires.

[0105] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers such instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0106] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, configuration data for integrated circuits, or source code or object code written in any combination of one or more programming languages, including Smalltalk®, C++ or similar object-oriented programming languages, and procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may run as a standalone software package, either entirely on the user's computer, partially on the user's computer, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or the connection may be to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, to carry out aspects of the present invention, an electronic circuit including, for example, a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) can execute a computer-readable program instruction by personalizing the electronic circuit using state information of the computer-readable program instruction.

[0107] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in a flowchart or block diagram, or a combination thereof, and any combination of blocks in a flowchart or block diagram, or a combination thereof, can be implemented by computer-readable program instructions.

[0108] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to create a machine, and the instructions executed via the processor of the computer or other programmable data processing device will create means for implementing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct a computer, a programmable data processing device, and / or other device to function in a particular manner, and the computer-readable storage medium having the instructions stored therein will comprise a product containing instructions that implement modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0109] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing device, or other device to execute a series of operational steps on the computer, other programmable device, or other device, thereby creating a computer implementation process in which the instructions executed on the computer, other programmable device, or other device implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0110] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or portion of instructions containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in the blocks may be performed in an order different from the order shown in the drawings. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the related functions. It should also be noted that each block or both in the block diagram or flowchart, and any combination of blocks or both in the block diagram or flowchart, may be implemented by a dedicated hardware-based system that performs a specified function or operation, or a combination of dedicated hardware and computer instructions.

[0111] To provide additional context for the various embodiments described herein, Figure 12 and the following description are intended to provide a general description of a preferred computing environment 1200 in which various embodiments of the embodiments described herein may be implemented. While embodiments have been described above in the general context of computer executable instructions that may run on one or more computers, those skilled in the art will recognize that embodiments may also be implemented in combination with other program modules, or as a combination of hardware and software, or both.

[0112] Generally, a program module includes routines, programs, components, or data structures, and / or the like, that perform a specific task or implement a specific abstract data type. Furthermore, those skilled in the art will understand that the methods of the invention can be implemented using single-processor or multi-processor computer systems, minicomputers, mainframe computers, Internet of Things ("IoT") devices, distributed computing systems, and other computer system configurations, including personal computers, handheld computing devices, or microprocessor-based electronic or programmable consumer electronics, each of which can be operably coupled to one or more associated devices.

[0113] The embodiments described herein may also be implemented in a distributed computing environment in which a particular task is performed by remote processing devices linked through a communication network. In a distributed computing environment, program modules may reside in both local and remote memory storage devices. For example, in one or more embodiments, a computer executable component may be executed from memory comprising, or composed of, one or more distributed memory units, or both. As used herein, the terms “memory” and “memory unit” are interchangeable. Furthermore, one or more embodiments described herein may execute the code of a computer executable component in a distributed manner, for example, by having multiple processors working together or in conjunction to execute the code from one or more distributed memory units. As used herein, the term “memory” may encompass one or more locations of memory, or a single memory or memory unit in memory units at one or more locations.

[0114] Computing devices typically include a variety of media, as used herein, which may include computer-readable storage media, machine-readable storage media, and / or communication media, the two terms being used as distinct from each other as follows: Computer-readable storage media or machine-readable storage media may be any available storage media that can be accessed by a computer and include both volatile and non-volatile media, and both removable and non-removable media. For example, but not limited to, computer-readable storage media or machine-readable storage media may be implemented in relation to any method or technique for storing information such as computer-readable instructions or machine-readable instructions, program modules, structured data or unstructured data.

[0115] Computer-readable storage media may include, but are not limited to, random access memory ("RAM"), read-only memory ("ROM"), electrically erasable programmable read-only memory ("EEPROM"), flash memory or other memory technologies, compact disk read-only memory ("CD-ROM"), digital versatile discs ("DVD"), Blu-ray discs ("BD") or other optical disc storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, solid-state drives or other solid-state storage devices, or other tangible or non-temporary media, or combinations thereof, that may be used to store desired information. In this regard, the terms "tangible" or "non-temporary" as used herein to apply to storage, memory, or computer-readable media should be understood to exclude only temporary signals that propagate on their own as modifiers, and not to waive any rights to all standard storage, memory, or computer-readable media that are not merely temporary signals that propagate on their own.

[0116] Computer-readable storage media can be accessed by one or more local or remote computing devices for various operations on the information stored in the media, for example, through access requests, queries, or other data retrieval protocols.

[0117] Communication media typically include any information delivery or transport medium that embodies computer-readable instructions, data structures, program modules, or other structured or unstructured data in modulated data signals, such as carrier waves or other transport mechanisms. The term “modulated data signal” or “signal” refers to a signal having one or more of its characteristic sets, or a signal that has been modified in such a way as to encode information within one or more signals. Examples, but not limited to, include wired media such as wired networks or direct wired connections, as well as wireless media such as sound waves, RF, infrared, and other wireless media.

[0118] Referring again to Figure 12, an exemplary environment 1200 for implementing various embodiments of the embodiments described herein includes a computer 1202, a computer 1202 including a processing unit 1204, system memory 1206, and a system bus 1208. The system bus 1208 connects system components, including but not limited to the system memory 1206, to the processing unit 1204. The processing unit 1204 may be any of various commercially available processors. Dual microprocessor and other multiprocessor architectures can also be used as the processing unit 1204.

[0119] The system bus 1208 may be one of several types of bus structures that can further interconnect with memory buses, peripheral buses, and local buses (with or without a memory controller) using any of the various commercially available bus architectures. The system memory 1206 includes ROM 1210 and RAM 1212. The basic input / output system ("BIOS") may be stored in non-volatile memory such as ROM, erasable programmable read-only memory ("EPROM"), or EEPROM, which contains basic routines that help transfer information between elements within the computer 1202, such as during startup. RAM 1212 may also include high-speed RAM, such as static RAM, for caching data.

[0120] Computer 1202 further includes an internal hard disk drive ("HDD") 1214 (e.g., EIDE, SATA), one or more external storage devices 1216 (e.g., magnetic floppy disk drive ("FDD") 1216, memory stick or flash drive reader, memory card reader, and / or combination thereof), and an optical disk drive 1220 (capable of reading and writing CD-ROM disks, DVDs, BDs, and / or combinations thereof). Although the internal HDD 1214 is shown as being located within computer 1202, the internal HDD 1214 may also be configured for external use in a suitable chassis (not shown). In addition, although not shown in environment 1200, a solid-state drive ("SSD") may be used in addition to or instead of the HDD 1214. The HDD 1214, external storage device 1216, and optical disc drive 1220 may be connected to the system bus 1208 by the HDD interface 1224, external storage interface 1226, and optical drive interface 1228, respectively. Interface 1224 for external drive implementations may include at least one or both of the Universal Serial Bus ("USB") and the Institute of Electrical and Electronics Engineers ("IEEE") 1394 interface technology. Other external drive connection technologies are included in the intent of the embodiments described herein.

[0121] The drive and its associated computer-readable storage medium provide non-volatile storage such as data, data structures, and computer-executable instructions. For computer 1202, the drive and storage medium accommodate the storage of any data in a suitable digital format. While the above description of computer-readable storage medium refers to each type of storage device, it should be understood by those skilled in the art that other types of computer-readable storage mediums, whether currently existing or to be developed in the future, may also be used in the exemplary operating environment, and furthermore, any such storage medium may contain computer-executable instructions for performing the methods described herein.

[0122] Several program modules, including an operating system 1230, one or more application programs 1232, other program modules 1234, and program data 1236, may be stored in the drive and RAM 1212. All or part of the operating system, applications, modules, and / or data may also be cached in RAM 1212. The systems and methods described herein may be implemented using various commercially available operating systems or combinations of operating systems.

[0123] Computer 1202 may optionally include emulation techniques. For example, a hypervisor (not shown) or other intermediary may emulate a hardware environment for operating system 1230, and the emulated hardware may optionally differ from the hardware shown in Figure 12. In such an embodiment, operating system 1230 may include one VM of multiple virtual machines ("VMs") hosted on computer 1202. Furthermore, operating system 1230 may provide a runtime environment for application 1232, such as a Java® runtime environment or a .NET Framework. The runtime environment is a consistent execution environment that enables application 1232 to run on any operating system that includes the runtime environment. Similarly, operating system 1230 may support containers, for example, lightweight, standalone, and executable packages of software including code, runtime, system tools, system libraries, and configuration for the application; application 1232 may be in the form of a container.

[0124] Furthermore, computer 1202 can be enabled using security modules such as a Trusted Processing Module ("TPM"). For example, with a TPM, a boot component hashs the next boot component in time before loading the next boot component and waits for the result to match a protected value. This process can occur at any layer in computer 1202's code execution stack, for example, at the application execution level or the operating system ("OS") kernel level, thereby enabling security at any level of code execution.

[0125] The user may input commands and information to the computer 1202 through one or more wired / wireless input devices, such as a keyboard 1238, a touchscreen 1240, and a pointing device such as a mouse 1242. Other input devices (not shown) may include microphones, infrared ("IR") remote controls, radio frequency ("RF") remote controls, or other remote controls, joysticks, virtual reality controllers or virtual reality headsets or combinations thereof, gamepads, stylus pens, cameras, gesture sensor input devices, vision movement sensor input devices, image input devices such as emotion detection devices or face detection devices, or biometric input devices such as fingerprint scanners or iris scanners. These and other input devices are often connected to the processing unit 1204 through an input device interface 1244 which can be coupled to the system bus 1208, but may also be connected through other interfaces such as a parallel port, an IEEE 1394 serial port, a game port, a USB port, an IR interface, or a BLUETOOTH® interface.

[0126] The monitor 1246 or other types of display devices may also be connected to the system bus 1208 via an interface such as the video adapter 1248. In addition to the monitor 1246, the computer typically includes other peripheral output devices (not shown), such as speakers, printers, and / or combinations thereof.

[0127] Computer 1202 may operate in a networked environment using logical connections via wired and / or wireless communication to one or more remote computers, such as remote computers 1250. Remote computers 1250 may be workstations, server computers, routers, personal computers, portable computers, microprocessor-based entertainment equipment, peer devices, or other common network nodes, typically including many or all of the elements described with respect to computer 1202, although typically only a memory / storage device 1252 is shown. The shown logical connections include wired / wireless connections to a local area network ("LAN") 1254, or a larger network, such as a wide area network ("WAN") 1256, or a combination thereof. Such LAN and WAN networking environments are common in offices and enterprises, facilitating enterprise-scale computer networks such as intranets, all of which may connect to global communication networks, such as the Internet.

[0128] When used in a LAN networking environment, computer 1202 may be connected to local network 1254 via a wired and / or wireless communication network interface or adapter 1258. Adapter 1258 may also include a wireless access point ("AP") installed therein to communicate with adapter 1258 in wireless mode, and can facilitate wired or wireless communication to LAN 1254.

[0129] When used in a WAN networking environment, computer 1202 may include a modem 1260 or may connect to a communication server on WAN 1256 via other means for establishing communication on WAN 1256, such as the Internet. The modem 1260, which may be internal or external and may be a wired or wireless device, may connect to the system bus 1208 via an input device interface 1244. In a networked environment, program modules shown with respect to computer 1202 or a part thereof may be stored in a remote memory / storage device 1252. The shown network connections are illustrative, and it will be understood that other means for establishing communication links between computers may be used.

[0130] When used in either a LAN networking environment or a WAN networking environment, computer 1202 may access a cloud storage system or other network-based storage system in addition to, or instead of, the external storage device 1216, as described above. Generally, the connection between computer 1202 and the cloud storage system may be established over LAN 1254 or WAN 1256, for example, by adapter 1258 or modem 1260, respectively. When computer 1202 is connected to the associated cloud storage system, the external storage interface 1226 may manage the storage provided by the cloud storage system, with the assistance of adapter 1258 and / or modem 1260, as to manage other types of external storage. For example, the external storage interface 1226 may be configured to provide access to the cloud storage sources as if those sources were physically connected to computer 1202.

[0131] Computer 1202 may be capable of communicating with any wireless device or entity configured to operate in wireless communication, such as printers, scanners, desktop and / or portable computers, personal digital assistants, communication satellites, any equipment or location associated with wirelessly discoverable tags (e.g., kiosks, newsstands, store shelves, etc.), and telephones. This may include Wireless Fidelity ("Wi-Fi" registered trademark) and Bluetooth® wireless technologies. Thus, the communication may have a predefined structure, similar to conventional networks, or it may simply be an ad-hoc communication between at least two devices.

[0132] The above descriptions include only examples of systems, computer program products, and computer implementation methods. Of course, for the purpose of illustrating this disclosure, it is not possible to describe every conceivable combination of components, products, or computer implementation methods, or combinations thereof, but those skilled in the art will recognize that many more combinations and substitutions of the disclosure are possible. Furthermore, to the extent that terms such as “includes,” “has,” and “possesse” are used in the detailed description, claims, appendices, and drawings, such terms are intended to be comprehensive in the same way as the term “comprising” is interpreted when “comprising” is used as a transitional term in the claims. The descriptions of various embodiments have been presented for illustrative purposes, but are not intended to be comprehensive or to limit oneself to the disclosed embodiments. Many modifications and variations will become apparent to those skilled in the art without departing from the scope of the embodiments described. The terminology used herein has been selected to best describe the principles of the embodiments, their practical applications, or the technical improvements to technologies available on the market, or to enable other persons skilled in the art to understand the embodiments disclosed herein. (Other possible items) [Item 1] Memory containing computer executable components; and A processor operably coupled to the memory and capable of executing the computer executable components stored in the memory. The computer executable component includes, A system having a model introspection component that analyzes the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the artificial intelligence model with respect to multiple test data subsets having different code complexity distributions. [Item 2] An extraction component that extracts code complexity metrics from multiple code samples included in the test dataset; and A test data subset component generates the multiple test data subsets by grouping the multiple code samples based on a performance metric that evaluates the output of the artificial intelligence model. The system described in item 1, further comprising the features described above. [Item 3] The system described in item 2, wherein the aforementioned multiple code samples are source code samples. [Item 4] A distribution component that determines the distribution of the multiple code samples within the multiple test data subsets based on the code complexity metric. The system described in item 2, which further includes the features described therein. [Item 5] A comparison component that compares a first distribution of code samples associated with a first test data subset from the plurality of test data subsets with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets. The system described in item 4, which further includes the features described therein. [Item 6] Memory containing computer executable components; and A processor operably coupled to the memory and capable of executing the computer executable components stored in the memory. The computer executable component is equipped with A system having a model introspection component that infers the learning behavior of an artificial intelligence model for a code comprehension task based on the performance of the artificial intelligence model on multiple test data subsets having different code complexity distributions. [Item 7] An extraction component that extracts code complexity metrics from multiple code samples included in the test dataset; and A test data subset component generates multiple test data subsets by grouping the multiple code samples based on performance metrics that evaluate the performance of the artificial intelligence model. The system described in item 6, which further includes the features described therein. [Item 8] The system described in item 7, wherein the aforementioned multiple code samples are source code samples. [Item 9] A distribution component that determines the distribution of the multiple code samples within the multiple test data subsets based on the code complexity metric. The system described in item 7, which further includes the features described therein. [Item 10] A comparison component that compares a first distribution of code samples associated with a first test data subset from the plurality of test data subsets with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets. The system described in item 9, which further includes the features described therein. [Item 11] The process involves analyzing the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the AI ​​model across multiple test data subsets with different code complexity distributions, using a system operablely coupled to a processor. A computer implementation method comprising the following features. [Item 12] The steps include: extracting a code complexity metric from multiple code samples included in the test dataset using the system; and The system generates a subset of test data by grouping the subset of code samples based on a performance metric that evaluates the output of the artificial intelligence model. A computer implementation method as described in item 11, further comprising the features described therein. [Item 13] The computer implementation method described in item 12, wherein the aforementioned multiple code samples are source code samples. [Item 14] The system determines the distribution of the multiple code samples within the multiple test data subsets based on the code complexity metric. A computer implementation method as described in item 12, further comprising the features described therein. [Item 15] The system compares a first distribution of code samples associated with a first test data subset from the plurality of test data subsets with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets. A computer implementation method as described in item 14, further comprising the features described therein. [Item 16] The system, operablely coupled to the processor, infers the learning behavior of an artificial intelligence model for a code comprehension task based on the performance of the artificial intelligence model on multiple test data subsets having different code complexity distributions. A computer implementation method comprising the following features. [Item 17] The steps include: extracting a code complexity metric from multiple code samples included in the test dataset using the system; and The system generates a plurality of test data subsets by grouping the plurality of code samples based on a performance metric that evaluates the performance of the artificial intelligence model. A computer implementation method as described in item 16, further comprising the features described therein. [Item 18] The computer implementation method described in item 17, wherein the aforementioned multiple code samples are source code samples. [Item 19] The system determines the distribution of the multiple code samples within the multiple test data subsets based on the code complexity metric. A computer implementation method as described in item 17, further comprising the features described therein. [Item 20] The system compares a first distribution of code samples associated with a first test data subset from the plurality of test data subsets with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets. A computer implementation method as described in item 19, further comprising the features described therein. [Item 21] A computer program product for code complexity-driven model introspection, the computer program product comprising a computer-readable storage medium having program instructions embodied thereby, the program instructions being transmitted to a processor The processor analyzes the learning behavior of an artificial intelligence model for a code comprehension task by comparing the output of the artificial intelligence model with respect to multiple test data subsets having different code complexity distributions. A computer program product that is executable by the aforementioned processor. [Item 22] The aforementioned program instruction further instructs the processor, For multiple code samples included in the test dataset, the code complexity metric is extracted by the processor; The processor generates the multiple test data subsets by grouping the multiple code samples based on a performance metric that evaluates the output of the artificial intelligence model. Computer program products as described in item 21. [Item 23] The computer program product described in item 22, wherein the aforementioned multiple code samples are source code samples. [Item 24] The aforementioned program instruction further instructs the processor, The computer program product described in item 23, wherein the processor determines the distribution of the multiple code samples within the multiple test data subsets based on the code complexity metric. [Item 25] The aforementioned program instruction further instructs the processor, The computer program product described in item 24, wherein the processor causes a first distribution of code samples associated with a first test data subset from the plurality of test data subsets to compare with a second distribution of code samples associated with a second test data subset from the plurality of test data subsets.

Claims

1. Memory containing computer executable components; and A processor operably coupled to the memory and capable of executing the computer executable components stored in the memory. The computer executable component includes, A model introspection component that analyzes the learning behavior of an artificial intelligence model for code comprehension tasks, An extraction component that extracts one or more code complexity metrics from multiple source code samples included in the test dataset, A test data subset component that generates a first test data subset and a second test data subset by grouping the plurality of source code samples based on a performance metric for evaluating the predictive accuracy of the artificial intelligence model, wherein the first test data subset is associated with true positive results of the artificial intelligence model, and the second test data subset is associated with false negative results of the artificial intelligence model. A distribution component that determines a first distribution of the plurality of source code samples in the first test data subset and a second distribution of the plurality of source code samples in the second test data subset based on the one or more code complexity metrics, A comparison component that compares a first range of values ​​of the one or more code complexity metrics associated with the first distribution with a second range of values ​​of the one or more code complexity metrics associated with the second distribution, It has, The model introspection component is a system that generates insights, based on the results of the comparison, showing aspects of the multiple source code samples that the artificial intelligence model has learned well from, and aspects of the multiple source code samples that could be targeted for improvement learning.

2. The system according to claim 1, wherein the performance metric includes the predictive accuracy of the artificial intelligence model, and the test data subset component is configured to group the plurality of source code samples into a test data subset based on whether the corresponding prediction of the artificial intelligence model is accurate or inaccurate.

3. The test data subset component groups the plurality of source code samples based on a performance metric that includes one or more of the prediction accuracy, precision, or recall derived from the predictions of the artificial intelligence model, The extraction component extracts from the plurality of source code samples a plurality of code complexity metrics, including at least one of cyclomatic complexity, Halsted complexity, maintainability index, or control flow complexity. The system according to claim 2.

4. Memory containing computer executable components; and A processor operably coupled to the memory and capable of executing the computer executable components stored in the memory. The computer executable component is equipped with Extracting one or more code complexity metrics from multiple source code samples included in the test dataset, The process involves programmatically generating multiple test data subsets by grouping the aforementioned multiple source code samples based on performance metrics that evaluate the predictive output of an artificial intelligence model, Based on the performance metric, the plurality of source code samples are divided into (i) a subset associated with accurate predictions and (ii) a subset associated with inaccurate predictions, Determining the distribution of the plurality of source code samples based on the subset associated with accurate predictions and the subset associated with inaccurate predictions, Comparing the value ranges of each of the one or more code complexity metrics in the distribution, Based on the results of the above comparison, the common code characteristics of the test data samples correctly predicted by the artificial intelligence model and the common code characteristics of the subset that the artificial intelligence model incorrectly predicted are identified. Based on the identified code characteristics, the artificial intelligence model generates insights that show aspects of the multiple source code samples that it has learned well and aspects of the multiple source code samples that can be targeted for improvement learning. A system having a model introspection component for inferring the learning behavior of the artificial intelligence model for a code comprehension task.

5. The system according to claim 4, wherein the performance metric includes the predictive accuracy of the artificial intelligence model, and the plurality of test data subsets include, based on the predictive output for each subset, (i) a subset associated with accurate predictions and (ii) a subset associated with inaccurate predictions.

6. A computer implementation method for analyzing the learning behavior of an artificial intelligence model for a code comprehension task, The processor performs the following steps: extracting one or more code complexity metrics from multiple source code samples included in the test dataset; The process involves the processor generating a first test data subset and a second test data subset by grouping the plurality of source code samples based on a performance metric for evaluating the predictive accuracy of the artificial intelligence model, wherein the first test data subset is associated with true positive results of the artificial intelligence model, and the second test data subset is associated with false negative results of the artificial intelligence model. The processor determines, based on the one or more code complexity metrics, a first distribution of the plurality of source code samples in the first test data subset and a second distribution of the plurality of source code samples in the second test data subset. The processor performs the steps of comparing a first range of values ​​for one or more code complexity metrics associated with the first distribution with a second range of values ​​for one or more code complexity metrics associated with the second distribution. The processor generates insights based on the comparison results, showing aspects of the multiple source code samples that the artificial intelligence model has learned well and aspects of the multiple source code samples that can be targeted for improvement learning. A computer implementation method comprising the above.

7. The computer implementation method according to claim 6, wherein the performance metric includes the predictive accuracy of the artificial intelligence model, and the step of generating the first test data subset and the second test data subset includes the step of grouping the plurality of source code samples into test data subsets based on whether the corresponding predictions of the artificial intelligence model are accurate or inaccurate.

8. The processor extracts one or more code complexity metrics from multiple source code samples included in a test dataset, The processor generates multiple test data subsets by grouping the multiple source code samples based on a performance metric for evaluating the predictive output of the artificial intelligence model. The processor divides the plurality of source code samples into (i) a subset associated with accurate predictions and (ii) a subset associated with inaccurate predictions based on the performance metric, The processor determines, based on the one or more code complexity metrics, the distribution of the subset of source code samples associated with accurate predictions and the subset associated with inaccurate predictions. A step of comparing the value ranges of each of the one or more code complexity metrics of the distribution, The processor, based on the results of the comparison, identifies common code characteristics of the test data samples correctly predicted by the artificial intelligence model and common code characteristics of the subsets incorrectly predicted by the artificial intelligence model. The processor generates insights based on each of the identified code characteristics, showing aspects of the plurality of source code samples that the artificial intelligence model has learned well and aspects of the plurality of source code samples that can be targeted for improvement learning. A computer implementation method comprising the above.

9. The computer implementation method according to claim 8, wherein the performance metric includes the predictive accuracy of the artificial intelligence model, and the step of generating a plurality of test data subsets includes the step of grouping the plurality of source code samples into (i) subsets associated with accurate predictions and (ii) subsets associated with inaccurate predictions, based on the predictive output for each subset.

10. A computer program for analyzing the learning behavior of an artificial intelligence model for a code comprehension task, the computer program comprising program instructions, the program instructions being on a processor Extract one or more code complexity metrics from multiple source code samples included in the test dataset. The processor generates a first test data subset and a second test data subset by grouping the plurality of source code samples based on a performance metric for evaluating the predictive accuracy of the artificial intelligence model, wherein the first test data subset is associated with true positive results of the artificial intelligence model, and the second test data subset is associated with false negative results of the artificial intelligence model. Based on the one or more code complexity metrics, a first distribution of the multiple source code samples in the first test data subset and a second distribution of the multiple source code samples in the second test data subset are determined. The first range of values ​​of the one or more code complexity metrics associated with the first distribution is compared with the second range of values ​​of the one or more code complexity metrics associated with the second distribution. Based on the results of the comparison, the artificial intelligence model generates insights that show aspects of the multiple source code samples that it has learned well from, and aspects of the multiple source code samples that can be targeted for further improvement learning. A computer program that is executable by the aforementioned processor.

11. The computer program according to claim 10, wherein the performance metric includes the predictive accuracy of the artificial intelligence model, and generating the first test data subset and the second test data subset includes grouping the plurality of source code samples into test data subsets based on whether the corresponding predictions of the artificial intelligence model are accurate or inaccurate.

12. A computer program for code complexity-driven model introspection, wherein the computer program comprises program instructions, and the program instructions are transmitted to a processor Extract one or more code complexity metrics from multiple source code samples included in the test dataset. Based on performance metrics that evaluate the predictive output of the artificial intelligence model, multiple test data subsets are generated to group the multiple source code samples. Based on the performance metrics, the multiple source code samples are divided into (i) a subset associated with accurate predictions and (ii) a subset associated with inaccurate predictions. Based on the one or more code complexity metrics, determine the distribution of the subset of source code samples associated with accurate predictions and the subset of source code samples associated with inaccurate predictions. Compare the value ranges of each of the one or more code complexity metrics in the distribution, Based on the results of the above comparison, the common code characteristics of the test data samples correctly predicted by the artificial intelligence model and the common code characteristics of the subset that the artificial intelligence model incorrectly predicted are identified. Based on the identified code characteristics, the artificial intelligence model generates insights that show aspects of the multiple source code samples that it has learned well from, and aspects of the multiple source code samples that could be targeted for improvement learning. A computer program that is executable by the aforementioned processor.

13. The performance metric includes the predictive accuracy of the artificial intelligence model, and generating the plurality of test data subsets includes grouping the plurality of source code samples into (i) subsets associated with accurate predictions and (ii) subsets associated with inaccurate predictions, based on the predictive output of each subset. The computer program according to claim 12.

Images