Evaluating code generated using artificial intelligence using complexity metrics

Abstract

Evaluating code generated using artificial intelligence using a complexity metric, including generating, by an artificial intelligence (AI) language model, output source code based on the input source code; identifying respective complexity scores for the input source code and the output source code using one or more complexity metrics; and generating a verification score for the output source code based on the evaluation of the respective complexity score.

Description

Technical Field

[0001] The present disclosure relates to methods, apparatuses, and products for evaluating code generated using artificial intelligence using complexity metrics. Background Art

[0002] Migrating the functionality of legacy source code to a more modern programming language can improve the maintainability and readability of the source code and enhance system performance. However, such migration is a daunting task that may involve writing, testing, validating, and debugging a large amount of code. Summary of the Invention

[0003] According to embodiments of the present disclosure, various methods, apparatuses, and products for evaluating code generated using artificial intelligence using complexity metrics are described herein. In some aspects, an artificial intelligence (AI) language model is used to remap application source code from an original codebase to a target codebase while maintaining the same functionality. In some aspects, complexity metrics are used to verify the transformation of the original application source code to the AI-generated source code. Using the assumption that the complexities of the input source code and the output source code are somewhat similar, the corresponding complexity metric scores of the input source code and the output source code indicate the transformation accuracy of the AI-generated code. In some aspects, when the complexity metric scores deviate, the AI language model is prompted to regenerate the code. Thus, when using AI-generated code to migrate from an original codebase to a new codebase, such as from a first programming language to a second programming language, or from a legacy system to a modern system, the comparison of the complexity scores aids in code verification.

[0004] In a particular embodiment, a method for evaluating code generated using artificial intelligence using complexity metrics includes generating output source code based on input source code by an artificial intelligence (AI) language model. The method further includes identifying corresponding complexity scores of the input source code and the output source code using one or more complexity metrics. The method further includes generating a verification score for the output source code based on an evaluation of the corresponding complexity scores. Thus, the comparison of the complexities of the input source code and the output source code can be used to evaluate the accuracy of automatic code generation to determine whether the control flow and structure of the original source code are maintained. For example, the input source code may be implemented in a first programming language, and the output source code may be implemented in a second programming language different from the first programming language. The one or more complexity metrics may include one or more of a cyclomatic complexity metric, one or more Halstead metrics, a live variable metric, a node metric, and a complexity index based on multiple complexity metrics.

[0005] In some variations, identifying corresponding scores for the input source code and the output source code using one or more complexity metrics includes calculating a first complexity score for the input source code using multiple complexity metrics such that the first complexity score represents a combination of the multiple complexity metrics. This variation also includes calculating a second complexity score for the output source code using multiple complexity metrics such that the second complexity score represents a combination of the multiple complexity metrics. In this way, the multiple complexity metrics can be represented by a single score for comparison.

[0006] In some variations, generating a validation score for the output source code based on an evaluation of the corresponding complexity scores includes adjusting the weights of the complexity scores of at least one of the input source code and the output source code based on its programming language. In this way, the inherent differences in complexity of different programming languages are compensated for.

[0007] In some variations, the method further includes regenerating the output source code from the input source code by an AI language model based on the validation score. In this way, the AI language module can iteratively regenerate the output source code until an acceptable validation score is reached.

[0008] In some variations, the method further includes indicating that the validation score exceeds an acceptable tolerance. In this way, software engineers can be alerted when the automatic code generation fails to accurately reproduce the input source code.

[0009] In some variations, the method further includes generating a second validation score for the regenerated output source code after retraining the AI language model. This variation further includes quantifying the improvement of the AI language model based at least on the validation score and the second validation score. In this way, the accuracy and reliability of the AI language model can be evaluated, and the results of retraining the AI language model can be measured.

[0010] In some aspects, an apparatus may include a processing device; and a memory operably coupled to the processing device, where the memory stores computer program instructions that, when executed, configure the processing device to perform the operations described above. In some aspects, a computer program product including a computer-readable storage medium may store computer program instructions that, when executed, may configure a computer to perform the operations described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A block diagram is listed showing an example computing environment for evaluating code generated using artificial intelligence using complexity metrics according to some embodiments of the present disclosure.

[0012] Figure 2 A flowchart is listed showing an example method for evaluating code generated using artificial intelligence using complexity metrics according to some embodiments of the present disclosure.

[0013] Figure 3 A flowchart is presented that depicts another example method for evaluating code generated using artificial intelligence using complexity metrics, according to some embodiments of the present disclosure.

[0014] Figure 4 A flowchart is presented that depicts another example method for evaluating code generated using artificial intelligence using complexity metrics, according to some embodiments of the present disclosure.

[0015] Figure 5 A flowchart is presented that depicts another example method for evaluating code generated using artificial intelligence using complexity metrics, according to some embodiments of the present disclosure.

[0016] Figure 6 A flowchart is presented that depicts another example method for evaluating code generated using artificial intelligence using complexity metrics, according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] In the field of software development, the need to modernize a codebase from one programming language to another is becoming increasingly common. For example, the source code of an application can be migrated from a legacy programming language (such as COBOL) to a modern programming language (such as Java). The motivation for such a migration may be to facilitate the maintenance and readability of the source code, improve security and error handling, enhance software and / or hardware performance, and other advantages that will be recognized by those skilled in the art.

[0018] According to the present disclosure, artificial intelligence (AI) is used to port or migrate the source code of an application to a different programming language. A large language model (LLM) is trained on a dataset that includes a large amount of source code to develop a generative AI that can output source code based on an input or prompt. That is, an AI language model is used to generate new source code based on an input of the original source code. For example, an AI language model can be prompted, such as "generate Java code that achieves the same goal as the following COBOL code", where the legacy COBOL source code is provided as an input. In response, the AI language model can output (at least ideally) AI-generated Java source code that performs the same function as the legacy code and produces the same output.

[0019] However, migrating a codebase to a new language introduces significant challenges in ensuring the accuracy and functionality of the converted code, especially when using AI for automated conversion. The difficulty lies in validating the code conversions generated by AI and determining whether the converted code preserves the intended logic, functionality, and structure of the original code. The inherent complexity of programming languages, combined with the way developers express their logic, poses challenges to reliably validating the correctness and similarity of AI-generated conversions. Additionally, validating the output source code converted from the input source code requires analyzing hundreds of thousands or even millions of lines of code.

[0020] The present disclosure addresses challenges related to validating the accuracy of AI-generated code conversions, with a specific focus on improving the reliability and maintainability of automated code conversions by comparing complexity scores. For example, cyclomatic complexity is a quantitative measure of program complexity and serves as an important metric for evaluating the complexity of control flow structures. By comparing the complexity scores of the input source code and the output source code, where the complexity scores are expected to be very similar, it can be ensured that the converted code not only replicates the logical flow of the original code but also maintains a similar structural complexity. A threshold can be set (e.g., the scores must be within a difference of 5) to verify whether they are indeed similar, and if the threshold is not met, the AI language model can regenerate the code until the threshold is satisfied.

[0021] Now referring to Figure 1 , an example of a computing environment in accordance with aspects of the present disclosure is shown. Computing environment 100 includes an example of an environment for executing at least some of the computer code involved in performing the various methods described herein, such as code analysis module 107. In addition to code analysis module 107, computing environment 100 also includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a set of processors 110 (including processing circuitry 120 and cache 121), communication fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and code analysis module 107, as identified above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage device 124, and a set of Internet of Things (IoT) sensors 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0022] The computer 101 can take the form of a desktop computer, a laptop computer, a tablet computer, a smartphone, a smartwatch or other wearable computer, a mainframe computer, a quantum computer, or any other form of computer or mobile device known now or developed in the future, which is capable of running programs, accessing networks, or querying databases, such as the remote database 130. As is well known in the field of computer technology and depending on the technology, the execution of computer-implemented methods can be distributed among multiple computers and / or multiple locations. On the other hand, in this introduction to the computing environment 100, the detailed discussion will focus on a single computer, specifically the computer 101, to make the introduction as simple as possible. The computer 101 can be located in the cloud, although Figure 1 it is not shown as being in the cloud. On the other hand, the computer 101 is not required to be in the cloud, except within any scope explicitly indicated.

[0023] The processor set 110 includes one or more computer processors of any type known now or developed in the future. The processing circuit 120 can be distributed across multiple packages, such as multiple coordinated integrated circuit chips. The processing circuit 120 can implement multiple processor threads and / or multiple processor cores. The cache 121 is a memory located within the processor chip package and is typically used for data or code that should be quickly accessible to the threads or cores running on the processor set 110. Cache memory is typically divided into multiple levels depending on its relative proximity to the processing circuit. Alternatively, some or all of the cache for the processor group can be located "off-chip". In some computing environments, the processor group 110 can be designed to process qubits and perform quantum computing.

[0024] Computer-readable program instructions are typically loaded onto the computer 101 to cause a series of operational steps to be executed by the processor set 110 of the computer 101, and thereby implement a computer-implemented method such that the instructions thus executed will instantiate the methods specified in the flowcharts and / or narrative descriptions of the computer-implemented methods contained in this document. These computer-readable program instructions are stored in various types of computer-readable storage media, such as the cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the processor set 110 to control and direct the execution of the computer-implemented method. In the computing environment 100, at least some of the instructions for executing the computer-implemented method can be stored in the code analysis module 107 in the persistent storage device 113.

[0025] The communication structure 111 is a signal conduction path that allows the various components of the computer 101 to communicate with each other. Generally, this structure consists of switches and conductive paths, such as those that make up a bus, a bridge, a physical input / output port, etc. Other types of signal communication paths, such as fiber optic communication paths and / or wireless communication paths, may also be used.

[0026] The volatile memory 112 is any type of volatile memory known now or developed in the future. For example, dynamic random access memory (RAM) or static RAM. Generally, the volatile memory 112 is characterized by random access, but this is not necessary unless explicitly stated. In the computer 101, the volatile memory 112 is located in a single package and inside the computer 101, but alternatively or in addition, the volatile memory may be distributed in multiple packages and / or located outside the computer 101.

[0027] The persistent storage device 113 is any form of non-volatile storage device for a computer known now or developed in the future. The non-volatility of such a storage device means that the stored data remains unchanged whether or not power is supplied to the computer 101 and / or directly to the persistent storage device 113. The persistent storage device 113 may be read-only memory (ROM), but generally at least a portion of the persistent device allows data to be written, deleted, and rewritten. Some familiar forms of persistent storage devices include magnetic disks and solid state storage devices. The operating system 122 may take various forms, such as various known proprietary operating systems or open source portable operating system interface type operating systems that employ a kernel. The code included in the code analysis module 107 generally includes at least some of the computer code involved in performing the computer-implemented methods described herein.

[0028] The peripheral device set 114 includes the peripheral device set of the computer 101. Data communication connections between the peripheral devices and other components of the computer 101 can be achieved in various ways, such as Bluetooth connections, near-field communication (NFC) connections, connections established through cables (such as universal serial bus (USB)-type cables), plug-in connections (e.g., secure digital (SD) cards), connections established through local communication networks, and even connections established through wide area networks (such as the Internet). In various embodiments, the UI device group 123 may include components such as display screens, speakers, microphones, wearable devices (such as goggles and smartwatches), keyboards, mice, printers, touchpads, game controllers, and tactile devices. The storage device 124 is an external storage device, such as an external hard drive, or a pluggable storage device, such as an SD card. The storage device 124 can be persistent and / or volatile. In some embodiments, the storage device 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where the computer 101 needs to have a large amount of storage space (e.g., the computer 101 locally stores and manages a large database), such a storage device may be provided by a peripheral storage device designed to store an extremely large amount of data, such as a storage area network (SAN) shared by multiple computers distributed in different geographical locations. The IoT sensor set 125 consists of sensors that can be used for Internet of Things applications. For example, one sensor can be a thermometer, and another sensor can be a motion detector.

[0029] The network module 115 is a collection of computer software, hardware, and firmware that allows the computer 101 to communicate with other computers through the WAN 102. The network module 115 may include hardware such as a modem or a Wi-Fi signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or network browser software for communicating data over the Internet. In some embodiments, the network control function and the network forwarding function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments utilizing software-defined network (SDN)), the control function and the forwarding function of the network module 115 are executed on physically independent devices so that the control function manages multiple different network hardware devices. Computer-readable program instructions for performing computer-implemented methods can generally be downloaded to the computer 101 from an external computer or an external storage device through a network adapter card or a network interface included in the network module 115.

[0030] The WAN 102 is any wide area network (e.g., the Internet) that can communicate computer data over non-local distances via any technology for computer data communication known now or developed in the future. In some embodiments, the WAN 102 can be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located within the local area (such as a Wi-Fi network). The WAN and / or LAN typically includes computer hardware such as copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and edge servers.

[0031] The end user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of the enterprise operating the computer 101) and can take any form discussed above in relation to the computer 101. The EUD 103 typically receives helpful and useful data from the operation of the computer 101. For example, in the hypothetical case where the computer 101 is designed to provide advice to the end user, the advice typically travels from the network module 115 of the computer 101 through the WAN 102 to the EUD 103. In this way, the EUD 103 can display or otherwise present the advice to the end user. In some embodiments, the EUD 103 can be a client device such as a thin client, thick client, mainframe, desktop computer, etc.

[0032] The remote server 104 is any computer system that provides at least some data and / or functionality to the computer 101. The remote server 104 can be controlled and used by the same entity operating the computer 101. The remote server 104 represents a machine that collects and stores helpful and useful data for use by other computers (such as the computer 101). For example, in the hypothetical case where the computer 101 is designed and programmed to provide recommendations based on historical data, then the historical data can be provided to the computer 101 from the remote database 130 of the remote server 104.

[0033] A public cloud 105 is any computer system that is available for use by multiple entities, providing on-demand availability of computer system resources and / or other computer capabilities, particularly data storage (cloud storage) and computing power, without the user directly performing active management. Cloud computing typically utilizes resource sharing to achieve consistency and economies of scale. The direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by a virtual computing environment that runs on various computers that make up the set of host physical machines 142, which is the complete set of physical computers within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the set of virtual machines 143 and / or containers from the set of containers 144. It can be understood that these VCEs can be stored as images and can be transferred among and between various physical machine hosts as images or after instantiation of the VCE. The cloud orchestration module 141 manages the transfer and storage of the images, deploys new instances of the VCE, and manages the active instances of the VCE deployment. The gateway 140 is a collection of computer software, hardware, and firmware that allows the public cloud 105 to communicate via the WAN 102.

[0034] Some further explanations of the virtualized computing environment (VCE) will now be provided. The VCE can be stored as an "image". New active instances of the VCE can be instantiated from the image. Two familiar types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to an operating system feature in which the kernel allows for the existence of multiple isolated user-space instances (called containers). From the perspective of the programs running within them, these isolated user-space instances typically appear as real computers. A computer program running on a normal operating system can use all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running within a container can only use the contents of the container and the devices allocated to the container, a feature known as containerization.

[0035] The private cloud 106 is similar to the public cloud 105, except that the computing resources are only available for a single enterprise. Although the private cloud 106 is described as communicating with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and can only be accessed through a local / private network. A hybrid cloud is a combination of multiple clouds of different types (e.g., private cloud, community cloud, or public cloud type), which are typically implemented by different vendors. Each of the multiple clouds remains an independent and discrete entity, but the larger hybrid cloud architecture is combined through standardized or proprietary technologies that enable coordination, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, the public cloud 105 and the private cloud 106 are both part of a larger hybrid cloud.

[0036] For further explanation, Figure 2 FIG. shows a flowchart of an example method for evaluating code generated using artificial intelligence using complexity metrics according to some embodiments of the present disclosure. For example, Figure 2 the method may be executed by a code analysis module 201, such as Figure 1 the code analysis module 107. In some examples, the code analysis module 201 may be implemented as a process or service that includes an AI language model that generates input source code in the form of output source code. In other examples, the code analysis module 201 may be implemented as part of a process or service that is separate from the process or service that includes the AI language model. In further examples, the code analysis module 201 may be implemented as part of a process or service that monitors the quality of the AI language model to evaluate whether retraining of the AI language model is appropriate or successful.

[0037] Figure 2The method includes generating an output source code 205 based on an input source code 203 by an artificial intelligence (AI) language model 211. The AI language model 211 can be trained on a massive dataset of original source codes in a first programming language, which have been remapped to source codes in a different programming language. Thus, the AI language model 211 is configured to automatically convert an input source code block in one programming language into an output source code block in a different programming language. In some examples, the output source code 205 is generated by prompting the AI language model to generate the output source code based on the input source code 203. For example, the AI language model can be prompted with "generate Java source code from block A of COBOL source code", where block A is provided as the input source code. In response, the AI language model generates Java source code that is intended to provide the same interface, perform the same functions, and generate the same output as the original COBOL source code. In some examples, the input source code and the output source code reflect the migration of the source code of an application from a first programming language (e.g., a legacy codebase) to a second programming language (e.g., a modern codebase). For example, the input source code may include legacy source code written in an old programming language (e.g., COBOL), while the output source code can be implemented in a modern programming language (e.g., Java); however, both the input source code and the output source code are intended to achieve the same goal, provide the same interface, and produce the same output.

[0038] Figure 2 The method includes identifying 204 corresponding complexity scores of the input source code 203 and the output source code 205 using one or more complexity metrics. In some embodiments, the code analysis module 201 identifies 204 the corresponding complexity scores by calculating the corresponding complexity scores of the input source code 203 and the output source code 205, which will be described in detail below. In other embodiments, instead of calculating the complexity scores of the input source code and the output source code, the code analysis module 201 identifies 204 the corresponding complexity scores by receiving the complexity scores of the input source code 203 and the output source code 205 calculated by a separate complexity analysis utility.

[0039] In some examples, the code analysis module 201 uses cyclomatic complexity as a complexity metric to identify the corresponding complexity scores of the input source code and the output source code. Cyclomatic complexity is a software metric used to measure the complexity of a program's control flow. It was developed by Thomas J. McCabe and is sometimes also referred to as the McCabe number or McCabe complexity. The cyclomatic complexity of a program is calculated based on the number of linearly independent paths through the program's source code. This metric is particularly useful for evaluating the maintainability and testability of a software system.

[0040] The cyclomatic complexity can be determined by constructing a control flow graph of a code module (e.g., a function or method), where each statement is a node, and where an edge connects the first node to the second node if control can pass from the first statement to the second statement. In some examples, the formula for cyclomatic complexity can be defined as V = E - N + 2P, where V is the cyclomatic complexity, E is the number of edges in the control flow graph of the program, N is the number of nodes in the control flow graph, and P is the number of connected components (for a single linear program, P = 1). For a single function or method, the cyclomatic complexity can be defined as V = E - N + 2.

[0041] Put simply, the cyclomatic complexity can be understood as the number of decision points or branches in a program. Thus, in some examples, the cyclomatic complexity can be defined as: V = D, where V is the cyclomatic complexity and D is the number of decision points in the code (e.g., the number of conditional statements or branch points). It is a measure of the structural complexity of a program and is typically related to the number of test cases required to achieve full test coverage.

[0042] The higher the cyclomatic complexity, the more complex the program structure, which may increase the difficulty of understanding, testing, and maintaining the code. Empirically, a lower cyclomatic complexity is desirable as it tends to indicate simpler and more manageable code. For a collection of modules (such as methods, classes, subroutines), the complexity of the individual functions they contain can be used to determine the overall, average, or maximum cyclomatic complexity. The cyclomatic complexity of each line of source code can be expressed as a decision density.

[0043] In some examples, the code analysis module 201 uses one or more Halstead metrics as complexity metrics to identify the corresponding complexity scores of the input source code and the output source code. The Halstead complexity metrics developed by Maurice H. Halstead are a set of metrics designed to quantify various aspects of a software program, with a focus on the quantity and difficulty of the code. These metrics are intended to provide a quantitative assessment of software complexity and help predict the software development effort.

[0044] To calculate the Halstead metrics, n1 is defined as the number of distinct operators, n2 is defined as the number of distinct operands, N1 is defined as the total number of operators, and N2 is defined as the total number of operands. The program vocabulary n can be expressed as n = n1 + n2. The program length N is expressed as N = N1 + N2. The calculated program length N' can be expressed as N' = n1 log2 n1 + n2 log2 n2. The program volume V is a measure representing the quantity or size of the program and can be calculated as V = N log2 n.

[0045] Since any program must have at least two operators: one for function calls and one for statement endings, the ratio (n1) / 2 can be considered as the relative difficulty level due to the larger number of operators in the program. The ratio (N2) / n2 represents the average number of operand uses. In programs with a higher variable change frequency, this ratio may be larger. Since such programs are more difficult to understand, the difficulty D of reading or writing a program can be calculated as D = (n1 * n2) / (2 * n2).

[0046] The effort E is a measure of the estimated time required for a human to write code, calculated as E = D x V, where D is the difficulty measure and V is the program volume discussed above. The time T to write the code can be calculated as T = E / 18 seconds. The number of delivery errors B can be estimated as B = V / 3000.

[0047] Halstead metrics provide insights into the program size, the diversity of operators and operands, and the difficulty of understanding the code. A large program volume may indicate a large and potentially complex program, while a high program difficulty indicates that the code may be challenging to understand.

[0048] In some examples, the code analysis module 201 uses raw metrics as complexity metrics to identify the corresponding complexity scores of the input source code and the output source code. Certain raw metrics can be used as indicators of complexity, including the lines of code (LOC) in the program, the logical lines of code (LLOC), the source lines of code (SLOC), the percentage of comment lines, and the percentage of blank lines.

[0049] In some examples, the code analysis module 201 uses live variable metrics as complexity metrics to identify the corresponding complexity scores of the input source code and the output source code. The live variable metric is a measure of program complexity based on the number of live variables associated with the statements in the program. It provides a quantitative assessment of the cognitive load and difficulty associated with understanding and maintaining the code. In the context of this metric, a live variable is a variable whose value is still relevant or needed at some point during program execution. The more live variables a program has, the greater the challenge of understanding and maintaining the program. Therefore, the live variable metric can be used as an indicator of program complexity.

[0050] Specifically, a live variable is a variable whose value is still in use or needed at a specific point in the program. A variable is considered "live" from its first reference to its last reference in a module, including all statements between these references. If a particular statement falls between the first and last occurrences of a variable in the program, that statement is considered to be associated with the live variable. Static code analysis can be used to calculate the live variable metric by counting the number of live variables associated with each statement. This metric is based on the number of live variables involved in each statement to understand the complexity of that statement.

[0051] By calculating the average number of live variables, this metric can be extended to the entire module. The average live variable metric is determined by adding up the count of live variables for all executable statements in the module and then dividing that sum by the total number of executable statements. A higher average live variable metric indicates a more complex module because it implies that, on average, more variables need to be tracked and understood for their values throughout the execution of the program. This metric provides a quantitative measure of the cognitive load borne by a programmer attempting to understand or maintain the code.

[0052] In some examples, the code analysis module 201 uses the knot metric as a complexity metric to identify the corresponding complexity scores for the input source code and the output source code. The knot metric represents the complexity and degree of unstructuredness of the module's control flow. The knot metric can be calculated by counting the number of intersection points of the control flow paths through the code module. To illustrate, an arrow can be drawn from a control transfer point to a destination. The more these arrows are intertwined, the more complex the program is.

[0053] In some examples, the code analysis module 201 uses the naturalness metric as a complexity metric to identify the corresponding complexity scores for the input source code and the output source code. The naturalness of a particular statement in the source code is represented by the number of times that statement appears in the corpus of training data provided to the AI language model. The statements of a portion of the code that appear less frequently in the training data may indicate that this portion of the code is more complex. The naturalness metric can be represented as the percentage of statements in a code block whose occurrence values are below a specific threshold.

[0054] In some examples, the code analysis module 201 uses the ultrametric topological metric as a complexity metric to identify the corresponding complexity scores for the input source code and the output source code. Ultrametric topology is related to the analysis of hierarchical functional relationships and can be used for landscape complexity modeling. The land units on a map are connected to a function that indicates the direction of movement or information exchange between a pair of land units. The land units and the function are part of an inclusive landscape unit. Here, by defining code modules (such as functions, methods, classes, subroutines) as "land units" or nodes that are interconnected by ultrametric functions indicating information exchange along control flow paths, the ultrametric topology can be applied to the code. The connections between nodes are edges such that the ultrametric distance between two nodes is the number of edges that must be traversed to get from one node to the other. The sum of the ultrametric distances between all nodes can be used as a score for code complexity. Additionally, the sum of the degrees of each node (the number of edges connected to the node) can also be used as a score for code complexity. Furthermore, the cyclomatic complexity of the code can be determined as the number of edges minus the number of nodes plus one. When constructing the ultrametric distance matrix, the eigenvectors of the matrix can be calculated and used to determine the "direction" or "influence" of a module, indicating how a change in one module may affect other modules.

[0055] In some examples, the code analysis module 201 uses one or more complexity metrics to identify 204 the respective complexity scores of the input source code 203 and the output source code 205 by using a first complexity metric to calculate a first complexity score for the input source code 203 and using the first complexity metric to calculate a second complexity score for the output source code. For example, the code analysis module 201 calculates 206 the first complexity score by applying one of the above-described complexity analysis techniques to the input source code 203 and calculates the second complexity score by applying the same complexity analysis technique to the output source code 205. In some embodiments, the calculation of the complexity score is performed by calculating a total complexity score or an average complexity score based on the individual complexity scores of each code block (e.g., functions, methods, classes, subroutines, etc.) in the source code.

[0056] It will be understood that this technique can be replicated using multiple complexity metrics such that multiple complexity scores are calculated for the input source code and multiple complexity scores are generated for the output source code. For example, the code analysis module can calculate a third complexity score for the input source code and a fourth complexity score for the output source code using a second complexity metric. Thus, the respective complexity scores for each of the input source code and the output source code can include one or more complexity scores based on one or more of the following metrics: cyclomatic complexity metric, one or more Halstead metrics, one or more primitive metrics (such as lines of source code), knot metrics, live variable metrics, hypermetric topological metrics, and naturalness metrics.

[0057] In some examples, the first complexity score and the second complexity score represent an aggregation of different complexity scores. Thus, in some examples, the code analysis module 201 uses one or more complexity metrics to identify 204 the respective complexity scores of the input source code 203 and the output source code 205 by using multiple complexity metrics to calculate 206 a first complexity score for the input source code, where the first complexity score represents a combination of multiple complexity metrics, and using multiple complexity metrics to calculate 208 a second complexity score for the output source code. For example, the complexity score can be a complexity index calculated based on a weighted average calculated using multiple complexity metrics. In a particular embodiment, the feature vector is constructed from multiple complexity metrics. The base value is calculated as the square root of the sum of the squares of each of these values. The respective base values calculated for the input source code and the output source code can be used as the respective complexity scores for comparison between the input source code and the AI-generated output source code.

[0058] Different programming languages (such as COBOL and Java) use different operator vocabularies and different grammars, and without considering the different operator vocabularies and different grammars, it may bias the complexity score. Therefore, in some examples, different complexity metric definitions are used for the input source code and the output source code. For example, when evaluating loop complexity based on the number of decision points, a conditional statement, a branching statement, or an operator that increases the number of decision points in one programming language should correspond to a statement with the same effect in another programming language. Similarly, statistical analysis may show that the number of source code lines in one programming language is expected to be a specific percentage more than the number of source code lines in another programming language. Therefore, the complexity calculation can be adjusted based on the differences between programming language grammars.

[0059] It will be understood that the code analysis module 201 can use any of the above single complexity metrics or a combination of complexity metrics to identify the corresponding complexity scores of the input source code and the output source code. Additionally, it will be understood that the code analysis module can use other complexity metrics and mathematical constructs not discussed above in a manner consistent with the present disclosure to quantify the complexity of the input source code and the output source code.

[0060] Figure 2 The method further includes generating 210 a verification score 209 for the output source code 205 based on an evaluation of the corresponding complexity scores. In some examples, the code analysis module 201 generates 210 the verification score 209 by comparing one or more complexity scores of the input source code with one or more complexity scores of the output source code and determining the verification score 209 representing their similarity or dissimilarity. For example, the verification score 209 can be the absolute or relative deviation of the complexity score of the output source code from the complexity score of the input source code. In some examples, the verification score 209 can be based on evaluating multiple complexity scores using multiple complexity metrics of the input source code and the output source code, such as the average or weighted average of the individual scores. In some examples, the code analysis module 201 can set a tolerance, such as a threshold or a range, to determine whether the output source code passes or fails the verification. For example, if the difference between the complexity scores is higher than a specific threshold, or the complexity score of the output source code is greater than the complexity score of the input source code, the code analysis module can determine that the output source code fails the verification. Therefore, in some embodiments, the verification score 209 can be a binary result, such as pass / fail.

[0061] For further explanation, Figure 3 The flowchart shows an example method of evaluating code generated using artificial intelligence using complexity metrics according to some embodiments of the present disclosure. Figure 3 The method extends Figure 2 the method in that Figure 3The method generates a verification score of the output source code 205 based on an evaluation of the corresponding complexity score, and further includes adjusting 302 the weight of its complexity score based on the programming language of at least one of the input source code 203 and the output source code 205. Some programming languages are inherently more complex than others. For example, a program written in assembly language is generally more complex than the same program written in Java. To account for this difference, in some examples, the code analysis module 201 weights at least one of the input source code and the output source code based on the expected complexity of the programming language in which the source code is written. For example, in a case where the input source code is part of a legacy codebase and the output source code is written in a more modern programming language, the code analysis module 201 can reduce the weight of the complexity score of the input source code to account for the expected reduction in complexity when converting to the modern programming language.

[0062] For further explanation, Figure 4 FIG. shows a flowchart of an example method for evaluating code generated using artificial intelligence using complexity metrics, according to some embodiments of the present disclosure. Figure 4 The method extends Figure 2 the method in that Figure 4 the method further includes regenerating 402 the output source code 205 from the input source code 203 by the AI language model 211 based on the verification score 209. In some examples, the code analysis module 201 determines that the verification score 209 of the output code exceeds an acceptable tolerance, or otherwise indicates that the output source code fails verification. Accordingly, the code analysis module 201 determines that the output source code should be regenerated. In some examples, the code analysis module 201 generates a second prompt in substantially the same manner as the first prompt was generated; but in this case, the prompt indicates to the AI language model that the AI language model should generate a different implementation. In such a case, the code analysis module 201 can generate a prompt such as "Regenerate the code for block A" or "Regenerate code for block A that is syntactically different from the previously generated code". In response, the AI language model regenerates alternative code for block A of the input code. In some embodiments, the code analysis module 201 iteratively re-prompts the AI language model to regenerate the source code until the output source code passes verification or reaches a threshold number of attempts.

[0063] In some embodiments, the code analysis module 201 adjusts one or more parameters of the AI language model in response to determining that a conversion score exceeds an acceptable tolerance. The AI language model can include configurable parameters that affect the creativity of the model's response to a prompt. For example, the temperature parameter adjustment can be used to select the probability distribution of the next token for the output stream. When selecting the next token for the output stream, a lower temperature causes the language model to select tokens with probabilities within a narrower range, thus tending to more deterministic outputs; while a higher temperature causes the language model to select tokens with probabilities within a wider range, thus tending to more random outputs. Another example parameter is the top-k parameter, which controls the randomness of selecting the next token by telling the language model that it must select from the top-k tokens with the highest probabilities. Yet another example parameter is the top-p parameter, which controls the randomness of selecting the next token by telling the language model that it must select from the highest probability tokens whose probability sum is equal to or exceeds the p value.

[0064] In some examples, the code analysis module 201 adjusts one or more parameters of the AI language model in response to determining that one or more iterations of generating the output source code have not met the tolerance threshold. For example, as the number of iterations increases, the parameters that control the creativity of the AI language model can be adjusted to increase the randomness of the output. In this way, the AI language model can be induced to generate solutions different from the failed solutions that occurred in previous iterations. In some examples, the adjustment of one or more parameters is performed by including a statement in the prompt, such as "set the temperature to 0.8". It will be understood that the parameters of the language model can be adjusted at any stage of the processing. For example, in some embodiments, before the AI language model generates new source code from the original source code, the preprocessing stage analyzes the original source code and sets the language model parameters based on the analysis results. For example, a statistical analysis of the original code can be employed to predict the degree of creativity or determinism of the language model in its output.

[0065] For further explanation, Figure 5 The flowchart listing an example method of evaluating code generated using artificial intelligence using a complexity metric according to some embodiments of the present disclosure is shown. Figure 5 The method of Figure 2 extends Figure 5 the method of

[0066] For further explanation,Figure 6 A flowchart listing an example method of evaluating AI-generated code using a complexity metric according to some embodiments of the present disclosure. Figure 6 The method of Figure 2 extends the method of Figure 6 in that the method of further includes generating, after retraining the AI language model 211, a second validation score 602 for the output source code to be regenerated. In some examples, the AI language model 211 is retrained on an additional training dataset to improve the quality of the AI code conversion of the input source code. To evaluate whether the AI language model has improved the quality and accuracy of the code conversion and to quantify the improvement, the AI model is prompted to regenerate the output source code based on the input source code for which the validation score was previously determined. In these examples, the code analysis module 201 generates the second validation score 602 in the above manner using the same complexity metric as used to generate the initial validation score.

[0067] Figure 6 The method of also includes quantifying 604 the improvement of the AI language model 211 based at least on the validation score and the second validation score. In some embodiments, the code analysis module 201 quantifies 604 the improvement of the AI language model 211 by comparing the initial validation score and the second validation score to determine whether the output source code generated by the AI language model 211 is more similar in complexity to the input source code.

[0068] Although the embodiments facilitate migrating or porting an application from one programming language to another programming language, and from a legacy programming language to a more modern programming language, it will be understood that in some examples, the original source code and the new source code may be written in the same programming language.

[0069] In view of the above, evaluating AI-generated code using a complexity metric according to the present disclosure provides many advantages. Embodiments of the present disclosure improve the accuracy and quality of automatic code generation and further improve the reliability and maintainability of the source code generated by automatic code generation. The complexity score evaluation facilitates quantifying the validation between the output source code and the input source code and further indicates whether the converted code not only replicates the logical flow of the original code but also maintains a similar level of structural complexity. The evaluation of the complexity score is beneficial for determining whether the AI-generated code needs to be regenerated, thereby reducing the manpower for validating the AI-generated code. In addition, the evaluation of the complexity score helps to quantify the improvement in the accuracy and ability of the AI language model to convert source code.

[0070] Aspects of the present disclosure are described by narrative text, flowcharts, computer system block diagrams, and / or machine logic block diagrams included in computer program product (CPP) embodiments. For any flowchart, depending on the technology involved, operations may be performed in a different order than shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or in a manner that at least partially overlaps in time.

[0071] The term "computer program product embodiment" ("CPP embodiment" or "CPP") as used in the present disclosure is used to describe any collection of one or more storage media (also referred to as "media") that are substantially included in a collection of one or more storage devices that collectively include machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" refers to any tangible device that can retain and store instructions for use by a computer processor. Without limitation, computer-readable storage media can be electronic storage media, magnetic storage media, optical storage media, electromagnetic storage media, semiconductor storage media, mechanical storage media, or any suitable combination of the foregoing. Some known types of storage devices that include these media include: floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices such as punch cards or pits / lands formed on the major surfaces of optical discs, or any suitable combination of the foregoing devices. As used in the present disclosure, the term computer-readable storage media should not be construed as storing in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, optical pulses propagating through an optical fiber cable, electrical signals communicated through a wire, and / or other transmission media. Those skilled in the art will understand that during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, data typically moves at certain incidental points in time, but this does not make the storage device a transitory device because the data is not transitory when stored.

[0072] The description of the various embodiments of the present disclosure is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, the practical application, or the technical improvement over technologies found in the marketplace, or to enable those of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for evaluating code generated using artificial intelligence using a complexity metric, comprising: Generate output source code based on input source code through an artificial intelligence (AI) language model; identifying respective complexity scores for the input source code and the output source code using one or more complexity metrics; as well as A verification score for the output source code is generated based on the evaluation of the corresponding complexity scores.

2. The method according to claim 1, wherein: The input source code is implemented in a first programming language, and the output source code is implemented in a second programming language different from the first programming language.

3. The method according to claim 1, wherein: The one or more complexity metrics include one or more of the following metrics: a cyclomatic complexity metric, one or more Halstead metrics, a live variable metric, a knot metric, a supermetric topological metric, and a complexity index based on multiple complexity metrics.

4. The method according to claim 1, wherein: Identifying respective scores of the input source code and the output source code using one or more complexity metrics includes: Using a plurality of complexity metrics, calculating a first complexity score for the input source code, wherein the first complexity score represents a combination of the plurality of complexity metrics; and Using the plurality of complexity metrics, a second complexity score for the output source code is calculated, wherein the second complexity score represents a combination of the plurality of complexity metrics.

5. The method according to claim 1, wherein: Generating a verification score for the output source code based on the evaluation of the corresponding complexity score includes: A weight of a complexity score of at least one of the input source code and the output source code is adjusted based on a programming language of at least one of the input source code and the output source code.

6. The method according to claim 1, further comprising: The output source code is regenerated from the input source code based on the verification score by the artificial intelligence language model.

7. The method according to claim 1, further comprising: Indicates that the verification score is outside an acceptable tolerance.

8. The method according to claim 1, further comprising: After retraining the artificial intelligence language model, generating a second verification score for the regenerated output source code; as well as Based at least on the validation score and the second validation score, quantify an improvement in the artificial intelligence language model.

9. An apparatus comprising: Memory; as well as a processing device operatively coupled to the memory, the processing device being configured to: Generate output source code based on input source code through an artificial intelligence (AI) language model; identifying respective complexity scores for the input source code and the output source code using one or more complexity metrics; as well as A verification score for the output source code is generated based on the evaluation of the corresponding complexity scores.

10. The device according to claim 9, wherein: The input source code is implemented in a first programming language, and the output source code is implemented in a second programming language different from the first programming language.

11. The device according to claim 9, wherein: The one or more complexity metrics include one or more of the following metrics: a cyclomatic complexity metric, one or more Halstead metrics, a live variable metric, a knot metric, a supermetric topological metric, and a complexity index based on multiple complexity metrics.

12. The device according to claim 9, wherein: To identify respective scores of the input source code and the output source code using one or more complexity metrics, the processing device is further configured to: Using a plurality of complexity metrics, calculating a first complexity score for the input source code, wherein the first complexity score represents a combination of the plurality of complexity metrics; and Using the plurality of complexity metrics, a second complexity score for the output source code is calculated, wherein the second complexity score represents a combination of the plurality of complexity metrics.

13. The device according to claim 9, wherein: To generate a verification score for the output source code based on the evaluation of the corresponding complexity score, the processing device is further configured to: A weight of a complexity score of at least one of the input source code and the output source code is adjusted based on a programming language of at least one of the input source code and the output source code.

14. The device according to claim 9, wherein: The processing device is further configured to: The output source code is regenerated from the input source code based on the verification score by the artificial intelligence language model.

15. The device according to claim 9, wherein: The processing device is further configured to: generating a second verification score for the regenerated output source code after retraining the artificial intelligence language model; and Based at least on the validation score and the second validation score, quantify an improvement in the artificial intelligence language model.

16. A computer program product comprising one or more computer-readable storage media and program instructions stored together on the one or more computer-readable storage media, wherein the program instructions, when executed, cause a processing device to: Using one or more complexity metrics, identifying respective complexity scores for the input source code and the output source code, wherein The output source code is generated based on the input source code by an artificial intelligence (AI) language model; as well as A verification score for the output source code is generated based on the evaluation of the corresponding complexity scores.

17. The computer program product of claim 16, wherein: The output source code is generated by the artificial intelligence language model in response to a prompt wherein the artificial intelligence language model uses the input source code as part of the prompt to generate the output source code.

18. The computer program product of claim 16, wherein: The input source code is implemented in a first programming language, and the output source code is implemented in a second programming language different from the first programming language.

19. The computer program product of claim 16, wherein: The instructions further cause the processing device to: Based on the verification score, the artificial intelligence language model is prompted to regenerate the output source code from the input source code.

20. The computer program product of claim 16, wherein: The instructions further cause the processing device to: generating a second verification score for the regenerated output source code after retraining the artificial intelligence language model; and An improvement of the artificial intelligence language model is quantified based on at least the validation score and the second validation score.

21. A system comprising modules respectively used to perform the steps of the method according to any one of claims 1 to 6.

Images