Method, device, and computer program (large language model code translation error detection)

Abstract

To provide a large language model code translation error detection method comprising receiving a code portion of a first programming language, and converting the code portion into a second programming language.SOLUTION: A first accuracy of conversion of a code portion into a second programming language is calculated. A difference between the first accuracy and historical accuracy of conversion from a first programming language into the second programming language is determined. A potential error in the code portion of the first programming language is indicated based on the difference between the first accuracy and the historical accuracy being greater than a predetermined value is indicated.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present disclosure relates to methods, apparatuses, and products for detecting large language model code translation errors.

Background Art

[0002] By migrating the functionality of legacy source code to a more modern programming language, the maintainability and readability of the source code can be improved, and the system performance can be enhanced. However, such migration can be a laborious task that may involve writing, testing, validating, and debugging large amounts of code. If the modernized source code is deployed with errors in a complex software system, there is a possibility that the expected operations cannot be completed, that an exhaustive retry logic may monopolize resources, that transactions may start to fail, and that it may lead to a halt that can occur when all components of the stack stop functioning. This can be devastating, especially in mission-critical production systems.

Summary of the Invention

Problems to be Solved by the Invention

[0003] According to embodiments of the present disclosure, various methods, apparatuses, and products for detecting large language model code translation errors are described herein.

Means for Solving the Problems

[0004] In some embodiments, a method for detecting large language model code translation errors includes receiving a code portion of a first programming language; converting the code portion into a second programming language; calculating a first accuracy of the step of converting the code portion into the second programming language; determining a difference between the first accuracy of the conversion from the first programming language to the second programming language and a past accuracy; and indicating a potential error in the code portion of the first programming language based on the difference between the first accuracy and the past accuracy being greater than a predetermined value.

[0005] In one embodiment, the step of calculating the first accuracy of the step of converting the code portion into the second programming language further includes calculating a first code structure representation of the code portion of the first programming language; calculating a second code structure representation of the converted code portion of the second programming language; and calculating the first accuracy based on a comparison between the first code structure representation and the second code structure representation.

[0006] In one embodiment, the step of calculating the first code structure representation of the code portion of the first programming language further includes calculating the first code structure representation of the code portion of the first programming language based on one or more software metrics. In one embodiment, the step of calculating the second code structure representation of the converted code portion of the second programming language further includes calculating the second code structure representation of the converted code portion of the second programming language based on the one or more software metrics. In one embodiment, the one or more software metrics include a code complexity metric.

[0007] In one embodiment, the step of calculating the first code structure representation of the code portion in the first programming language based on one or more software metrics further includes calculating the first code structure representation of the code portion in the first programming language based on a weighted combination of a plurality of the one or more software metrics.

[0008] In one embodiment, the method comprises determining the past accuracy based on the accuracy of at least one previous conversion of another code portion from the first programming language to the second programming language.

[0009] In one embodiment, the step of converting the code portion to a second programming language includes using a generative artificial intelligence model to convert the code portion to the second programming language. In one embodiment, the generative artificial intelligence model includes a large language model.

[0010] In one embodiment, the step of indicating the potential error in the code portion based on the difference between the first accuracy and the past accuracy being greater than a predetermined value further includes providing an indication of the potential error in the code portion to the generative artificial intelligence model.

[0011] In some aspects, the apparatus can comprise a processing device; and a memory operably coupled to the processing device, the memory storing computer program instructions that, when executed, cause the processing device to perform the method. In some aspects, a computer program product comprising a computer-readable storage medium can store computer program instructions that, when executed, perform the method.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

[0013]

Figure 4

[0014]

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0015] With the emergence of large language models and other improvements in artificial intelligence (AI) technology, it has become possible for AI to generate source code. For example, a large language model (LLM) is trained on a large dataset of source code to provide a generative AI that outputs source code based on some input or prompt. As will be discussed in more detail below, the migration of an application from its original source code to new source code can be assisted by such AI. That is, AI can be used to generate new source code based on an input of the original source code. For example, an LLM can be given a prompt such as "generate Java (registered trademark) code that achieves the same purpose as the following COBOL code", where the legacy COBOL source code is provided as an input. In response, the LLM can output AI-generated Java (registered trademark) source code that, at least ideally, implements the same functionality as the legacy and produces the same output. Of course, any new source code, whether generated by a human or by AI, can have bugs, runtime errors, or other glitches.

[0016] Embodiments according to the present disclosure advantageously utilize past accuracy data to detect potential bugs, runtime errors, or other glitches in the original source code of a first programming language (e.g., COBOL) that is later translated into converted source code of a second programming language (e.g., Java (registered trademark)). The process of translating code from one programming language to another by an LLM has advanced significantly. However, problems arise when the translated output code deviates from an acceptable standard. In such situations, it is difficult to determine whether the error in the translated source code is due to a translation error or a fundamental error in the original source code. In various embodiments, it has been recognized that this divergence between an accurate translation for good inputs and a sub-optimal result for bad inputs can function as an indicator as to whether the original source code should be corrected before translating the source code again.

[0017] In certain examples, the LLM used to convert COBOL to Java® produces a specific translation accuracy that can fall within a certain standard deviation, assuming that the COBOL is written correctly without bugs. However, if the COBOL contains bugs or errors, accurately translating the "buggy" COBOL will result in "buggy" Java®. As a result, the accuracy of the code translation can drop dramatically below the standard deviation, which in some cases indicates that there are errors in the input COBOL. When the translation accuracy drops from the past accuracy of translating COBOL to Java®, the hypothesis can arise that either the LLM is translating code it has never seen before or there are errors in the source code. In one example, when the accuracy drops by 20% from the prediction accuracy, the hypothesis that there are bugs in the COBOL code arises with an 80% probability.

[0018] By identifying potential errors in the original source code, the original source code can be verified to identify potential errors. Once the errors are corrected, the LLM can be used to translate the original source code again to produce a more accurate translated code. This process may be repeated until an acceptable translation accuracy is achieved.

[0019] Referring now to FIG. 1, an exemplary computing environment in accordance with an aspect of the present disclosure is shown. Computing environment 100 includes an example of an environment for executing at least some of the computer code associated with the implementation of the various methods described herein, such as code analysis module 107. In addition to code analysis module 107, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuit 120 and cache 121), communication fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and the code analysis module 107 identified above), a set of peripheral devices 114 (including user interface (UI) device set 123, storage 124, and Internet of Things (IoT) sensor set 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, host physical machine set 142, virtual machine set 143, and container set 144.

[0020] The computer 101 may take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device now known or to be developed in the future that is capable of executing a program, accessing a network, or querying a database such as remote database 130. As is well understood in the field of computer technology and depending on the technology, the implementation of computer-implemented methods may be distributed among multiple computers and / or among multiple locations. On the other hand, in this description of the computing environment 100, for the sake of simplicity as much as possible, the detailed discussion focuses on a single computer, specifically the computer 101. Although the computer 101 is not shown within the cloud in FIG. 1, it may be located within the cloud. On the other hand, the computer 101 does not need to exist within the cloud except to the extent that it may be affirmatively shown.

[0021] The processor set 110 includes one or more computer processors of any type now known or to be developed in the future. The processing circuitry 120 may be distributed among multiple packages, for example, multiple interconnected integrated circuit chips. The processing circuitry 120 may implement multiple processor threads and / or multiple processor cores. The cache 121 is a memory located within the processor chip package and is typically used for data or code that should be available for rapid access by threads or cores executing on the processor set 110. Cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set may be located "off-chip". In some computing environments, the processor set 110 may be designed to operate using qubits to perform quantum computing.

[0022] Computer-readable program instructions are typically loaded onto computer 101 and cause a series of operational steps to be performed by a processor set 110 of computer 101, thereby executing a computer-implemented method, and thus the instructions so executed will instantiate the method specified in the flowchart and / or description of the computer-implemented method included in this document. These computer-readable program instructions are stored in various types of computer-readable storage media such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by processor set 110 to control and direct the implementation of the computer-implemented method. In computing environment 100, at least some of the instructions for implementing the computer-implemented method may be stored in code analysis module 107 of persistent storage 113.

[0023] Communication fabric 111 is a signal conduction path that enables various components of computer 101 to communicate with each other. Typically, this fabric is created with switches and conductive paths such as buses, bridges, physical input / output ports, and switches and conductive paths that make up the like. Other types of signal communication paths such as optical fiber communication paths and / or wireless communication paths may be used.

[0024] Volatile memory 112 is any type of volatile memory known now or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not required unless expressly indicated. In computer 101, volatile memory 112 is located within a single package and exists inside computer 101, but alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0025] The persistent storage 113 is any form of non-volatile storage for a computer, whether currently known or to be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is being supplied to the computer 101 and / or directly to the persistent storage 113. The persistent storage 113 may be read-only memory (ROM), but typically at least a portion of the persistent storage enables writing of data, deletion of data, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 may take any of several forms, such as various known proprietary operating systems that employ a kernel or open-source portable operating system interface type operating systems. The code included in the code analysis module 107 typically includes at least some of the computer code related to the implementation of the computer-implemented methods described herein.

[0026] The peripheral device set 114 includes a set of peripheral devices of the computer 101. The data communication connections between the peripheral devices of the computer 101 and other components may be implemented in various ways, such as a Bluetooth (registered trademark) connection, a Near Field Communication (NFC) connection, a connection by a cable (such as a Universal Serial Bus (USB) type cable), an insertion type connection (such as a Secure Digital (SD) card), a connection through a local area communication network, and even a connection through a wide area network such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, a speaker, a microphone, wearable devices (such as goggles and smartwatches), a keyboard, a mouse, a printer, a touchpad, a game controller, and a haptic device. The storage 124 is an external storage such as an external hard drive or an insertable storage such as an SD card. The storage 124 may be persistent and / or volatile. In some embodiments, the storage 124 may take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where the computer 101 is required to have a large amount of storage (for example, the computer 101 locally stores and manages a large-scale database), this storage may be provided by a peripheral storage device designed to store a very large amount of data, such as a Storage Area Network (SAN) shared by a plurality of geographically distributed computers. The IoT sensor set 125 is composed of sensors that can be used in Internet of Things applications. For example, one sensor may be a thermometer, and another sensor may be a motion detector.

[0027] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 101 to communicate with other computers through the WAN 102. The network module 115 may include hardware such as a modem or a Wi-Fi (registered trademark) signal transceiver, software for packetizing and / or depacketizing data for communication network transmission, and / or web browser software for transmitting data over the Internet. In some embodiments, the network control function and the network transfer function of the network module 115 are implemented on the same physical hardware device. In other embodiments (e.g., embodiments that utilize software-defined networking (SDN)), the control function and the transfer function of the network module 115 are physically implemented on separate devices, and thus the control function manages several different network hardware devices. The computer-readable program instructions for implementing the computer-implemented method are typically downloaded to the computer 101 from an external computer or an external storage device through a network adapter card or a network interface included in the network module 115.

[0028] The WAN 102 is any wide area network (e.g., the Internet) that can transmit computer data over a non-local distance by any technique for transmitting computer data known now or to be developed in the future. In some embodiments, the WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to transmit data between devices located in a local area, such as a Wi-Fi (registered trademark) network. The WAN and / or LAN typically includes computer hardware such as copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0029] An end - user device (EUD) 103 is any computer system used and controlled by an end - user (e.g., a customer of an enterprise operating computer 101) and can take any of the forms discussed above in relation to computer 101. The EUD 103 typically receives beneficial and useful data from the operation of computer 101. For example, in a hypothetical case where computer 101 is designed to provide recommendations to an end - user, this recommendation would typically be transmitted from the network module 115 of computer 101, via the WAN 102, to the EUD 103. Thus, the EUD 103 can display or otherwise present the recommendation to the end - user. In some embodiments, the EUD 103 may be a client device such as a thin - client, a thick - client, a mainframe computer, a desktop computer, etc.

[0030] A remote server 104 is any computer system that provides at least some data and / or functionality to computer 101. The remote server 104 may be controlled and used by the same entity that operates computer 101. The remote server 104 represents a machine that collects and stores beneficial and useful data for use by other computers such as computer 101.

[0031] The public cloud 105 is any computer system that provides on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, for use by multiple entities without direct active management by the user. Cloud computing typically exploits resource sharing to achieve consistency and economies of scale. The direct and active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments that run on various computers that make up the host physical machine set 142, which is an aggregate of physical computers that are within and / or available to the public cloud 105. The virtual computing environment (VCE) typically takes the form of virtual machines from the virtual machine set 143 and / or containers from the container set 144. These VCEs may be stored as images and can be understood to be transferable as images, or after instantiation of the VCE, among and within various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of the VCE, and manages the active instantiation of VCE deployments. The gateway 140 is a collection of computer software, hardware, and firmware that enables the public cloud 105 to communicate via the WAN 102.

[0032] Next, some further explanations are provided for the virtualized computing environment (VCE). The VCE can be stored as an "image". A new active instance of the VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of the operating system that enables the kernel to have multiple isolated user-space instances called containers. These isolated user-space instances typically behave as actual computers from the perspective of the programs running within them. A computer program running on a general operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to that container, and this feature is known as containerization.

[0033] The private cloud 106 is similar to the public cloud 105, except that computing resources are only available for use by a single enterprise. The private cloud 106 is shown as being in communication with the WAN 102, but in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible via a local / private network. A hybrid cloud is a composite of multiple different types of clouds (e.g., private, community, or public cloud types), often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0034] For further explanation, FIG. 2 depicts a flowchart of an exemplary method for large language model code translation error detection according to some embodiments of the present disclosure. The method of FIG. 2 may be implemented by a code analysis module 201, such as the code analysis module 107 of FIG. 1, for example. In some embodiments, the code analysis module 201 may be implemented as a separate process or service from the application or software implementing the code analysis. For example, the code analysis module 201 may be implemented by an operating system or other software that monitors the behavior and execution of the application implementing the code analysis. As another example, in some embodiments, the code analysis module 201 may be implemented as a process or service that modifies the application implementing the code analysis.

[0035] The method of FIG. 2 includes a step 202 of receiving a code portion of a first programming language and a step 204 of converting the code portion into a second programming language. Although the execution of an application is described in FIG. 2, it is understood that the techniques described herein may be applied to any software or module capable of implementing code analysis, including applications, operating systems, libraries, and the like. In one example, the code portion may include legacy source code written in an older programming language (e.g., COBOL), and the converted code portion may include code written in a modern programming language (e.g., Java (registered trademark)). In a particular example, both the code portion and the converted code portion are intended to achieve the same purpose, provide the same interface, and produce the same output. In a particular example, the code portion may include an application, subroutine, function, or driver. In one embodiment, the code portion is translated from a first programming language to a second programming language using generative AI such as an LLM. In some examples, some or all of the converted code is generated using an AI language model. For example, some or all of the original source code can be provided as input to an AI language model along with a request (e.g., a prompt) to generate source code in a different programming language that achieves the same purpose as the original source code.

[0036] In one embodiment, the code analysis module 201 calculates 206 the first accuracy of the stage of converting a code portion into a second programming language. In certain embodiments, the first accuracy indicates the degree of accuracy of the translation from the original source code to the converted source code. In one or more embodiments, the first accuracy of the stage of converting a code portion into a second programming language is based on comparing the code structures of the original code portion and the converted code portion to be similar within a predetermined acceptable threshold. In certain embodiments, a first code structure representation of the code portion in the first programming language and a second code structure representation of the converted code portion in the second programming language are calculated. The first code structure representation and the second code structure representation are compared to calculate the first accuracy.

[0037] In one or more embodiments, the calculation of the first code structure representation of the code portion in the first programming language and the second code structure representation of the converted code portion in the second programming language is based on one or more software metrics. In certain embodiments, an Abstract Syntax Tree (AST) is used to calculate the first accuracy of the stage of converting a code portion into a converted code portion. An AST is a data structure used to represent the structure of a code portion in a tree representation of the abstract syntactic structure of the text of a programming language. Each node of the tree points to a construct that appears in the text. The syntax is "abstract" in the sense that it represents only the structural or content-related details, rather than all the details as they appear in the actual syntax. Compared to source code, an AST typically does not include inessential punctuation and delimiters. By comparing the ASTs of the original code portion and the converted code portion, the degree of accuracy of the conversion is calculated.

[0038] In another embodiment, one or more code complexity metrics are used to calculate the code complexity for each of the original code portion and the transformed code portion. The correctly translated source code is expected to exhibit a similar degree of complexity as the original source code. In one or more embodiments, a comparison of the code complexity of the original source code portion and the transformed code portion can be an indicator of possible logical errors, faults, bugs, or other program errors in the original source code portion.

[0039] In one example, the cyclomatic complexity metric is used to calculate the complexity of the original code portion and the transformed code portion. The cyclomatic complexity metric represents the number of linearly independent paths throughout the source code. A path is linearly independent if there exists a subset of the set of edges of those paths whose symmetric difference is empty. For example, if the source code does not contain control flow statements, there is only a single path throughout the code, so the complexity is 1. If the source code contains a single conditional statement, there are two paths throughout the code, indicating a complexity of 2.

[0040] In another example, the Halstead complexity metric is used to calculate the complexity of the original code portion and the transformed code portion. The Halstead complexity metric is calculated statistically without program execution and takes into account elements such as the number of distinct operators, the number of distinct operands, the total number of operators, and the total number of operands. In another example, the maintainability index metric is used to calculate the complexity of the original code portion and the transformed code portion. The maintainability index metric is calculated to represent the relative ease of maintaining the code portion and is calculated based on the number of statements in the code, the cyclomatic complexity, and the Halstead volume (calculated as a function of the code length, the number of distinct operators, and the number of distinct operands).

[0041] In another embodiment, the calculation of the first code structure representation of the code portion in the first programming language and the second code structure representation of the converted code portion in the second programming language is based on a combination of one or more software metrics. In a particular embodiment, the calculation of the first code structure representation of the code portion in the first programming language and the second code structure representation of the converted code portion in the second programming language is based on a weighted combination of a plurality of the one or more software metrics. For example, when calculating each code structure, one or more of the software metrics may be weighted differently from other software metrics.

[0042] In one embodiment, the code analysis module 201 determines 208 the difference between the first accuracy of the conversion from the first programming language to the second programming language and the past accuracy. In one embodiment, the past accuracy is based on the accuracy of at least one previous conversion of another code portion from the first programming language to the second programming language. In a particular embodiment, the accuracy of a previous conversion of the code from the first programming language to the second programming language is used to determine the predicted accuracy within a particular deviation of the translation that converts the code programmed in the first programming language into the converted code in the second programming language. In a particular embodiment, a previous determined accuracy information is used to construct an accuracy model of the code translation from the first programming language to the second programming language.

[0043] In one embodiment, the code analysis module 201 indicates 210 potential errors in a code portion of a first programming language based on the difference between a first accuracy and a past accuracy being greater than a predetermined value. In a specific example, the code analysis module 201 indicates 210 potential errors in a code portion of a first programming language when the difference between the first accuracy and the past accuracy is greater than 20%. In one example, the indication of potential errors in the code portion is used to verify errors in the code portion, correct the errors, and perform a conversion of the code portion of the first programming language to a second programming language using the corrected code. In a specific embodiment, this process is repeatedly iterated until an acceptable accuracy is obtained. In another embodiment, the indication further includes the probability of an error in the code portion.

[0044] In another embodiment, the code analysis module 201 determines that the difference between the first accuracy and the past accuracy is less than a predetermined value, determines that the converted code portion is unexecutable, and indicates that the converted code portion is unexecutable. In a specific embodiment, the step of determining that the converted code portion is unexecutable includes the step of determining that the converted code portion is non-compilable or includes one or more errors.

[0045] For further illustration, FIG. 3 depicts a flowchart of another exemplary method for large language model code translation error detection according to some embodiments of the present disclosure. The method of FIG. 3 extends the method of FIG. 2 in that the step 206 of calculating a first accuracy for the step of converting a code portion to a second programming language includes the step 302 of calculating a first code structure representation of the code portion of the first programming language and the step 304 of calculating a second code structure representation of the converted code portion of the second programming language. The method of FIG. 3 further includes the step 306 of calculating the first accuracy based on a comparison of the first code structure representation and the second code structure representation.

[0046] In one embodiment, the step of calculating the first code structure representation of the code portion of the first programming language is based on one or more software metrics. In one embodiment, the step of calculating the second code structure representation of the converted code portion of the second programming language is also based on one or more software metrics. In one or more embodiments, the one or more software metrics include code complexity metrics. In one embodiment, the step of calculating the first code structure representation of the code portion of the first programming language is based on a weighted combination of a plurality of the one or more software metrics.

[0047] For further illustration, FIG. 4 depicts a flowchart of another exemplary method for large language model code translation error detection according to some embodiments of the present disclosure. The method of FIG. 4 extends the method of FIG. 2 in that the step 208 of determining the difference between the first accuracy and the past accuracy of the conversion from the first programming language to the second programming language includes a step 402 of determining the past accuracy based on the accuracy of at least one previous conversion from the first programming language to the second programming language of another code portion.

[0048] For further illustration, FIG. 5 depicts a flowchart of another exemplary method for large language model code translation error detection according to some embodiments of the present disclosure. The method of FIG. 5 extends the method of FIG. 2 in that the step 204 of converting the code portion to the second programming language includes a step 502 of using a generative artificial intelligence model to convert the code portion to the second programming language. In certain embodiments, the generative artificial intelligence model includes a large language model.

[0049] The method of FIG. 5 further extends the method of FIG. 2 in that stage 210, which indicates potential errors in a code portion of a first programming language based on the difference between a first accuracy and a past accuracy being greater than a predetermined value, includes providing an indication of the potential error in the code portion to a generative artificial intelligence model. In certain embodiments, the generative artificial intelligence model is further trained based on an indication of potential errors in a code portion of the first programming language. In a specific example, the generative artificial intelligence model causes the code portion not to be used in training based on an indication that the potential for an error in the code portion has been indicated.

[0050] The examples described above are provided in the context of translation error detection when migrating an application from an original source code to a new source code. It should be understood that the term "original source code" as used herein is not to be construed as limiting the term to mean the earliest implementation of an application whose accuracy is being verified against a new source code. Embodiments are useful when migrating or porting an application from one programming language to a different programming language and from legacy code to a more modernized programming language, although in some examples it will be further understood that the original source code and the new source code may be written in the same programming language.

[0051] In view of the foregoing, large language model code translation error detection according to the present disclosure provides a number of advantages. By identifying errors in the original source code, greater success is provided in producing an accurate conversion of the original source code to a new source code in a different programming language. Such errors can be corrected to enable the conversion of the original source code to a new source code to increase the accuracy of the new source code and prevent or reduce execution errors in the converted code.

[0052] Various aspects of the present disclosure are described by the description, flowchart, block diagram of a computer system, and / or block diagram of machine logic included in embodiments of a computer program product (CPP). For any flowchart, operations may be performed in an order different from that shown in a given flowchart, depending on the relevant technology. For example, also depending on the relevant technology, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or at least partially in a temporally overlapping manner.

[0053] An embodiment of a computer program product (referred to as a "CPP embodiment" or "CPP") is a term used in this disclosure to describe any set of one or more storage media (also referred to as "mediums") collectively included in a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing computer operations specified in a given CPP claim. A "storage device" is any tangible device capable of holding and storing instructions for use by a computer processor. A computer-readable storage medium may be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media are floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or pits / lands formed on the major surfaces of disks), or any suitable combination of the foregoing. A computer-readable storage medium is not to be construed as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, optical pulses passing through an optical fiber cable, electrical signals transmitted through a wire, and / or other transmission media. As will be understood by those skilled in the art, data is typically moved at some irregular points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but since the data is not transient while it is stored, the above does not cause the storage device to be considered transient.

[0054] The description of various embodiments of the present disclosure is presented for purposes of illustration and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, the practical application, or technical improvements made to the technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. Receiving a code portion of a first programming language; Converting the code portion into a second programming language; Calculating a first accuracy of the step of converting the code portion into the second programming language; Determining a difference between the first accuracy of the conversion from the first programming language to the second programming language and a past accuracy; and Indicating a potential error in the code portion of the first programming language based on the difference between the first accuracy and the past accuracy being greater than a predetermined value A method comprising the steps of:

2. The step of calculating the first accuracy of the step of converting the code portion into the second programming language comprises: Calculating a first code structure representation of the code portion of the first programming language; Calculating a second code structure representation of the converted code portion of the second programming language; and Calculating the first accuracy based on a comparison between the first code structure representation and the second code structure representation The method according to claim 1, further comprising the steps of:

3. The step of calculating the first code structure representation of the code portion of the first programming language further comprises calculating the first code structure representation of the code portion of the first programming language based on one or more software metrics. The method according to claim 2

4. The step of calculating the second code structure representation of the converted code portion of the second programming language further comprises calculating the second code structure representation of the converted code portion of the second programming language based on the one or more software metrics. The method according to claim 3

5. The method according to claim 4, wherein the one or more software metrics include a code complexity metric

6. The step of calculating the first code structure representation of the code portion of the first programming language based on one or more software metrics comprises: Further comprising calculating the first code structure representation of the code portion of the first programming language based on a weighted combination of a plurality of the one or more software metrics. The method according to claim 3

7. The method according to claim 1, further comprising determining the past accuracy based on the accuracy of at least one previous conversion of the other code portion from the first programming language to the second programming language. **Claim 8** The method according to claim 1, wherein the step of converting the code portion into a second programming language comprises using a generative artificial intelligence model to convert the code portion into the second programming language. **Claim 9** The method according to claim 8, wherein the generative artificial intelligence model includes a large language model. **Claim 10** The method according to claim 8, wherein the step of indicating the potential error in the code portion based on the difference between the first accuracy and the past accuracy being greater than a predetermined value further comprises providing an indication of the potential error in the code portion to the generative artificial intelligence model. **Claim 11** Determining that the difference between the first accuracy and the past accuracy is less than the predetermined value; Determining that the converted code portion is unexecutable; and Indicating that the converted code portion is unexecutable The method according to any one of claims 1 to 10, further comprising. **Claim 12** A processing device; and A memory operably coupled to the processing device Comprising, the memory stores computer program instructions which, when executed, cause the processing device to Receive a code portion of a first programming language; Convert the code portion into a second programming language; Calculate a first accuracy of the step of converting the code portion into the second programming language; Determine a difference between the first accuracy of the conversion from the first programming language to the second programming language and the past accuracy; and Indicate a potential error in the code portion of the first programming language based on the difference between the first accuracy and the past accuracy being greater than a predetermined value An apparatus for performing. **Claim 13** When the computer program instructions are executed to calculate the first accuracy of the step of converting the code portion into the second programming language, the processing device is caused to A procedure for calculating a first code structure representation of the code portion in the first programming language; A procedure for calculating a second code structure representation of the converted code portion in the second programming language; and A procedure for calculating the first accuracy based on a comparison between the first code structure representation and the second code structure representation The apparatus according to claim 12, which causes the above to be performed.

14. The procedure for calculating the first code structure representation of the code portion in the first programming language further includes a procedure for calculating the first code structure representation of the code portion in the first programming language based on one or more software metrics. The apparatus according to claim 13.

15. The procedure for calculating the second code structure representation of the converted code portion in the second programming language further includes a procedure for calculating the second code structure representation of the converted code portion in the second programming language based on the one or more software metrics. The apparatus according to claim 14.

16. The one or more software metrics include a code complexity metric. The apparatus according to claim 15.

17. A computer program having computer program instructions, which when executed, A procedure for receiving a code portion in a first programming language; A procedure for converting the code portion into a second programming language; A procedure for calculating a first accuracy of the procedure for converting the code portion into the second programming language; A procedure for determining a difference between the first accuracy of the conversion from the first programming language to the second programming language and a past accuracy; and A procedure for indicating a potential error in the code portion in the first programming language based on the difference between the first accuracy and the past accuracy being greater than a predetermined value A computer program that performs the above.

18. When the computer program instructions are executed to calculate the first accuracy of the procedure for converting the code portion into the second programming language, A procedure for calculating a first code structure representation of the code portion in the first programming language; A procedure for calculating a second code structure representation of the converted code portion in the second programming language; and A procedure for calculating the first accuracy based on a comparison between the first code structure representation and the second code structure representation The computer program according to claim 17, which performs the procedure.

19. The procedure for calculating the first code structure representation of the code portion in the first programming language further includes a procedure for calculating the first code structure representation of the code portion in the first programming language based on one or more software metrics. The computer program according to claim 18.

20. The procedure for calculating the second code structure representation of the converted code portion in the second programming language further includes a procedure for calculating the second code structure representation of the converted code portion in the second programming language based on the one or more software metrics. The computer program according to claim 19.

Images