Method, apparatus and computer program (tagging deterministic code in artificial intelligence-generated code)

Abstract

To solve the following problems in which AI-generated code may also include non-deterministic code that may produce unreliable or non-consistent outputs despite using the same set of inputs, the non-deterministic code may introduce unwanted entropy into a system, leading to errors or performance degradation.SOLUTION: Tagging deterministic code in artificial intelligence-generated code includes the steps of: receiving a code generated by a generative artificial intelligence (AI) model; identifying at least one portion of the code by identifying at least one of one or more portions of deterministic code or one or more portions of non-deterministic code; and tagging the identified at least one portion of the code.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to methods, apparatuses, and products for tagging deterministic code within AI-generated code. Generative artificial intelligence (AI) can be used to generate code such as application source code. Such code can be based on a prompt provided to a generative AI model that describes the function the code is to perform. Such code can also include a conversion from one programming language to another programming language that is executed by the generative AI model. AI-generated code can include deterministic code that always generates the same set of outputs when given the same input. However, AI-generated code can also include non-deterministic code that can generate unreliable or inconsistent outputs despite using the same set of inputs. This non-deterministic code can introduce unnecessary entropy into the system, which can lead to errors or performance degradation.

Summary of the Invention

Problems to be Solved by the Invention

[0002] AI-generated code can also include non-deterministic code that can generate unreliable or inconsistent outputs despite using the same set of inputs. This non-deterministic code can introduce unnecessary entropy into the system, which can lead to errors or performance degradation.

Means for Solving the Problems

[0003] According to embodiments of the present disclosure, various methods, apparatuses, and products for tagging deterministic code in artificial intelligence generated code are described herein. In some aspects, the step of tagging deterministic code in artificial intelligence generated code includes receiving code generated by a generative artificial intelligence (AI) model; identifying at least one portion of the code by identifying one or more portions of the deterministic code, or one or more portions of the non-deterministic code; and tagging at least one portion of the identified code. Thereby, the advantage of identifying the deterministic or non-deterministic portion of the code in the AI-generated code is provided, the importance of which is recognized for the first time by the present disclosure. In some aspects, the apparatus may comprise a processing device; and a memory operably coupled to the processing device, wherein the memory stores computer program instructions that, when executed, cause the processing device to execute the method. In some aspects, a computer program product comprising a computer-readable storage medium may store computer program instructions that, when executed, execute the method.

[0004] In some aspects, the step of tagging deterministic code in artificial intelligence generated code may also include generating a portion of multi-path code corresponding to the portion of the deterministic code in response to tagging the portion of the deterministic code. Thereby, multi-path code becomes available for known portions of deterministic code, which may be more suitable by multi-path as compared to non-deterministic code. In some aspects, the step of tagging deterministic code in artificial intelligence generated code may also include providing data describing at least one portion of the identified code to the generative AI model. This provides the advantage of enhancing the generation of deterministic code by the generative AI model and reducing the generation of non-deterministic code. BRIEF DESCRIPTION OF THE DRAWINGS

[0005]

Figure 1

[0006]

Figure 2

[0007]

Figure 3

[0008]

Figure 4

[0009]

Figure 5

[0010] A deterministic code is a code that always generates the same output when given the same set of inputs. In other words, a deterministic code is operationally consistent. A sequence of deterministic codes can be used as known truth examples when training a generative artificial intelligence (AI) model to generate code. Generative artificial intelligence (AI) uses models such as neural networks to respond to prompts and generate content including code such as source code. For example, a generative AI model can be used to generate code based on a description of the function that the generated code is to implement. As another example, a generative AI model can be used to convert code from one programming language to another by providing base code as input to the generative AI model.

[0011] A generative AI model can generate non-deterministic code. For example, there may be fundamental structural or architectural problems in a program that can generate unreliable or inconsistent outputs despite using the same set of inputs. When used within a system and accepted as known truth, the system will inherit a level of entropy that can lead to errors or bugs. Therefore, it can be advantageous to identify deterministic and non-deterministic code portions within AI-generated code, provide feedback to the generative AI model, and enhance the learning of its deterministic outputs while reducing the future generation of non-deterministic code.

[0012] The identification of deterministic and non-deterministic code can also be useful when introducing multi-path code into a program or application. Multi-path code describes the use of multiple distinct code paths configured to perform similar functions. For example, different code paths may be configured or designed to produce the same output when applied to the same input, or otherwise may be configured or designed to perform similar functions. These different code paths designed to perform the same function may be implemented by different engineers or teams, so the resulting code paths are not identical but are designed to achieve the same outcome. These different code paths may also be written in different languages, access different libraries, or otherwise be different while still being designed to achieve similar functions. Each path of the multi-path code can be accessed using a shared interface such as an Application Program Interface (API) or other understandable interface.

[0013] When encountering a portion of multi-path code during the execution of an application or other software, state information describing the execution state at the time of encountering the portion of the multi-path code can be saved. The state information may describe values for various registers, memory locations, counters, attributes, and the like. If an error occurs in a code path of the multi-path code, the saved state information can be used as a checkpoint to restore or roll back the execution state of the application to the point before executing the code path of the multi-path code where the error occurred. Different code paths of the multi-path code can then be executed. This process can be repeated until the code paths of the multi-path code are executed without errors.

[0014] When the multi-path code is non-deterministic (e.g., when the executed path is non-deterministic), it may be computationally difficult to roll back the execution state in case of an error. For example, if a certain code path changes some shared memory locations or resources, another process or service may access that shared memory location before the error occurs. To roll back the execution state, the shared memory location or resource needs to have its value restored to its state before the change. However, another process may already have accessed potentially incorrect data from that memory location or resource. Therefore, that other process may also need to have its execution state rolled back or restored. As the interdependencies between processes increase, restoring the execution state for non-deterministic code becomes increasingly complex. In contrast, deterministic code that lacks these interdependencies does not require these complex steps to roll back its execution state, so they are better candidates for introducing multi-path code. Therefore, it may be beneficial to identify deterministic code and / or non-deterministic code in order to identify candidates for multi-path code or to exclude candidates from multi-path code.

[0015] Referring now to FIG. 1, an exemplary computing environment in accordance with aspects of the present disclosure is shown. Computing environment 100 includes an example of an environment for the execution of at least some of the computer code involved in the execution of the various methods described herein, such as code analysis module 107. In addition to block 107, computing environment 100 includes, for example, computer 101, wide area network (WAN) 102, end user device (EUD) 103, remote server 104, public cloud 105, and private cloud 106. In this embodiment, computer 101 includes a processor set 110 (including processing circuitry 120 and cache 121), communication fabric 111, volatile memory 112, persistent storage 113 (including operating system 122 and block 107 as specified above), a set of peripheral devices 114 (including a set of user interface (UI) devices 123, storage 124, and a set of Internet of Things (IoT) sensors 125), and network module 115. Remote server 104 includes remote database 130. Public cloud 105 includes gateway 140, cloud orchestration module 141, a set of host physical machines 142, a set of virtual machines 143, and a set of containers 144.

[0016] Computer 101 can take the form of a desktop computer, laptop computer, tablet computer, smartphone, smartwatch, or other wearable computer, mainframe computer, quantum computer, or any other form of computer or mobile device that is currently known or will be developed in the future and is capable of executing programs, accessing a network, or querying a database such as remote database 130. As is well understood in the field of computer technology and depending on the technology, the execution of computer implementation methods can be distributed among multiple computers and / or multiple locations. On the other hand, in the present presentation regarding computing environment 100, for the sake of simplicity of the presentation as much as possible, the detailed discussion focuses on a single computer, specifically computer 101. Although computer 101 is not shown within the cloud in FIG. 1, it can be located within the cloud. On the other hand, computer 101 is not required to exist within the cloud except within any range that can be affirmatively shown.

[0017] Processor set 110 includes one or more computer processors of any type that are currently known or will be developed in the future. Processing circuitry 120 can be distributed across multiple packages, for example, multiple integrated circuit chips that have been adjusted. Processing circuitry 120 can implement multiple processor threads and / or multiple processor cores. Cache 121 is memory located within a processor chip package and is typically used for high-speed access to data or code that should be available to threads or cores executing on processor set 110. Cache memory is typically organized into multiple levels depending on its relative proximity to the processing circuitry. Alternatively, some or all of the cache for the processor set can be located "off-chip". In some computing environments, processor set 110 can be designed to operate using qubits and execute quantum computing.

[0018] Computer-readable program instructions are typically loaded onto computer 101 and executed by a set of processors 110 of computer 101 in a series of operational steps, thereby enabling a computer-implemented method, such that the instructions so executed will instantiate the method specified in the flowchart and / or description of the computer-implemented method included herein. These computer-readable program instructions are stored in various types of computer-readable storage media, such as cache 121 and other storage media discussed below. The program instructions and associated data are accessed by the set of processors 110 to control and direct the execution of the computer-implemented method. In computing environment 100, at least some of the instructions for executing the computer-implemented method may be stored in block 107 within persistent storage 113.

[0019] Communication fabric 111 is a signal conduction path that enables the various components of computer 101 to communicate with each other. Typically, this fabric is created with switches and conductive paths, such as buses, bridges, physical input / output ports, and switches and conductive paths that make up the equivalent, etc. Other types of signal communication paths, such as optical fiber communication paths and / or wireless communication paths, may be used.

[0020] Volatile memory 112 is any type of volatile memory, known currently or to be developed in the future. Examples include dynamic random access memory (RAM) or static RAM. Typically, volatile memory 112 is characterized by random access, although this is not essential unless affirmatively shown. In computer 101, volatile memory 112 is located within a single package and exists inside computer 101, although alternatively or additionally, volatile memory may be distributed across multiple packages and / or located external to computer 101.

[0021] The persistent storage 113 is any form of non-volatile storage for a computer that is currently known or will be developed in the future. The non-volatility of this storage means that the stored data is maintained regardless of whether power is directly supplied to the computer 101 and / or to the persistent storage 113. The persistent storage 113 can be read-only memory (ROM), but usually at least a part of the persistent storage enables writing, deleting, and rewriting of data. Some well-known forms of persistent storage include magnetic disks and solid-state storage devices. The operating system 122 can take several forms, such as various known proprietary operating systems or open-source Portable Operating System Interface (POSIX)-type operating systems that employ a kernel. The code included in block 107 typically includes at least some of the computer code involved in executing the computer-implemented methods described herein.

[0022] The peripheral device set 114 includes a set of peripheral devices of the computer 101. The data communication connections between the peripheral devices of the computer 101 and other components can be implemented in various ways, such as a Bluetooth (registered trademark) connection, a Near-Field Communication (NFC) connection, a connection by a cable (such as a Universal Serial Bus (USB) type cable), an insertion type connection (for example, a Secure Digital (SD) card), a connection through a local area communication network, and even a connection through a wide area network such as the Internet. In various embodiments, the UI device set 123 may include components such as a display screen, a speaker, a microphone, wearable devices (such as goggles and smartwatches), a keyboard, a mouse, a printer, a touchpad, a game controller, and a haptic device. The storage 124 is an external storage such as an external hard drive or an insertable storage such as an SD card. The storage 124 can be persistent and / or volatile. In some embodiments, the storage 124 can take the form of a quantum computing storage device for storing data in the form of qubits. In embodiments where the computer 101 is required to have a large amount of storage (for example, the computer 101 locally stores and manages a large-scale database), this storage can be provided by a peripheral storage device designed to store a very large amount of data, such as a Storage Area Network (SAN) shared by a plurality of geographically dispersed computers. The IoT sensor set 125 is composed of sensors that can be used in applications of the Internet of Things. For example, one sensor can be a thermometer, and another sensor can be a motion detector.

[0023] The network module 115 is a collection of computer software, hardware, and firmware that enables the computer 101 to communicate with other computers through the WAN 102. The network module 115 may include hardware such as a modem or a Wi-Fi (registered trademark) signal transceiver, software for packetizing and / or depacketizing data for communication over a communication network, and / or web browser software for communicating data over the Internet. In some embodiments, the network control function and the network transfer function of the network module 115 are executed on the same physical hardware device. In other embodiments (e.g., embodiments that utilize Software-Defined Networking (SDN)), the control function and the transfer function of the network module 115 are executed on physically separate devices such that the control function manages several different network hardware devices. Computer-readable program instructions for executing the computer-implemented method may typically be downloaded to the computer 101 from an external computer or an external storage device through a network adapter card or a network interface included in the network module 115.

[0024] The WAN 102 is any wide area network (e.g., the Internet) capable of communicating computer data over a non-local distance by any technique for communicating computer data that is currently known or that will be developed in the future. In some embodiments, the WAN 102 may be replaced and / or supplemented by a local area network (LAN) designed to communicate data between devices located in a local area, such as a Wi-Fi network. The WAN and / or LAN typically includes computer hardware such as copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and edge servers.

[0025] The end-user device (EUD) 103 is any computer system used and controlled by an end user (e.g., a customer of an enterprise operating the computer 101), and can take any of the forms discussed above in relation to the computer 101. The EUD 103 typically receives beneficial and useful data from the operation of the computer 101. For example, in a virtual case where the computer 101 is designed to provide recommendations to the end user, this recommendation will typically be communicated from the network module 115 of the computer 101 to the EUD 103 through the WAN 102. In this way, the EUD 103 can display or otherwise present the recommendation to the end user. In some embodiments, the EUD 103 can be a client device such as a thin client, a thick client, a mainframe computer, a desktop computer, and the like.

[0026] The remote server 104 is any computer system that provides at least some data and / or functions to the computer 101. The remote server 104 can be controlled and used by the same entity that operates the computer 101. The remote server 104 represents a machine that collects and stores beneficial and useful data for use by other computers such as the computer 101. For example, in the case of a virtual scenario where the computer 101 is designed and programmed to provide recommendations based on past data, this past data can be provided from the remote database 130 of the remote server 104 to the computer 101.

[0027] The public cloud 105 is any computer system available for use by multiple entities that provides on-demand availability of computer system resources and / or other computing capabilities, particularly data storage (cloud storage) and computing power, without direct active management by the user. Cloud computing typically exploits resource sharing to achieve coherence and economies of scale. The direct active management of the computing resources of the public cloud 105 is performed by the computer hardware and / or software of the cloud orchestration module 141. The computing resources provided by the public cloud 105 are typically implemented by virtual computing environments that run on various computers that make up the host physical machine set 142, which is the universe of physical computers within and / or available to the public cloud 105. Virtual computing environments (VCEs) typically take the form of virtual machines from a virtual machine set 143 and / or containers from a container set 144. It is understood that these VCEs can be stored as images and transferred either as images or after instantiation of the VCE, within and among various physical machine hosts. The cloud orchestration module 141 manages the transfer and storage of images, deploys new instantiations of the VCE, and manages the active instantiation of VCE deployments. The gateway 140 is an aggregate of computer software, hardware, and firmware that enables the public cloud 105 to communicate through the WAN 102.

[0028] Here, some further explanations are provided regarding a virtualized computing environment (VCE). A VCE can be stored as an "image". A new active instance of a VCE can be instantiated from the image. Two well-known types of VCEs are virtual machines and containers. A container is a VCE that uses operating system-level virtualization. This refers to a feature of the operating system where the kernel enables the existence of multiple isolated instances of user space, called containers. These isolated instances of user space typically behave as actual computers from the perspective of the programs running within them. A computer program running on a normal operating system can utilize all the resources of that computer, such as connected devices, files and folders, network shares, CPU power, and quantifiable hardware capabilities. However, a program running inside a container can only use the contents of the container and the devices assigned to the container, and this feature is known as containerization.

[0029] The private cloud 106 is similar to the public cloud 105, except that computing resources are only available for use by a single enterprise. While the private cloud 106 is shown as being in communication with the WAN 102, in other embodiments, the private cloud may be completely disconnected from the Internet and only accessible through a local / private network. A hybrid cloud is a composite of multiple different types of clouds (e.g., private cloud, community cloud, or public cloud types) and is often implemented by different vendors. Each of the multiple clouds remains a separate discrete entity, but the larger hybrid cloud architecture is coupled together by standardized or proprietary technologies that enable orchestration, management, and / or data / application portability between the multiple constituent clouds. In this embodiment, both the public cloud 105 and the private cloud 106 are part of a larger hybrid cloud.

[0030] For further explanation, FIG. 2 shows a flowchart of an exemplary method for tagging deterministic code within AI-generated code, according to some embodiments of the present disclosure. The method of FIG. 2 may be performed, for example, by the code analysis module 107 of FIG. 1. As an example, in some embodiments, the method of FIG. 2 may be performed in response to a request to compile some AI-generated code or during the compilation of such code. As another example, in some embodiments, the method of FIG. 2 may be performed in response to receiving some code from a generative AI model based on some request or prompt.

[0031] The method of FIG. 2 includes receiving 202 the code generated by the generative AI model. In some embodiments, the step 202 of receiving the code generated by the generative AI model includes the step of receiving the code as the output by the generative AI model. For example, the generative AI model may be configured to output the code and provide the code to the process or service executing the method of FIG. 2. In some embodiments, the step 202 of receiving the code generated by the generative AI model includes the step of loading the code from the storage where the code was stored after being generated by the generative AI model.

[0032] In some embodiments, the code generated by the generative AI model has the code generated based on a prompt including a natural language description of some function that the code is to implement or execute. For example, the prompt to the generative AI model may request code to perform a particular task (e.g., opening a network connection to a particular destination, instantiating a database table with particular features, or any other task that can be understood).

[0033] In some embodiments, the code generated by the generative AI model includes code in a particular programming language that has been converted from other code in different programming languages. For example, the generative AI model may be provided with some Cobol code and be prompted to convert the Cobol code to Java (registered trademark). In the examples regarding the conversion of code using the generative AI model described below, the code input to the generative AI model is hereinafter referred to as "base code", while the code output by the generative AI model is hereinafter referred to as "converted code".

[0034] The method of FIG. 2 also includes a step 204 of identifying at least one portion of the code by identifying at least one portion of one or more portions of the deterministic code or one or more portions of the non-deterministic code. As described above, deterministic code includes code that consistently provides the same output when given the same set of inputs. In contrast, non-deterministic code includes code that may not or may not provide the same output when given the same set of inputs. Portions of the code can include various degrees of granularity or scope. For example, a portion of the code can include a defined block of code, a particular function or method, a combination of multiple blocks of code or functions, and the like.

[0035] In some embodiments, identifying at least one portion of the code 204 can include identifying one or more portions of the deterministic code 206 within the code. In other words, portions of the deterministic code are specifically identified 206 from the code. In some embodiments, identifying at least one portion of the code 204 can include identifying one or more portions of the non-deterministic code 208 within the code. In other words, portions of the non-deterministic code are specifically identified 208 from the code. In some embodiments, identifying at least one portion of the code 204 can include both identifying one or more portions of the deterministic code 206 and identifying one or more portions of the non-deterministic code 208. Those skilled in the art will understand that in some embodiments, some portions of the code may not be identified as being deterministic or non-deterministic. For example, in analyzing a given portion of the code, it may not be necessary to definitively determine whether the code is deterministic or non-deterministic.

[0036] In some embodiments, identifying at least one portion of the code 204 may be based on one or more user inputs that identify portions of deterministic and / or non-deterministic code within the code. For example, in some embodiments, the user may highlight or otherwise select a portion of the code and use a menu input, hotkey input, or other input to identify whether the portion of the code is deterministic or non-deterministic. In other words, in some embodiments, identifying at least one portion of the code 204 may be performed using a manual selection by the user of at least one portion of the code.

[0037] In some embodiments, identifying at least one portion of the code 204 can be performed dynamically. For example, in some embodiments, identifying at least one portion of the code 204 can include analyzing the code and applying one or more rules to identify at least one portion of the code 204. For example, the code can be analyzed to identify a particular portion of the code (e.g., a particular block of code or other subsection). One or more rules can be applied to these portions of the code to determine whether a given portion of the code is deterministic or non-deterministic. For example, in some embodiments, the presence of a particular function known to be non-deterministic (e.g., a function that loads or stores data to or from a shared memory location or resource without locking or otherwise restricting access to such location or resource, or a function that depends on random number generation) in a portion of the code can be a factor in identifying that portion of the code as non-deterministic. As another example, the absence of a function known to be non-deterministic in a portion of the code can be a factor in identifying that portion of the code as deterministic. For example, a function that only performs a mathematical calculation on some input and provides some output can be identified as deterministic. In some embodiments, the code can be provided to a trained model (e.g., a generative AI model or some other model that can be understood) configured to identify portions of deterministic code and / or portions of non-deterministic code.

[0038] In some embodiments, the step 204 of identifying at least one portion of the code may include the step of monitoring the execution of the code or a portion thereof to identify deterministic or non-deterministic behavior. For example, by repeatedly executing the code or a portion thereof using multiple sets of the same input, it can be determined whether the output of a portion of the code is consistent across all instances of the same input, thereby indicating deterministic code. As another example, the execution of an application or code may be monitored in the context of a larger system or complete application to detect unexpected or incorrect outputs of functions within the code, thereby indicating non-deterministic code or potentially non-deterministic code (e.g., excluding the code from being identified as deterministic while not explicitly identifying it as non-deterministic).

[0039] The method of FIG. 2 also includes tagging at least one portion of the code 210. In some embodiments, the step 210 of tagging at least one portion of the code may include tagging one or more portions of the identified deterministic code as deterministic code. In some embodiments, the step 210 of tagging at least one portion of the code may include tagging one or more portions of the identified non-deterministic code as non-deterministic code. In some embodiments, the step 210 of tagging at least one portion of the code may include adding some non-compilable label or comment to the code that indicates a portion of deterministic or non-deterministic code. In some embodiments, the step 210 of tagging at least one portion of the code may include generating some data within the code that indicates portions identified as deterministic and / or portions identified as non-deterministic. Thus, in some embodiments, tagging at least one portion of the code 210 may include generating metadata 212 that identifies at least one portion of the code.

[0040] In some embodiments, metadata can identify where in the code a given portion of the code is located (e.g., using line numbers, an offset from some function name or other component of the code to the line, and the like) for a given portion of the code. In some embodiments, metadata can identify whether a given portion of the code has been identified as being deterministic or non - deterministic. In some embodiments, metadata can indicate the reason a given portion of the code has been identified as being deterministic or non - deterministic. For example, the metadata can indicate a particular function in a portion of the code that can be a factor in labeling that portion of the code as non - deterministic, or can indicate a particular rule that was a factor in labeling that portion of the code as being deterministic and / or non - deterministic.

[0041] Identifying 204 and tagging 210 the code as being deterministic and / or non - deterministic provides a number of advantages that are described in more detail below. Identifying 206 and tagging 210 one or more portions of the deterministic code can enable multi - path code to be generated for that deterministic code. Further, the identified 206 and tagged 210 deterministic code can be provided to a generative AI model as feedback and / or training data to enhance the generation of deterministic code. Identifying 208 and tagging 210 one or more portions of the non - deterministic code can be used to limit or prevent the use of multi - path code for the non - deterministic code. Further, the identified 208 and tagged 210 non - deterministic code can be provided to a generative AI model as feedback and / or training data to reduce or mitigate the future generation of non - deterministic code by the generative AI model.

[0042] For further explanation, FIG. 3 shows a flowchart of an exemplary method for tagging deterministic code within AI-generated code, according to some embodiments of the present disclosure. The method of FIG. 3 includes receiving code generated by a generative artificial intelligence (AI) model 202; identifying one or more portions of deterministic code within the code 206; and identifying one or more portions of non-deterministic code within the code 208, thereby identifying at least one portion of the code 204 by identifying at least one of one or more portions of deterministic code or one or more portions of non-deterministic code; and tagging at least one portion of the code 210. In this regard, the method of FIG. 3 is similar to FIG. 2.

[0043] The method of FIG. 3 differs from FIG. 2 in that it also includes generating a portion of multi-path code corresponding to the portion of deterministic code in response to tagging the portion of deterministic code 302. As described above, multi-path code describes the use of multiple distinct code paths configured to perform similar functions. When a portion of multi-path code is encountered during the execution of an application or other software, state information describing the execution state at the time the portion of multi-path code was encountered may be saved. The state information may describe values for various registers, memory locations, counters, attributes, and the like. If an error occurs in a code path of the multi-path code, the saved state information can be used as a checkpoint to restore or roll back the execution state of the application to the time prior to executing the code path of the multi-path code where the error occurred. Different code paths of the multi-path code may then be executed.

[0044] The step 302 of generating multi-path code has the step of generating, selecting, or identifying a portion of other code that is different from but functionally identical to the portion of the deterministic code. The portion of other code may include, for example, code generated by different developers, code written in different languages, code accessing or included in different libraries, and the like. In some embodiments, the step 302 of generating multi-path code has the step of modifying the code to include a reference to an API or other interface for executing the multi-path code. For example, an API call for multi-path code may be inserted before the portion of the deterministic code, or may replace the portion of the deterministic code while being configured to execute the portion of the deterministic code first.

[0045] The method of FIG. 3 describes the use of generating multi-path code in response to tagging a portion of deterministic code, but the tagged portion of non-deterministic code may also affect the use of multi-path code. For example, if multi-path code can be added to the code automatically or dynamically, the portion of the code tagged as non-deterministic may be prevented from being included in the multi-path code. As another example, when compiling or performing some other action on the code including multi-path code, a warning may be generated or the compilation may be blocked in response to detecting multi-path code including some non-deterministic code.

[0046] For further explanation, FIG. 4 shows a flowchart of an exemplary method for tagging deterministic code within AI-generated code, according to some embodiments of the present disclosure. The method of FIG. 4 includes receiving code generated by a generative artificial intelligence (AI) model 202; identifying one or more portions of deterministic code within the code 206; and identifying one or more portions of non-deterministic code within the code 208, thereby identifying at least one portion of the code 204 by identifying at least one of one or more portions of deterministic code or one or more portions of non-deterministic code; tagging at least one portion of the code 210; and generating a portion of multi-path code corresponding to the portion of deterministic code 302 in response to tagging the portion of deterministic code. In this regard, the method of FIG. 4 is similar to FIG. 3.

[0047] The method of FIG. 4 differs from FIG. 3 in that generating a portion of multi-path code corresponding to the portion of deterministic code 302 in response to tagging the portion of deterministic code also includes selecting other code for the portion of multi-path code from a base code 402. In the exemplary method of FIG. 3, assume that the code received from the generative AI model includes converted code in a certain programming language generated from base code in a different programming language. For example, the base code may include code written in Cobol that has been converted to converted code written in Java by the generative AI model.

[0048] Here, other code for the portion of multi-path code (e.g., a code path that would be executed if an error occurs during execution of the portion of deterministic code) is selected from the base code from which the deterministic converted code was generated. In particular, the other code selected from the base code may include a portion of the base code from which the deterministic converted code was generated by the generative AI model. Thus, if execution of the deterministic converted code experiences an error, the portion of the base code from which the deterministic converted code was generated is executed.

[0049] This approach provides additional resilience in AI-generated code. For example, assume that the base code is written in a legacy programming language but is known to be functional and resilient. The base code can be converted to a different programming language so that it runs more optimally on modern computing systems. If an error occurs in the converted code, the base code, which is known to be reliable and functional, can be executed instead by using multi-path code. Further, this approach enables multi-path code without the need to manually write different, functionally equivalent code by using known, functionally equivalent code in different programming languages.

[0050] For further illustration, FIG. 5 shows a flowchart of an exemplary method for tagging deterministic code within artificial intelligence-generated code, according to some embodiments of the present disclosure. The method of FIG. 3 is similar to FIG. 2 in that it also includes receiving code generated by a generative artificial intelligence (AI) model 202; identifying one or more portions of deterministic code within the code 206; and identifying one or more portions of non-deterministic code within the code 208, thereby identifying at least one portion of the code 204; and tagging at least one portion of the code 210.

[0051] The method of FIG. 5 differs from that of FIG. 2 in that it also includes 502 providing data describing at least one portion of the identified code to a generative AI model. In some embodiments, the data provided 502 to the generative AI model may include the generated 212 metadata described above. In some embodiments, the data provided 502 to the generative AI model may be based on, or may include, various data points from the generated 212 metadata. For example, the data provided 502 to the generative AI model may identify specific portions of deterministic and / or non-deterministic code and provide indications and the like as to why a particular portion of the code was identified as deterministic or non-deterministic.

[0052] In some embodiments, the data provided 502 to the generative AI model may include training data for retraining the generative AI model. In other words, the generative AI model may be retrained based on previously generated code and the identification of deterministic and / or non-deterministic code. This provides a number of advantages in using the generative AI model to generate code. For example, by showing the generative AI model portions of deterministic code, this may enhance and facilitate the generation of deterministic code by the generative AI model. As another example, by showing the generative AI model portions of non-deterministic code, this may reduce or decrease the amount of non-deterministic code generated by the generative AI model.

[0053] Various aspects of the present disclosure are illustrated by descriptions, flowcharts, block diagrams of computer systems, and / or block diagrams of machine logic included in embodiments of a computer program product (CPP). For any flowchart, depending on the technology involved, operations may be performed in an order different from that shown in a given flowchart. For example, again depending on the technology involved, two operations shown in consecutive flowchart blocks may be performed in reverse order, as a single integrated step, simultaneously, or at least partially overlapping in time.

[0054] An embodiment of a computer program product (referred to herein as a "CPP embodiment" or "CPP") is, in the present disclosure, a term used to describe any set of one or more storage media (also referred to as "media") collectively included in a set of one or more storage devices that collectively contain machine-readable code corresponding to instructions and / or data for performing the computer operations specified in a given CPP claim. A "storage device" is any tangible device that can hold and store instructions for use by a computer processor. A computer-readable storage medium can be, but is not limited to, an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing. Some known types of storage devices that include these media are floppy disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disc (DVD), memory stick, floppy disk, mechanically encoded devices (such as punch cards or pits / lands formed on the major surfaces of discs), or any suitable combination of the foregoing. A computer-readable storage medium is not to be construed as storage in the form of a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, optical pulses passing through an optical fiber cable, electrical signals transmitted through a wire, and / or other transmission media, as the term is used in the present disclosure. As will be understood by those skilled in the art, data is normally moved at some irregular points during the normal operation of a storage device, such as during access, defragmentation, or garbage collection, but since the data is not transient while it is stored, this does not cause the storage device to be considered transient.

[0055] The description of various embodiments of the present disclosure is presented for illustrative purposes and is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terms used herein are chosen to best explain the principles of the embodiments, the practical application, or technical improvements made to the technologies found in the marketplace, or to enable other skilled artisans to understand the embodiments disclosed herein.

Claims

1. Receiving code generated by an artificial intelligence (AI) model; Identifying at least one portion of the code by identifying at least one portion of deterministic code or at least one portion of non-deterministic code; and Tagging at least one portion of the identified code A method comprising.

2. The method of claim 1, wherein the step of identifying at least one portion of the code includes the step of identifying one or more portions of deterministic code within the code.

3. The method of claim 1, wherein the step of identifying at least one portion of the code includes the step of identifying one or more portions of non-deterministic code within the code.

4. The method according to any one of claims 1 to 3, further comprising generating a portion of multi-path code corresponding to the portion of deterministic code in response to tagging the portion of deterministic code.

5. The method of claim 4, wherein the portion of the multi-path code includes other code in a first programming language, and the portion of the deterministic code is encoded in a second programming language different from the first programming language.

6. The method of claim 5, wherein the code generated by the generating AI model includes converted code in the second programming language converted from base code in the first programming language, and the step of generating the portion of the multi-path code includes selecting the other code from the base code.

7. The method according to any one of claims 1 to 3, further comprising providing data describing at least one portion of the identified code to the generating AI model.

8. The method of claim 7, wherein the data is provided as training data for retraining the generating AI model.

9. The method according to any one of claims 1 to 3, wherein the step of tagging at least one portion of the code includes the step of generating metadata identifying at least one portion of the code.

10. A processing device; and A memory operably coupled to the processing device, wherein the memory, when executed, causes the processing device to: Causing reception of code generated by an artificial intelligence (AI) model; Causing at least one part of the code to be identified by identifying at least one of one or more parts of deterministic code or one or more parts of non-deterministic code; and Causing at least one part of the identified code to be tagged Storing computer program instructions An apparatus comprising. **Claim 11** The apparatus according to claim 10, wherein the procedure for identifying at least one part of the code has a procedure for identifying one or more parts of deterministic code within the code. **Claim 12** The apparatus according to claim 10, wherein the procedure for identifying at least one part of the code has a procedure for identifying one or more parts of non-deterministic code within the code. **Claim 13** The computer program instructions, when further executed, cause the processing device to generate a part of multi-path code corresponding to the part of the deterministic code in response to a procedure for tagging the part of the deterministic code. The apparatus according to any one of claims 10 to 12. **Claim 14** The part of the multi-path code includes other code in a first programming language, and the part of the deterministic code is encoded in a second programming language different from the first programming language. The apparatus according to claim 13. **Claim 15** The code generated by the generating AI model includes converted code in the second programming language converted from base code in the first programming language by the generating AI model, and the procedure for generating the part of the multi-path code has a procedure for selecting the other code from the base code. The apparatus according to claim 14. **Claim 16** The computer program instructions, when further executed, cause the processing device to provide data describing at least one part of the identified code to the generating AI model. The apparatus according to any one of claims 10 to 12. **Claim 17** The apparatus according to claim 16, wherein the data is provided as training data for re-training the generating AI model. **Claim 18** The procedure of tagging at least one part of the code has a procedure of generating metadata for identifying at least one part of the code, the apparatus according to any one of claims 10 to 12.

19. To a computer, A procedure of receiving code generated by a generating artificial intelligence (AI) model; A procedure of identifying at least one part of the code by identifying one or more parts of deterministic code or one or more parts of non-deterministic code; and A procedure of tagging at least one part of the identified code A computer program for causing to execute.

20. The procedure of identifying at least one part of the code has a procedure of identifying one or more parts of deterministic code within the code, the computer program according to claim 19.

21. The procedure of identifying at least one part of the code has a procedure of identifying one or more parts of non-deterministic code within the code, the computer program according to claim 19.

22. To the computer, A procedure of generating a part of multi-path code corresponding to the part of the deterministic code in response to a procedure of tagging the part of the deterministic code The computer program according to any one of claims 19 to 21 for further causing to execute.

23. The part of the multi-path code includes other code in a first programming language, and the part of the deterministic code is encoded in a second programming language different from the first programming language, the computer program according to claim 22.

24. The code generated by the generating AI model includes converted code in the second programming language converted from base code in the first programming language by the generating AI model, and the procedure of generating the part of the multi-path code has a procedure of selecting the other code from the base code, the computer program according to claim 23.

25. To the computer, A procedure of providing data describing at least one part of the identified code to the generating AI model The computer program according to any one of claims 19 to 21 for further causing to execute.

Images