Programming language interacting with large language models

Abstract

In an example, a software package is provided that provides an interface such that when software is executed, a user can provide a natural language hint using a scripting language that guides the interface to interact with a large language model (LLM), adding contextual information to the hint sent to the LLM. The interface can then also extract and evaluate programming code generated by the LLM so that the generated code can be used immediately in the software package without the need for user editing and / or copying / pasting.

Description

Technical Field

[0001] This document generally relates to computer systems. More specifically, this document relates to the use of large language models (LLMs). Background Art

[0002] LLM refers to artificial intelligence (AI) systems that have been trained on extensive datasets to understand and generate human language. These models are designed to process and understand natural language in a way that allows them to answer questions, engage in conversations, generate text, and perform a variety of language-related tasks. Summary of the invention

[0003] In an embodiment, a system includes: at least one hardware processor; and a computer-readable medium storing instructions, which, when executed by the at least one hardware processor, cause the at least one hardware processor to perform operations, the operations including: identifying commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library; converting the commands into a user portion of a natural language prompt; adding a predefined system portion to the natural language prompt; sending the natural language prompt to a large language model (LLM) to generate a first piece of software code in a first programming language based on the natural language prompt; verifying the first piece of software code using a verification function of the first programming language; and inserting the first piece of software code into the software programming code. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The present disclosure is illustrated by way of example, and not limitation, in the figures of the accompanying drawings in which like references indicate similar elements.

[0005] Figure 1 is a block diagram illustrating a system for automatically generating software programming code according to an example embodiment.

[0006] Figure 2 is a flow chart illustrating a method for automatically generating a portion of software programming code according to an example embodiment.

[0007] Figure 3 is a block diagram illustrating a software architecture that may be installed on any one or more of the above-described devices.

[0008] Figure 4 A diagrammatic representation of a machine in the form of a computer system within which a set of instructions may be executed, causing the machine to perform any one or more of the methodologies discussed herein, is shown according to an example embodiment. DETAILED DESCRIPTION

[0009] The following description discusses illustrative systems, methods, techniques, instruction sequences, and computing machine program products. In the following description, for the purpose of explanation, many specific details are set forth in order to provide an understanding of various example embodiments of the present subject matter. However, it will be apparent to those skilled in the art that various example embodiments of the present subject matter may be practiced without these specific details.

[0010] Generating programming code to create new software applications can be tedious and time consuming. LLMs offer the opportunity to greatly speed up this process by allowing a computer model to create programming code based on natural language user prompts. However, using LLMs to create programming code is subject to technical problems. Specifically, the generated code often has errors that must be fixed manually and iteratively by the user, often because the LLMs do not have access to contextual information that would help make the code better conform to the user's intent or match the syntax of the desired programming language. In addition, using the generated code requires the user to perform a lot of copying and pasting.

[0011] In an example embodiment, a software package is provided that provides an interface so that when the software is executed, a user can provide natural language prompts using a scripting language that directs the interface to interact with the LLM, adding contextual information to the prompts sent to the LLM. The interface is then also able to extract and evaluate programming code generated by the LLM so that the generated code can be used immediately in the software package without the need for user editing and / or copy / paste.

[0012] LLMs for generating information are often referred to as Generative Artificial Intelligence (GAI) models. GAI models can be implemented as Generative Pre-Trained Transformer (GPT) models or bidirectional encoders. The GPT model is a machine learning model that uses a transformer architecture, which is a deep neural network that excels at processing sequential data such as natural language.

[0013] A bidirectional encoder is a type of neural network architecture in which the input sequence is processed in two directions: forward and backward. The forward direction starts at the beginning of the sequence and processes the input one token at a time, while the backward direction starts at the end of the sequence and processes the input in reverse order.

[0014] By processing the input sequence in two directions, the bidirectional encoder can capture more contextual information and dependencies between words, leading to better performance.

[0015] The bidirectional encoder can be implemented as a Bidirectional Long Short-Term Memory (BiLSTM) or BERT (Bidirectional Encoder Representations from Transformers) model.

[0016] Each direction has its own hidden state, and the final output is the combination of the two hidden states.

[0017] Long Short-Term Memory (LSTM) is a type of recurrent neural network (RNN) designed to overcome the vanishing gradient problem in traditional RNNs, which can make it difficult to learn long-term dependencies in sequential data.

[0018] LSTM includes a cell state, which acts as a memory to store information over time. The cell state is controlled by three gates: input gate, forget gate, and output gate. The input gate determines how much new information is added to the cell state, while the forget gate determines how much old information is discarded. The output gate determines how much of the cell state is used to calculate the output. Each gate is controlled by a sigmoid activation function, which outputs a value between 0 and 1 that determines the amount of information that passes through the gate.

[0019] In BiLSTM, there are separate LSTMs for the forward direction and the backward direction. At each time step, the forward and backward LSTM cells receive the current input token and the hidden state from the previous time step. The forward LSTM processes the input tokens from left to right, while the backward LSTM processes them from right to left.

[0020] The output of each LSTM cell at each time step is a combination of the input token and the previous hidden state, which allows the model to capture both short-term and long-term dependencies between input tokens.

[0021] BERT applies bidirectional training of models called transformers to language modeling. This contrasts with existing technology solutions that view text sequences from left to right or right to left. Bidirectionally trained language models have a deeper sense of language context and fluency than unidirectional language models.

[0022] More specifically, the Transformer encoder reads the entire sequence of information at once and is therefore considered bidirectional (although one could argue that it is actually non-directional). This property allows the model to learn the context of a piece of information based on all of its surroundings.

[0023] In other example embodiments, a Generative Adversarial Network (GAN) embodiment may be used. A GAN is a supervised machine learning model with two sub-models: a generator model that is trained to generate new examples and a discriminator model that attempts to classify examples as real or generated. The two models are trained together in an adversarial manner (using a zero-sum game according to game theory) until the discriminator model is fooled about half of the time, meaning that the generator model is generating reasonable examples.

[0024] The generator model takes as input a random vector of fixed length, and generates samples in the domain in question. The vectors are randomly drawn from a Gaussian distribution, and the vectors are used to seed the generative process. After training, the points in this multidimensional vector space will correspond to points in the problem domain, forming a compressed representation of the data distribution. This vector space is called a latent space or a vector space consisting of latent variables. Latent variables or hidden variables are those variables that are important to the domain but cannot be directly observed.

[0025] The discriminator model takes as input examples from the domain (real or generated) and predicts a binary class label of real or fake (generated).

[0026] Generative modeling is an unsupervised learning problem, although a clever property of the GAN architecture is that the training of the generative model is framed as a supervised learning problem.

[0027] Both models (generator and discriminator) are trained together. The generator generates a batch of samples and these samples along with real examples from the domain are given to the discriminator and classified as real or fake.

[0028] The discriminator is then updated to better distinguish real from fake samples in the next round, and importantly, the generator is updated based on how well the generated samples fool the discriminator.

[0029] In another example embodiment, the GAI model is a variational autoencoder (VAE) model. A VAE includes an encoder network that compresses input data into a low-dimensional representation called a latent code, and a decoder network that generates new data from the latent code.

[0030] Regardless of the form of the GAI model, the GAI model contains a generative classifier, which can be implemented as, for example, a Naive Bayes classifier.

[0031] The present solution works with any type of GAI model, but an implementation will be described specifically for use with a GPT model. In such an implementation, the programming language provided by the interface may be referred to as PromptScript, because it is designed to interact with a GPT model via prompts, and it is a scripting language. In some example embodiments, the scripting language may be built on top of an existing scripting language, such as JavaScript, but this is not required.

[0032] Furthermore, in an example embodiment, a caching mechanism is introduced for caching the generated code, so that if the same natural language prompt in a scripting language (e.g., PromptScript) is provided later, the previously generated code can be retrieved from the cache and utilized without the need to generate the code again, thereby greatly speeding up the execution of future calls to the prompt.

[0033] This solution also works with any type of application developer software.

[0034] Cloud computing can be described as Internet-based computing that provides shared computer processing resources and data to computers and other devices on demand. Users can establish corresponding sessions during which processing resources and bandwidth are consumed. During a session, for example, users are provided with on-demand access to a shared pool of configurable computing resources (e.g., computer networks, servers, storage devices, applications, and services). Computing resources can be provisioned and released (e.g., scaled) to meet user needs.

[0035] The common architecture in the cloud platform includes services (also called microservices), which are popularized in service-oriented architecture (SOA). In such an SOA, an application consists of multiple independent services. Services are deployed and managed within the cloud platform and run on top of the cloud infrastructure. In some examples, an application programming framework can be used to create and / or extend service-based applications. In an example embodiment, the created software server implements the service.

[0036] Figure 1 1 is a block diagram illustrating a system 100 for automatically generating software programming code according to an example embodiment. Here, an application development server 102 provides a server that allows one or more clients (such as an application development client 104) to create and manage software servers (such as OData servers for web services). The application development client 104 is typically used to control the application development server 102 to create an application using a first programming language, and the application development client 104 can be where a user creates a software application 106 (although in this case, it will also typically be stored as a backup at the application development server 102 and / or allow other users to edit it). However, this aspect is not intended to be limiting, and in some example embodiments, the software application 106 is created only on the application development server 102. The software application 106 (which is essentially in programming language form before being compiled or interpreted) can typically include a library 108, which includes a programming language specific file 110. For example, if a user wants to use the application development client 104 to create an application in JavaScript, the programming language specific file is a JavaScript library.

[0037] In the exemplary embodiment, a dedicated programming language is added on top of the programming language (here JavaScript) that the user wishes to program in, so an interface package (here referred to as a prompt script package 112) is included in the programming language specific files 110 in the library 108. The prompt script package 112 includes all the files required to create and execute the LLM programming language interface described herein.

[0038] Thus, when executed, the prompt script package 112 launches the prompt script interface 114 with which a user may communicate to generate application code using the LLM 116 .

[0039] As previously mentioned, a file system cache 118 may be provided that caches code generated by the LLM 116 for future calls.

[0040] More specifically, the prompt script interface 114 receives a prompt (optionally including one or more test functions) from a user (via the application development client 104). First, the file system cache 118 is checked to determine whether the same prompt has been previously received and processed. This can be performed, for example, by applying a hash function to the provided prompt and determining whether a hash generated by the hash function is already stored in the system cache. If so, the generated code associated with the hash can be returned from the file system cache 118 to the prompt script package 112 for use as part of the application code that the user is constructing.

[0041] If not, the prompt script interface 114 adds a predefined system message to the prompt, and sends the combined system message / prompt (including any provided test functions) to the LLM 116, which generates code based on the combined system message / prompt. The prompt script interface 114 then parses the code. The system message may have instructed the LLM 116 to return code within a markdown fence. Markdown fences (also known as code fences or code blocks) are features in the grammar in which code blocks are represented by inserting special characters before and after the highlighted code block. For example, backticks (`) or tildes (~) can be inserted at the beginning and end of a code block to delineate the edges of the code.

[0042] The hint script interface 114 can then parse the returned code by looking for a fence, which allows the hint script interface 114 to discard any extraneous text returned outside the fence. Thus, for example, the LLM 116 may return some generated JavaScript code within a fence and some extraneous text before or after the JavaScript code, and the hint script interface 114 can then extract the JavaScript code using the fence. However, if no such fence is found, the hint script interface 114 can accept the returned code as is.

[0043] The prompt script interface 114 then validates the code using a validation function specific to the programming language. JavaScript contains a function called "eval" that evaluates whether the provided code meets JavaScript's syntax and other programming requirements and then executes it. The eval function returns the result of any expression within the input string. If the code is invalid, it will return an error. A test function is a generated code segment that is called using a specified input with an expected specified output. Each test function provides a way for the prompt script interface 114 to determine whether the generated code meets the user's intention for the generated code.

[0044] As an example, a user may wish to generate a Fibonacci function in JavaScript that, when executed, generates a value of the Fibonacci sequence at a position specified by the function's input. Thus, for example, if "4" is input, the user may wish the Fibonacci function to return "3" because "3" is at the 4th position of the Fibonacci sequence (assuming that the initial position in the sequence is assigned position 0). If the user simply enters a prompt to prompt script interface 114 that "generates a Fibonacci function in JavaScript," there is a possibility that the generated function will not satisfy the user's intent for the function. For example, the generated function may return a function that outputs the entire sequence of values ​​in the Fibonacci sequence up to and including the input position (e.g., if "4" is input, it will return "0, 1, 1, 2, 3"). Therefore, to avoid this, the user may provide test functions "assert.equal(x(0), 0)" and "assert.equal(x(1), 1)" that specify the expected output of the generated function given a particular input.

[0045] If the prompt script interface 114 determines that the returned code fails at any stage (e.g., is not parseable, fails the validation function, or fails any test function provided by the user), it repeats the request to the LLM to generate code, essentially retrying the code generation. This continues until all of these evaluations pass.

[0046] Once code that passes all evaluations has been received, it may be returned to the programming language specific file 110 and / or the application development client 104 for use in a computer program (eg, to be later compiled or interpreted), and written to the file system cache 118 for storage.

[0047] The prompt script interface 114 allows scripting language commands in a prompt script to be specified within computer code written by a user. Thus, for example, if a user is creating a JavaScript program, the user may first indicate in the JavaScript program that the prompt script package 112 should be loaded, such as by specifying a "constp=require('PromptScript')" command at or near the beginning of the JavaScript code. The prompt script package then specifies a series of commands that can be used within the JavaScript code to trigger the LLM calls required to generate the code.

[0048] Thus, the user may type the following into the software code of the software application 106 to continue the Fibonacci function described above:

[0049] Const result=p'a fibonnaci function'.t(x=>{

[0050] assert.equal(x(0), 0)

[0051] assert.equal(x(1), 1)

[0052] }

[0053] Thus, the prompt script interface 114 generates a natural language prompt as follows, which can then be passed to the LLM 116 along with the system message:

[0054] Creating the Javascript

[0055] Fibonnaci Function

[0056] This function satisfies

[0057] assert.equal(x(0), 0)

[0058] assert.equal(x(1), 1)

[0059] System messages can be as follows:

[0060] const systemMsg = "You are a Node.js expert. Create the described JavaScript object and return it in a ```js...```` fence, so I can write `const result = eval(...)` to get it, without outputting additional text. Example:\n```js\nconst x = function(a) {console.log(a)}; x```"

[0061] Let userMsg = string

[0062] if(testFunction)userMsg+='\nso it satisfies the following test function\n'+testFunction.toString()

[0063] Figure 2 2 is a flow chart illustrating a method 200 for automatically generating a portion of software programming code according to an example embodiment. Here, at operation 202, a scripting language interface specified in the software programming code is called. The scripting language interface is designed to identify one or more specific commands in the software programming code. At operation 204, the scripting language interface identifies a command in the one or more specific commands in the software programming code. The command includes a first portion containing a first natural language prompt to generate a section of software code, and optionally a second portion containing one or more test functions.

[0064] At operation 206, a hash is created from the first portion by passing the first portion through a hash function. At operation 208, the file system cache is checked using the hash to determine if the file system cache already contains the hash. If so, the software code segment has already been generated and stored in the file system cache, and thus at operation 210, the software code segment is retrieved from the file system cache and inserted into the software programming code. More specifically, the eval function is used to evaluate the code (in the form of a string) and thus essentially interpret it on the fly.

[0065] If not, at operation 212, the command (including the first part and the second part, if present) is converted by the scripting language interface into a user portion of a second natural language prompt, which also includes a predefined system message in the system portion. At operation 214, the second natural language prompt is sent as input to the LLM, which generates a software code segment based on the second natural language prompt. At operation 216, the generated software code segment is received by the scripting language interface.

[0066] At operation 218, extraneous text is removed from the generated first segment of software code. This may include removing any text outside the markdown fence if the system message instructs the LLM to generate code within the markdown fence.

[0067] At operation 220, the generated software code segment is verified using a verification function of the programming language for the software programming code to determine whether the generated software code segment is valid according to one or more requirements of the programming language. If not, the method 200 returns to operation 214, which retries sending the second natural language prompt to the LLM.

[0068] If yes, then at operation 222, the generated software code segment is tested using one or more test functions (if any) to determine whether all of the one or more test functions evaluate to true. If no, the method 200 returns to operation 214, which retries sending the second natural language prompt to the LLM.

[0069] If so, then at operation 224, the generated software code segment is inserted into the software programming code. More specifically, the code (in the form of a string) is evaluated using the eval function and thus essentially interpreted on the fly. At operation 226, the generated software code segment is stored in the file system cache so that future calls of the first prompt do not need to regenerate the code, thereby speeding up future calls.

[0070] If an error is returned at any stage, the method 200 may cause a second natural language prompt to be resent to the LLM. This may cause the LLM to generate a different output, at least in a non-zero temperature LLM (non-deterministic LLM). Optionally, any error returned may be fed to the LLM along with the second natural language prompt to essentially instruct the LLM to avoid the error in the future.

[0071] In view of the above disclosure, various examples are set forth below. It should be noted that one or more features of the examples taken alone or in combination should be considered within the disclosure of the present application.

[0072] Example 1. A system comprising:

[0073] at least one hardware processor; and

[0074] A computer readable medium storing instructions, which when executed by at least one hardware processor causes the at least one hardware processor to perform operations, the operations comprising:

[0075] Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library;

[0076] The user portion that converts commands into natural language prompts;

[0077] Add predefined system sections to natural language prompts;

[0078] sending the natural language prompt to the LLM to generate a first piece of software code in a first programming language based on the natural language prompt;

[0079] verifying the first piece of software code using a verification function in a first programming language; and

[0080] Insert the first piece of software code into the software programming code.

[0081] Example 2. The system of Example 1, wherein the command comprises a natural language portion and one or more test functions, wherein the natural language portion describes the desired programming code segment in natural language.

[0082] Example 3. The system of Example 2, wherein the operations further comprise:

[0083] executing one or more test functions against the first piece of software code; and

[0084] The inserting is in response to determining that the first segment of software code is executed by all of the one or more test functions.

[0085] Example 4. A system according to any of Examples 1-3, wherein, in response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

[0086] Example 5. A system according to Example 3, wherein, in response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

[0087] Example 6. The system of any of Examples 1-5, wherein the operations further comprise:

[0088] generating a hash of the command by applying a hash function to the command; and

[0089] The hash and the first piece of software code are saved in a file system cache.

[0090] Example 7. The system of any of Examples 1-6, wherein the operations further comprise:

[0091] identifying a second command in the software programming code through the scripting language interface;

[0092] generating a hash of a second command by applying a hash function to the command;

[0093] determining whether a hash of the second command is contained in the file system cache; and

[0094] In response to determining that the hash of the second command is contained in the file system cache, retrieving the first segment of software code from the file system cache and inserting the first segment of software code into the software programming code to replace the user portion that converts the command into a second natural language prompt, and sending the second natural language prompt to the LLM.

[0095] Example 8. A method comprising:

[0096] Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library;

[0097] The user portion that converts commands into natural language prompts;

[0098] Add predefined system sections to natural language prompts;

[0099] sending the natural language prompt to the LLM to generate a first piece of software code in a first programming language based on the natural language prompt;

[0100] verifying the first piece of software code using a verification function in a first programming language; and

[0101] Insert the first piece of software code into the software programming code.

[0102] Example 9. The method of Example 8, wherein the command comprises a natural language portion and one or more test functions, wherein the natural language portion describes the desired programming code segment in natural language.

[0103] Example 10. The method according to Example 9, further comprising:

[0104] executing one or more test functions against the first piece of software code; and

[0105] Wherein the inserting is in response to determining that the first segment of software code is executed by all test functions of the one or more test functions.

[0106] Example 11. A method according to any of Examples 8-10, wherein, in response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

[0107] Example 12. The method of Example 11, wherein, in response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending a natural language prompt to the LLM.

[0108] Example 13. The method according to any one of Examples 8-12, further comprising:

[0109] generating a hash of the command by applying a hash function to the command; and

[0110] The hash and the first piece of software code are saved in a file system cache.

[0111] Example 14. The method according to any one of Examples 8-13, further comprising:

[0112] identifying a second command in the software programming code through the scripting language interface;

[0113] generating a hash of a second command by applying a hash function to the command;

[0114] determining whether a hash of the second command is contained in the file system cache; and

[0115] In response to determining that the hash of the second command is contained in the file system cache, retrieving the first segment of software code from the file system cache and inserting the first segment of software code into the software programming code to replace the user portion that converts the command into a second natural language prompt, and sending the second natural language prompt to the LLM.

[0116] Example 15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising:

[0117] Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library;

[0118] The user portion that converts commands into natural language prompts;

[0119] Add predefined system sections to natural language prompts;

[0120] sending the natural language prompt to the LLM to generate a first piece of software code in a first programming language based on the natural language prompt;

[0121] verifying the first piece of software code using a verification function in a first programming language; and

[0122] Insert the first piece of software code into the software programming code.

[0123] Example 16. The non-transitory machine-readable medium of Example 15, wherein the command comprises a natural language portion in which the desired programming code segment is described in natural language and one or more test functions.

[0124] Example 17. The non-transitory machine-readable medium of Example 16, wherein the operations further comprise:

[0125] executing one or more test functions against the first piece of software code; and

[0126] The inserting is in response to determining that the first segment of software code is executed by all of the one or more test functions.

[0127] Example 18. The non-transitory machine-readable medium of any of Examples 15-17, wherein, in response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

[0128] Example 19. The non-transitory machine-readable medium of Example 18, wherein, in response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

[0129] Example 20. The non-transitory machine-readable medium of any of Examples 15-19, wherein the operations further comprise:

[0130] generating a hash of the command by applying a hash function to the command; and

[0131] The hash and the first piece of software code are saved in a file system cache.

[0132] Figure 3 is a block diagram 300 illustrating a software architecture 302 that may be installed on any one or more of the devices described above. Figure 3 This is merely a non-limiting example of a software architecture, and it should be understood that many other architectures may be implemented to facilitate the functionality described herein. In various embodiments, the software architecture 302 is comprised of, for example, Figure 4 4, and an embodiment of the present invention is a hardware implementation of a machine 400, which includes a processor 410, a memory 430, and an input / output (I / O) component 450. In this example architecture, the software architecture 302 can be conceptualized as a stack of layers, wherein each layer can provide specific functionality. For example, the software architecture 302 includes layers such as an operating system 304, a library 306, a framework 308, and an application 310. In operation, consistent with some embodiments, the application 310 calls an API call 312 through the software stack and receives a message 314 in response to the API call 312.

[0133] In various embodiments, operating system 304 manages hardware resources and provides public services. Operating system 304 includes, for example, kernel 320, services 322, and drivers 324. Consistent with some embodiments, kernel 320 acts as an abstraction layer between hardware and other software layers. For example, kernel 320 provides memory management, processor management (e.g., scheduling), component management, networking and security settings, and other functions. Services 322 can provide other public services to other software layers. According to some embodiments, drivers 324 are responsible for controlling or interfacing with the underlying hardware. For example, drivers 324 may include display drivers, camera drivers, Bluetooth drivers, etc. or Low energy drivers, flash drives, serial communication drivers (e.g., Universal Serial Bus (USB) drivers), Drivers, audio drivers, power management drivers, etc.

[0134] In some embodiments, the library 306 provides a low-level common infrastructure utilized by the application 310. The library 306 may include a system library 330 (e.g., a C standard library), which may provide functions such as memory allocation functions, string manipulation functions, mathematical functions, etc. In addition, the library 306 may include an API library 332, such as a media library (e.g., a library that supports the presentation and manipulation of various media formats, such as Moving Picture Experts Group-4 (MPEG4), Advanced Video Coding (H.264 or AVC), Moving Picture Experts Group Layer-3 (MP3), Advanced Audio Coding (AAC), Adaptive Multi-Rate (AMR) audio codec, Joint Photographic Experts Group (JPEG or JPG), or Portable Network Graphics (PNG)), a graphics library (e.g., an OpenGL framework for rendering in two dimensions (2D) and three dimensions (3D) in a graphics context on a display). A database library (e.g., SQLite that provides various relational database functions), a web library (e.g., WebKit that provides web browsing functions), etc. The library 306 may also include a variety of other libraries 334 to provide many other APIs to the application 310.

[0135] According to some embodiments, the framework 308 provides a high-level common infrastructure that can be utilized by the applications 310. For example, the framework 308 provides various graphical user interface functions, advanced resource management, advanced location services, etc. The framework 308 can provide a wide range of other APIs that can be utilized by the applications 310, some of which can be specific to a particular operating system 304 or platform.

[0136] In an exemplary embodiment, applications 310 include a home application 350, a contacts application 352, a browser application 354, a book reader application 356, a location application 358, a media application 360, a messaging application 362, a game application 364, and various other applications, such as third-party applications 366. According to some embodiments, application 310 is a program that executes the functions defined in the program. Various programming languages ​​can be used to create one or more applications 310 structured in various ways, such as an object-oriented programming language (e.g., Objective-C, Java, or C++) or a procedural programming language (e.g., C or assembly language). In a specific example, third-party application 366 (e.g., a third-party application created by an entity other than a vendor of a particular platform using ANDROID TM or IOS TM Software Development Kit (SDK) can be used to develop applications on mobile operating systems such as IOS TMANDROID TM , Phone or another mobile operating system). In this example, third-party applications 366 can call API calls 312 provided by operating system 304 to facilitate the functions described herein.

[0137] Figure 4 A diagrammatic representation of a machine 400 in the form of a computer system according to an example embodiment is shown within which a set of instructions may be executed to cause the machine 400 to perform any one or more of the methodologies discussed herein. Specifically, Figure 4 A diagrammatic representation of a machine 400 in the example form of a computer system is shown in which instructions 416 (e.g., software, programs, applications, applet, apps, or other executable code) may be executed to cause the machine 400 to perform any one or more of the methodologies discussed herein. For example, the instructions 416 may cause the machine 400 to perform Figure 2 Additionally or alternatively, instruction 416 may implement Figure 1-Figure 2 Etc. Instructions 416 transform a general unprogrammed machine 400 into a specific machine 400 that is programmed to perform the functions described and shown in the described manner. In alternative embodiments, the machine 400 operates as a standalone device or can be coupled (e.g., networked) to other machines. In a networked deployment, the machine 400 can operate in the capacity of a server machine or a client machine in a server-client network environment, or as a peer machine in a peer (or distributed) network environment. The machine 400 may include, but is not limited to, a server computer, a client computer, a personal computer (PC), a tablet computer, a laptop computer, a netbook, a set-top box (STB), a personal digital assistant (PDA), an entertainment media system, a cellular phone, a smart phone, a mobile device, a wearable device (e.g., a smart watch), a smart home device (e.g., a smart appliance), other smart devices, a network device, a network router, a network switch, a bridge, or any machine capable of executing instructions 416 sequentially or otherwise. It specifies the actions to be taken by the machine 400. Further, while a single machine 400 is illustrated, the term "machine" shall also be taken to include any collection of machines 400 that individually or jointly execute instructions 416 to perform any one or more of the methodologies discussed herein.

[0138] Machine 400 may include processor 410, memory 430, and I / O components 450, which may be configured to communicate with each other, such as via bus 402. In an example embodiment, processor 410 (e.g., a central processing unit (CPU), a reduced instruction set computing (RISC) processor, a complex instruction set computing (CISC) processor, a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a radio frequency integrated circuit (RFIC), another processor, or any suitable combination thereof) may include, for example, processor 412 and processor 414 that may execute instructions 416. The term "processor" is intended to include multi-core processors, which may include two or more independent processors (sometimes referred to as "cores") that may execute instructions 416 simultaneously. Although Figure 4 Multiple processors 410 are shown, but the machine 400 may include a single processor 412 with a single core, a single processor 412 with multiple cores (e.g., a multi-core processor 412), multiple processors 412, 414 with a single core, multiple processors 412, 414 with multiple cores, or any combination thereof.

[0139] The memory 430 may include a main memory 432, a static memory 434, and a storage unit 436, each of which is accessible by the processor 410, such as via the bus 402. The main memory 432, the static memory 434, and the storage unit 436 store instructions 416 that embody any one or more of the methodologies or functions described herein. The instructions 416 may also reside, in whole or in part, within the main memory 432, within the static memory 434, within the storage unit 436, within at least one of the processors 410 (e.g., within a cache memory of a processor), or any suitable combination thereof during execution by the machine 400.

[0140] I / O components 450 may include a variety of components for receiving input, providing output, generating output, sending information, exchanging information, capturing measurements, etc. The specific I / O components 450 included in a particular machine will depend on the type of machine. For example, a portable machine such as a mobile phone will likely include a touch input device or other such input mechanism, while a headless server machine will likely not include such a touch input device. It should be understood that I / O components 450 may include Figure 4Many other components not shown in the figure. The I / O components 450 are grouped according to function only to simplify the following discussion, and the grouping is by no means limiting. In various example embodiments, the I / O components 450 may include output components 452 and input components 454. The output components 452 may include visual components (e.g., displays such as plasma display panels (PDPs), light emitting diode (LED) displays, liquid crystal displays (LCDs), projectors, or cathode ray tubes (CRTs)), acoustic components (e.g., speakers), tactile components (e.g., vibration motors, resistance mechanisms), other signal generators, etc. The input components 454 may include alphanumeric input components (e.g., keyboards, touch screens configured to receive alphanumeric input, optical keyboards, or other alphanumeric input components), point-based input components (e.g., mice, touch pads, trackballs, joysticks, motion sensors, or another pointing instrument), tactile input components (e.g., physical buttons, touch screens or other tactile input components that provide the location and / or force of touch or touch gestures), audio input components (e.g., microphones), etc.

[0141] In further example embodiments, the I / O component 450 may include a biometric component 456, a motion component 458, an environmental component 460, or a positioning component 462, as well as various other components. For example, the biometric component 456 may include a component for detecting expressions (e.g., hand expressions, facial expressions, voice expressions, body postures, or eye tracking), measuring biosignals (e.g., blood pressure, heart rate, body temperature, sweating, or brain waves), identifying people (e.g., voice recognition, retinal recognition, facial recognition, fingerprint recognition, or EEG-based recognition), etc. The motion component 458 may include an acceleration sensor component (e.g., an accelerometer), a gravity sensor component, a rotation sensor component (e.g., a gyroscope), etc. Environmental component 460 may include, for example, an illumination sensor component (e.g., a photometer), a temperature sensor component (e.g., one or more thermometers that detect ambient temperature), a humidity sensor component, a pressure sensor component (e.g., a barometer), an acoustic sensor component (e.g., one or more microphones that detect background noise), a proximity sensor component (e.g., an infrared sensor that detects nearby objects), a gas sensor (e.g., a gas detection sensor that detects concentrations of hazardous gases or measures pollutants in the atmosphere for safety), or other components that can provide indications, measurements, or signals corresponding to the surrounding physical environment. Positioning component 462 may include a position sensor component (e.g., a global positioning system (GPS) receiver component), an altitude sensor component (e.g., an altimeter or barometer that detects the altitude from which the air pressure can be derived), an orientation sensor component (e.g., a magnetometer), etc.

[0142] A variety of technologies may be used to implement communications. I / O components 450 may include a communication component 464 operable to couple machine 400 to network 480 or device 470 via coupling 482 and coupling 472, respectively. For example, communication component 464 may include a network interface component or another suitable device that interfaces with network 480. In other examples, communication component 464 may include a wired communication component, a wireless communication component, a cellular communication component, a near field communication (NFC) component, a Bluetooth® communication component, or a wireless communication component. Components (e.g. Low power consumption), Device 470 may be another machine or any of a variety of peripheral devices (eg, coupled via USB).

[0143] In addition, the communication component 464 can detect an identifier or include a component operable to detect an identifier. For example, the communication component 464 can include a radio frequency identification (RFID) tag reader component, an NFC smart tag detection component, an optical reader component (e.g., an optical sensor for detecting one-dimensional bar codes such as universal product codes (UPC) bar codes, multi-dimensional bar codes such as QR codes, Aztec codes, data matrix, Dataglyph, MaxiCode, PDF417, Supercode, UCC RSS-2D bar codes, and other optical codes), or an acoustic detection component (e.g., a microphone for identifying an audio signal of a tag). In addition, various information can be derived via the communication component 464, such as a location via Internet Protocol (IP) geolocation, a location via Internet Protocol (IP), ... The location of signal triangulation, the location of an NFC beacon signal via detection that can indicate a specific location, etc.

[0144] Various memories (e.g., 430, 432, 434 and / or the memory of processor 410) and / or storage unit 436 may store sets and data structures (e.g., software) of one or more instructions 416 embodying or used by any one or more of the methods or functions described herein. When executed by processor 410, these instructions (e.g., instructions 416) cause various operations to implement the disclosed embodiments.

[0145] As used herein, the terms "machine storage medium," "device storage medium," and "computer storage medium" mean the same thing and may be used interchangeably. These terms refer to a single or multiple storage devices and / or media (e.g., centralized or distributed databases and / or associated caches and servers) that store executable instructions and / or data. Thus, these terms should be considered to include, but are not limited to, solid-state memory and optical and magnetic media, including memory internal or external to the processor. Specific examples of machine storage medium, computer storage medium, and / or device storage medium include nonvolatile memory, including, for example, semiconductor memory devices such as erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), field programmable gate arrays (FPGA), and flash memory devices; magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The terms "machine storage medium," "computer storage medium," and "device storage medium" specifically exclude carrier waves, modulated data signals, and other such media, at least some of which are covered by the term "signal media" discussed below.

[0146] In various example embodiments, one or more portions of network 480 may be an ad hoc network, an intranet, an extranet, a virtual private network (VPN), a local area network (LAN), a wireless LAN (WLAN), a wide area network (WAN), a wireless WAN (WWAN), a metropolitan area network (MAN), the Internet, a portion of the Internet, a portion of a public switched telephone network (PSTN), a plain old telephone service (POTS) network, a cellular telephone network, a wireless network, 4G network, another type of network, or a combination of two or more such networks. For example, network 480 or a portion of network 480 may include a wireless or cellular network, and coupling 482 may be a code division multiple access (CDMA) connection, a global system for mobile communications (GSM) connection, or another type of cellular or wireless coupling. In this example, coupling 482 may implement any of various types of data transmission technologies, such as single carrier radio transmission technology (1xRTT), evolution data optimized (EVDO) technology, general packet radio service (GPRS) technology, enhanced data rates for GSM evolution (EDGE) technology, the third generation partnership project (3GPP) including 3G, fourth generation wireless (4G) networks, universal mobile telecommunications system (UMTS), high speed packet access (HSPA), world wide interoperability for microwave access (WiMAX), long term evolution (LTE) standards, other standards defined by various standard setting organizations, other remote protocols, or other data transmission technologies.

[0147] Instructions 416 may be sent or received over network 480 using a transmission medium via a network interface device (e.g., a network interface component included in communication component 464) and utilizing any of a number of well-known transmission protocols (e.g., HTTP). Similarly, instructions 416 may be sent or received to device 470 using a transmission medium via coupling 472 (e.g., a peer-to-peer coupling). The terms "transmission medium" and "signal medium" mean the same thing and may be used interchangeably in this disclosure. The terms "transmission medium" and "signal medium" should be deemed to include any intangible medium capable of storing, encoding, or carrying instructions 416 for execution by machine 400, and include digital or analog communication signals or other intangible media to facilitate the communication of such software. Thus, the terms "transmission medium" and "signal medium" should be deemed to include any form of modulated data signal, carrier wave, etc. The term "modulated data signal" means a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal.

[0148] The terms "machine-readable medium," "computer-readable medium," and "device-readable medium" mean the same thing and are used interchangeably in this disclosure. These terms are defined to include both machine storage media and transmission media. Thus, the term includes both storage devices / medium and carrier / modulated data signals.

Claims

1. A system comprising: at least one hardware processor; as well as A computer readable medium storing instructions, which when executed by the at least one hardware processor causes the at least one hardware processor to perform operations, the operations comprising: Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library; The user portion that converts commands into natural language prompts; Add predefined system sections to natural language prompts; sending the natural language prompt to a large language model (LLM) to generate a first piece of software code in a first programming language based on the natural language prompt; verifying the first piece of software code using a verification function in a first programming language; and Insert the first piece of software code into the software programming code.

2. The system according to claim 1, wherein: The command includes a natural language portion in which a desired programming code segment is described in a natural language, and one or more test functions.

3. The system according to claim 2, wherein: The operations also include: executing the one or more test functions on the first piece of software code; and The inserting is in response to determining that the first segment of software code is executed by all of the one or more test functions.

4. The system according to claim 1, wherein: In response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

5. The system according to claim 3, wherein: In response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

6. The system according to claim 1, wherein: The operations also include: generating a hash of the command by applying a hash function to the command; and The hash and the first piece of software code are saved in a file system cache.

7. The system according to claim 6, wherein: The operations also include: identifying a second command in the software programming code through the scripting language interface; generating a hash of a second command by applying a hash function to the command; determining whether a hash of the second command is contained in the file system cache; and In response to determining that the hash of the second command is contained in the file system cache, retrieving the first segment of software code from the file system cache and inserting the first segment of software code into the software programming code to replace the user portion that converts the command to a second natural language prompt and sends the second natural language prompt to the LLM.

8. A method comprising: Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library; The user portion that converts commands into natural language prompts; Add predefined system sections to natural language prompts; sending the natural language prompt to a large language model (LLM) to generate a first piece of software code in a first programming language based on the natural language prompt; verifying the first piece of software code using a verification function in a first programming language; as well as Insert the first piece of software code into the software programming code.

9. The method according to claim 8, wherein: The command includes a natural language portion in which a desired programming code segment is described in a natural language, and one or more test functions.

10. The method according to claim 9, further comprising: executing the one or more test functions for the first piece of software code; as well as The inserting is in response to determining that the first segment of software code is executed by all of the one or more test functions.

11. The method according to claim 8, wherein: In response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

12. The method according to claim 10, wherein: In response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

13. The method according to claim 8, further comprising: generating a hash of the command by applying a hash function to the command; as well as The hash and the first piece of software code are saved in a file system cache.

14. The method according to claim 13, further comprising: identifying a second command in the software programming code through the scripting language interface; generating a hash of a second command by applying a hash function to the command; determining whether a hash of the second command is contained in the file system cache; as well as In response to determining that the hash of the second command is contained in the file system cache, retrieving the first segment of software code from the file system cache and inserting the first segment of software code into the software programming code to replace the user portion that converts the command to a second natural language prompt and sends the second natural language prompt to the LLM.

15. A non-transitory machine-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations comprising: Identify commands in software programming code through a scripting language interface, the commands being defined in a scripting language interface library; The user portion that converts commands into natural language prompts; Add predefined system sections to natural language prompts; sending the natural language prompt to a large language model (LLM) to generate a first piece of software code in a first programming language based on the natural language prompt; verifying the first piece of software code using a verification function in a first programming language; as well as Insert the first piece of software code into the software programming code.

16. The non-transitory machine-readable medium of claim 15, wherein: The command includes a natural language portion in which a desired programming code segment is described in a natural language, and one or more test functions.

17. The non-transitory machine-readable medium of claim 16, wherein: The operations also include: executing the one or more test functions on the first piece of software code; and The inserting is in response to determining that the first segment of software code is executed by all of the one or more test functions.

18. The non-transitory machine-readable medium of claim 15, wherein: In response to determining that the first segment of software code fails the validation function, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

19. The non-transitory machine-readable medium of claim 17, wherein: In response to determining that the first segment of software code fails one or more of the one or more test functions, the first segment of software code is regenerated by resending the natural language prompt to the LLM.

20. The non-transitory machine-readable medium of claim 15, wherein: The operations also include: generating a hash of the command by applying a hash function to the command; and The hash and the first piece of software code are saved in a file system cache.

Images