Sampling large language models with equivalence verification

Abstract

A generative pre-trained large language model (LLM) may create domain-specific text answers in various formats such as JSON, XML, HTML, SQL, or programming language. However, LLM may "appear illusion", generating incorrect or meaningless answers that deviate from reality, thereby weakening trust on its output or causing more serious consequences. The disclosed techniques use a sampling-based method and an equivalence checker. Generating, by the LLM, a plurality of answers (samples) to the cue; if the plurality of answers are equivalent, the LLM is likely to answer correctly. If the samples are not consistent or contradictory, it is more likely that the LLM has illusion, or ambiguity is prompted. The functional equivalence of the sample is validated using an automatic inference equivalence verifier, which provides a method of detecting and possibly correcting hallucination questions in the LLM generated answers.

Description

Background Technology

[0001] The pre-trained Large Language Models (LLMs) are becoming increasingly prevalent. Some of these pre-trained LLMs are designed to understand and generate domain-specific machine-readable text (including domain-specific machine-executable text) given natural language cues. A challenge associated with generating domain-specific machine-readable text using pre-trained LLMs is validating the quality and accuracy of the generated content. One approach to determining the quality and correctness of the generated domain-specific machine-readable text is to evaluate it against a set of test cases. Unfortunately, manually creating test cases has proven to be both expensive and time-consuming. Attached Figure Description

[0002] Various examples according to this disclosure will be described with reference to the accompanying drawings, in which: Figure 1 Examples of systems and methods for sampling large language models using equivalence checks are provided.

[0003] Figure 2 Examples of systems and methods for sampling and cascading large language models using equivalence checks are provided.

[0004] Figure 3 An example of an equivalence checker is shown for determining whether a pair of domain-specific machine-readable texts are functionally / semantically equivalent.

[0005] Figure 4 An example of a first graphical user interface for an intelligent integrated development environment (IDE) using the disclosed technology for automated code generation is shown.

[0006] Figure 5 An example of a second graphical user interface for an intelligent integrated development environment (IDE) using the disclosed technology for automated code generation is shown.

[0007] Figure 6 An example of a third graphical user interface using an intelligent integrated development environment (IDE) employing the disclosed technologies for automated code generation is provided.

[0008] Figure 7 Examples of methods for sampling and cascading large language models using equivalence checks are provided.

[0009] Figure 8 An example of a provider network environment is illustrated, in which techniques for sampling and cascading large language models using equivalence checks are employed.

[0010] Figure 9 An example of a computing device is shown for implementing a technique of sampling and cascading large language models using equivalence checks. Detailed Implementation

[0011] This disclosure relates to systems, methods, and non-transitory computer-readable storage media (collectively, the “Technology”) for sampling large language models using equivalence checks.

[0012] Generative pre-trained large language models (LLMs) can produce highly fluent, domain-specific text responses to a wide range of natural language cues. Domain-specific text encompasses various formats, such as text structured according to data exchange standards (e.g., JavaScript Object Notation (JSON), Extensible Markup Language (XML), Hypertext Markup Language (HTML)) or similar specifications. It also includes text formatted according to query languages ​​of structured databases (e.g., SQL, SparQL, Datalog, etc.) and text written in programming languages ​​(e.g., Python, JavaScript, Java, C, C++, etc.). However, LLMs are prone to "illusions."

[0013] In the context of LLM, hallucinations arise when LLM produces inaccurate, meaningless, or unrealistic answers. These hallucinations erode trust in the LLM output and can lead to the spread of misinformation or other harm. Therefore, techniques designed to determine whether answers generated by LLM are hallucinations are highly valuable.

[0014] The technique disclosed in this paper uses a sampling-based approach combined with an equivalence checker to verify generated answers, thereby determining whether an LLM is experiencing hallucination. When an LLM is proficient in providing accurate responses to a particular cue, multiple answers (samples) generated by the LLM in response to that cue are expected to be consistent. Conversely, in cases where the LLM is hallucinating, these samples are more likely to exhibit inconsistencies, contradictions, or divergences among themselves. An automated reasoning equivalence checker is employed to determine whether the generated samples actually share equivalence.

[0015] In some instances, the LLM is sampled multiple times using the same or similar cues. An equivalence checker is then employed to determine whether the domain-specific text in the samples generated by the LLM possesses functional equivalence. If these samples lack functional equivalence, it may indicate that the LLM is experiencing illusions or that the cues themselves are ambiguous, necessitating refinement or rewriting of the cues. Conversely, when samples exhibit the functional equivalence determined by the equivalence checker, it serves as an indicator that the LLM is adept at producing consistent responses.

[0016] While in some examples equivalence validators are used to determine whether domain-specific text generated by an LLM possesses functional equivalence (e.g., exhibiting the same behavior or producing the same output for all possible inputs and states)—sometimes referred to as semantic equivalence—equivalence validators can also verify other types of equivalence or relations between domain-specific texts. For example, equivalence validators can verify syntactic or model equivalence between texts (e.g., whether texts are equivalent with respect to specified attributes) or whether one text implies another. More generally, the specific meaning of domain-specific text equivalence can vary depending on the application scenario and the requirements of the specific implementation. However, broadly speaking, equivalence exists when domain-specific texts have the same meaning within the domain to which they belong.

[0017] To illustrate the problem addressed in this context, consider a scenario where an enterprise aims to leverage LLM to generate domain-specific text for the implementation of various IT projects. However, as previously mentioned, generating such text using LLM raises concerns about the illusion that LLM might (or might not) appear accurate on the surface but ultimately result in meaningless or incorrect instances of domain-specific text. Therefore, in this context, the advantages associated with using LLM to generate text (such as reduced development time) can be offset by the costs incurred by the enterprise in establishing comprehensive quality assurance and testing mechanisms to ensure the accuracy and reliability of the LLM-generated text. The illusion induced by LLM can lead to malfunctioning code, resulting in program defects, errors, and wasted development effort. The techniques disclosed in this paper mitigate or eliminate the need for quality assurance and testing of LLM-generated domain-specific text by employing an automated reasoning (equivalence check) sampling process to distinguish between instances where LLM is experiencing illusions, hints of ambiguity, and instances where the generated text is consistent.

[0018] Figure 1 Examples of systems and methods for sampling large language models using equivalence checks are described. Figure 1The configuration presented includes a provider network, represented as "Provider Network 100." This type of network, often referred to as a "cloud" provider network, extends users' ability to utilize a wide range of compute-related resources. These resources include compute elements such as virtual machine (VM) instances or containers, batch processing jobs, and code that runs without requiring pre-configured servers. Additionally, data and storage resources are included, such as object storage, block storage, archive data storage, databases, and database tables. Network-related assets such as virtual network setups containing compute resource clusters, content delivery networks (CDNs), and domain name services (DNS) are also part of this network. Application resources used for application building and deployment, such as databases and services, are integral components. The network includes configurations for access policies, roles, identity policies, machine images, routers, and various other data processing resources, forming a comprehensive set of capabilities offered to users.

[0019] These computing resources, along with other resources, are available as services through provider network 100. These services include, for example, hardware virtualization services capable of executing computing instances and storage services designed to store data objects. Individuals utilizing provider network 100 (generally referred to as “users” or “customers”) can utilize one or more user accounts linked to a customer account. While these terms may be used interchangeably depending on the context, they collectively represent the association between a user and the provider network. Users can interact with provider network 100 through an intermediate network 102 (which may be the Internet) using various interfaces. These interfaces include interaction via application programming interface (API) calls, or using a console implemented as a website or application, among others.

[0020] An API (Application Programming Interface) represents an interface or communication protocol that facilitates interaction between a client and a server. When a client submits a request in a predefined format, it expects to receive a response in a specific format or trigger a predefined action. In the context of Provider Network 100, the API acts as a gateway, enabling customers to access the infrastructure and resources within the network. This access allows customers to retrieve data from or initiate actions within Provider Network 100, thereby facilitating the creation of applications that interact with the resources and services hosted within the network. The API also facilitates data exchange between various services within Provider Network 100. These APIs can form integrated components of the control plane of Provider Network 100 or act as front-end operations of that control plane. The control plane includes “back-end” services that provide support and enable more direct delivery of services to customers.

[0021] For example, Provider Network 100 typically refers to a broad repository of accessible virtualized computing resources, encompassing compute, storage, network resources, applications, and services. Provider Network 100 facilitates streamlined, on-demand network access to a shared pool of configurable computing resources. These resources can be programmatically allocated and released based on client instructions. The flexibility of these resources allows them to be dynamically provisioned and reconfigured to adapt to changing workloads. In this way, Provider Network 100 can be viewed as facilitating the delivery of services by encompassing both applications delivered as services on publicly accessible networks (e.g., the Internet, cellular communication networks) and hardware and software located within data centers.

[0022] Provider network 100 provides an automated reasoning service, referred to as "Automated Reasoning Service 104". This service utilizes automated reasoning technologies to solve problems and facilitate automated decision-making. These technologies encompass algorithms and systems that excel at deriving logical reasoning, solving complex problems, and autonomously executing decision-making functions. The provision of Automated Reasoning Service 104 is extended to other services within provider network 100 via API 106. This integration allows these services to embed automated reasoning capabilities into their own applications and systems, eliminating the need for them to build basic automated reasoning technologies from scratch.

[0023] Automated reasoning service 104 can be used in a variety of applications, such as: using formal methods and logical reasoning to verify the accuracy of mathematical theorems or logical statements (e.g., theorem proving); evaluating and validating the behavior of hardware or software systems against specified attributes or requirements (e.g., model validation); developing plans or sequences of actions to achieve specific purposes or goals in dynamic and uncertain environments (e.g., automated planning); identifying solutions to problems involving constraints and variables—such as scheduling, resource allocation, or configuration problems (e.g., constraint solving); using formal logic systems to derive conclusions or solve queries based on provided facts and rules (e.g., logic-based reasoning); systematically building and processing knowledge to provide answers and make decisions (e.g., knowledge representation and reasoning); and other suitable applications.

[0024] Within the framework of the automated reasoning service 104, there exists an equivalence checker 108. The equivalence checker 108 is structured to determine whether a given pair of domain-specific texts is functionally / semantically equivalent. For illustration, this pair of texts might consist of two code samples associated with the same prompt from a large language model. The equivalence checker 108 undertakes the task of verifying whether these two code samples produce the same output on all possible inputs.

[0025] The term "domain-specific text" is intended to include text formatted in a domain-specific language (DSL). A domain-specific language (DSL) can be a programming language or specification language designed and tailored to solve problems in a specific domain, industry, or problem space. For example, domain-specific text may be formatted as JSON, XML, HTML, or other markup languages; formatted as SQL or other database query languages; or formatted as Python, Java, JavaScript, or other programming languages. However, it is not required that domain-specific text be formatted in a standard or well-known domain-specific language, and domain-specific text may include text formatted in a proprietary or single-purpose domain-specific language, or a domain-specific language designed for a single application. Domain-specific text includes domain-specific machine-readable text. Domain-specific machine-readable text includes domain-specific text that is structured and formatted in a way that is easily understood by machines, particularly computers and automated systems. This type of text is designed to be processed, analyzed, and interpreted by software applications faster and more accurately than natural language text.

[0026] While there is no precise boundary between domain-specific machine-readable text and natural language text, domain-specific machine-readable text is designed for automated processing and is typically structured using specific formats, while natural language text is geared towards human communication and contains rich meanings that require sophisticated tools and techniques for computer-based understanding. Natural language text is generally more ambiguous than domain-specific machine-readable text, requiring contextual understanding based on context or cultural knowledge for proper interpretation. Requests to generate domain-specific machine-readable text for submission to a large language model can be formatted as natural language text, domain-specific machine-readable text, or a combination of both.

[0027] The equivalence checker 108 can employ various automated reasoning techniques to analyze the pair of domain-specific texts and explore all possible scenarios to ensure exhaustive and systematic functional / semantic equivalence. In some examples, the equivalence checker 108 employs formal methods. Formal methods include using mathematical techniques such as symbolic model checking, theorem proving, Boolean satisfiability (SAT) solvers, or satisfiability modular theory (SMT) solvers to formally verify equivalence. Additionally or alternatively, formal methods may involve the use of a rewriting system. For example, a set of semantically preserving rewriting rules for obtaining valid transformations of the domain-specific texts can be applied to the texts to convert them into a normalized form. The normalized forms can be compared to determine whether the domain-specific texts are functionally / semantically equivalent. For example, if the domain-specific texts have the same normalized form, they can be determined to be functionally / semantically equivalent.

[0028] In some examples, the equivalence checker 108 encodes the pair of domain-specific texts into a logical formula. The equivalence checker can then determine whether the pair of domain-specific texts is functionally / semantically equivalent by applying an automated theorem prover to the logical formula. Furthermore, the equivalence checker 108 can utilize model validation techniques to verify functional / semantic equivalence. Through model validation, the equivalence checker 108 can systematically explore the state space of the pair of domain-specific texts to verify their functional / semantic equivalence.

[0029] While the examples in this paper discuss pairwise comparisons of domain-specific texts for functional / semantic equivalence, the technique is not limited to such pairwise comparisons. For example, given a set of domain-specific texts, each text can be converted to a canonical or normalized form with the aim of capturing the functional essence of the text and removing irrelevant details or variability. The canonical or normalized forms can then be compared with each other. For example, if all canonical or normalized forms of the domain-specific texts are identical or equivalent, they can be considered functionally / semantically equivalent.

[0030] LLM Service 114 is a cloud-based or online service that grants access to the corresponding Large Language Model (LLM) 116. LLM 116 has been trained on large amounts of text data and can perform a wide variety of natural language processing tasks, including but not limited to text completion, language translation, question answering, and sentiment analysis. LLM Service 114 is available on the web (e.g., intermediate network 102) through its corresponding API 118. API 118 enables application developers to seamlessly integrate the capabilities of LLM 116 into their own applications, products, or services, eliminating the need for them to host and train LLM 116 themselves.

[0031] LLM 116 can be categorized as a type of artificial intelligence model designed to understand and generate text based on large amounts of training text data. LLM 116 is likely a deep learning model, possibly employing a Transformer architecture, and possesses the ability to process and generate natural language text information. It may contain a large number of model parameters, potentially reaching billions. Generally, a larger number of parameters tends to improve the performance of LLM 116 in generating accurate text across a wide range of prompts. However, in some cases, LLM 116 with a smaller number of model parameters has the potential to generate responses that are as accurate as, or even more accurate than, those of LLMs with a larger number of model parameters. However, it is important to note that more parameters correspond to increased cost, environmental impact, and energy consumption associated with LLM 116.

[0032] LLM prompt 126 can be transmitted from client device 122 located within client network 120 to API 112 of LLM sampling service 110. Client device 122 may take the form of a personal computing device (such as a desktop computer, laptop computer, tablet computer, or mobile phone) equipped with client application 124. Client application 124 facilitates the delivery of LLM prompt 126 and may provide support for graphical user interface (GUI), command-line interface (CLI), or even software development kit (SDK) to facilitate the submission of LLM prompt 126. As an example, client application 124 may manifest as a web application, web browser application, mobile application, or another software program executing on client device 122.

[0033] The customer network 120 may include various configurations, including a local area network (LAN), a virtual private network (VPN), or any suitable form of data communication network, through which the customer device 122 can exchange data with the provider network 100 via an intermediate network 102 (such as the Internet).

[0034] Figure 1 The method is depicted by numbered circles superimposed on directed arrows. The direction of the arrows indicates the direction of data flow, but is not necessarily the only direction. The numbers in the circles are merely markers for the various operations of the method. These numbers do not imply exclusive sorting operations, or that operations cannot overlap during execution, or cannot be executed simultaneously or in parallel.

[0035] During Operation 1, prompt 126 is dispatched from the client network 120, then transmitted via intermediate network 102, and finally received by the sampled LLM service 110 via API 112. In some instances, prompt 126 embodies a natural language specification designed to generate domain-specific text by the LLM.

[0036] In some scenarios, the domain-specific text generated by the LLM according to Hint 126 will belong to a specific domain, including: (1) text-based structured data exchange formats; (2) text-based structured database query languages; or (3) code expressed in a specific programming language. Exemplary examples of text-based structured data exchange formats that Hint 126 may request include JavaScript Object Notation (JSON), Extensible Markup Language (XML), and Hypertext Markup Language (HTML). To provide a simple example, Hint 126 may instruct the LLM to generate JSON data representing an employee roster, including a list of names, ages, departments, and assigned items. As another example, Hint 126 may instruct the LLM to generate statements directly in logic, such as propositional logic, first-order logic, or modal logic. For example, Hint 126 might ask the LLM to “express the statement ‘If it is raining, then the ground is wet’ in propositional logic.” In this case, the LLM might generate domain-specific text such as “Let p represent ‘It is raining’ and q represent ‘The ground is wet’.” This statement could be expressed as: p → q.

[0037] Furthermore, the types of text hints that domain-specific text may involve extend to Structured Database Query Language (SQL). For example, hint 126 could instruct the LLM to generate a Structured Query Language (SQL) query that retrieves the names and ages of employees over 30 years old who belong to the Sales department from the Employees table.

[0038] Another example exists in the realm of programming languages, where hint 126 might require code generation for languages ​​such as Python, Java, C, C++, JavaScript, C#, PHP, Swift, Ruby, Go, R, Kotlin, TypeScript, etc. As a simple example, hint 126 might instruct an LLM to generate Python code designed to calculate the factorial of a positive integer 'n'.

[0039] During operation 2, sampling LLM service 110 samples LLM 116 multiple times using prompt 128. This process includes dispatching prompt 128 to API 118 of LLM service 114 via intermediate network 102, and subsequently retrieving one or more sample answers generated by LLM 116 and provided by LLM service 114 via the same intermediate network 102. The act of sampling LLM 116 can include various steps, such as transmitting prompt 128 and then receiving the corresponding sample answer from LLM service 114.

[0040] Hint 128 may involve requesting LLM 116 to generate multiple sample answers, thereby generating a series of samples. Alternatively, the method may include sending hint 128 to LLM service 114 multiple times, each occurrence causing LLM 116 to generate sample answers, and subsequently returning the sample answers from LLM service 114 in response to hint 128.

[0041] Hint 128 can be mirrored from or derived from hint 126. As an example, hint 128 can represent a preprocessing iteration of hint 126. Within the sampling LLM service 110, a preprocessing engine can be configured to perform the transformation from hint 126 to hint 128. This preprocessing engine may be able to perform one or more optimizations on hint 126, thereby causing the creation of hint 128.

[0042] These optimizations can include incorporating relevant details (such as variable names, function names, input / output requirements, and expected behavior) into the prompt 128. Furthermore, the preprocessing engine can combine the input and its corresponding expected output within the prompt 128. This inclusion enhances the LLM 116's understanding of the sought functionality and helps generate domain-specific text capable of producing the expected results.

[0043] Furthermore, the preprocessing engine can incorporate information relevant to a specific question, domain, or scenario into the hints 128. This enhancement is designed to guide LLM 116 to align its answer generation with the context, resulting in domain-specific text that serves its intended purpose. Additionally, the preprocessing engine can directly incorporate coding style conventions, such as indentation or naming conventions, into the hints 128.

[0044] Finally, the preprocessing engine can embed the specification of the "temperature" parameter of the LLM 116 in prompt 128. This temperature parameter plays a role in controlling the randomness of the LLM 116 output and may affect the degree of variability present in the resulting responses.

[0045] The "temperature" parameter is used in some Large Language Models (LLMs) to configure the output randomness during text generation. Its role is to guide the LLM's decision-making when selecting the next word or token during the text generation process. Increasing the value assigned to the temperature parameter introduces greater diversity and creativity into the generated output. This encourages the LLM to explore a wider range of possibilities, resulting in text exhibiting more pronounced variations. However, this high degree of diversity can also translate into occasionally more random and inconsistent outputs.

[0046] Conversely, lower temperature values ​​will lead to more focused and deterministic results. In such cases, LLMs tend to generate more predictable and controllable text by supporting the most likely next lexical term based on their training data.

[0047] In certain scenarios, when the preprocessing engine incorporates temperature parameter values ​​into cue 128, a lower value is used (e.g., ranging from 0.2 to 0.5, inclusive). This selection is used to guide LLM 116 to produce domain-specific text that is inherently more focused and deterministic.

[0048] In some instances, the LLM 116 uses the same cue 128 for multiple samplings, resulting in a variety of sample answers. Conversely, in other cases, the LLM 116 is sampled multiple times using slightly modified cue 128. As an example, when the LLM 116 is sampled with cue 128, one or more cue 128 used to derive sample answers from the LLM 116 may exhibit minor variations from cue 128. These variations are strategically introduced to avoid affecting the functional / semantic equivalence of subsequent sample answers. If the sample answers generated by the LLM 116 for these subtly different cue 128 maintain functional / semantic equivalence, this underscores the LLM 116's ability to generate consistent responses. This result highlights the LLM 116's ability to perceive subtle deviations in cue 128 without compromising the functional integrity of sample answers.

[0049] To illustrate this concept, consider a scenario where hint 126 asks for the creation of a Python function to compute the factorial of a non-negative integer 'n'. In this case, for three different hints, three distinct sample answers can be drawn from LLM 116. These hints may differ in that they request the generation of a Python function to compute the factorial of 'n' using recursion, iteration (e.g., employing a for loop or while loop), or the math.factorial function from the Python standard library. Despite these different hints, the expectation remains that all three sampled answers are functionally / semantically equivalent, thus producing consistent results on all feasible inputs. If LLM 116 does indeed successfully produce functionally / semantically equivalent sample answers for these different hints—hints that deviate in ways unrelated to functional / semantic equivalence—it suggests that LLM 116 likely does not have an illusory response to the current hint 126.

[0050] Additionally, in operation 2, LLM service 114 provides sample answers 130 to sampling LLM service 110. If multiple prompts are used for the purpose of sampling LLM 116, each of these sample answers may contain domain-specific text generated by LLM 116 in direct response to prompt 128 or one of the prompts in prompt 128.

[0051] In operation 3, domain-specific text 132 extracted from sample answer 130 is transmitted to the automated reasoning service 104 via API 106. For each provided text pair, the automated reasoning service 104 triggers an equivalence checker 108 to determine the functional / semantic equivalence of the text pair. The result of this evaluation provided by the equivalence checker 108 constitutes a determination for a given text pair. This determination indicates whether the two texts in the text pair are functionally / semantically equivalent. Furthermore, if the determination indicates that the text pair is not functionally / semantically equivalent, the result may include a "witness." This "witness" may include exact assignments to variables within the text of the text pair, effectively proving that they lack functional / semantic equivalence. For example, in the case where the two texts represent different programs within a programming language, the determination may include assignments to program inputs, resulting in different outputs determined by the equivalence checker 108. A witness may include a counterexample to the functional / semantic equivalence of the two texts. For example, a counterexample could include a specific assignment of the input, which the equivalence checker 108 determines will cause the text to produce different outputs when it is executed or otherwise processed.

[0052] In some scenarios, when sample answer 130 contains more than two domain-specific texts, a comparison process is performed. The texts within sample answer 130 can be individually evaluated for functional / semantic equivalence against another text in sample answer 130, or the comparison can occur in pairs.

[0053] For example, when sample answer 130 includes four texts (A, B, C, D), assuming text A is used as a reference, equivalence checker 108 can be invoked to determine the functional / semantic equivalence of text pairs (A, B), (A, C), and (A, D). For each such text pair, equivalence checker 108 generates findings 134.

[0054] Furthermore, if the text of text 132 is determined by the equivalence checker 108 to be functionally / semantically inequivalent, the equivalence checker 108 may abandon the evaluation of the functional / semantic equivalence of any remaining text of text 132. It is important to note that, depending on the specific implementation, the evaluation of text 132 by the equivalence checker 108 can be performed simultaneously or in parallel.

[0055] Moving to operation 4, guided by discovery 134, the sampling LLM service 110 determines the response 136 to be delivered. These findings 134 (possibly one for each pair of texts evaluated in text 132) summarize the results generated by the equivalence checker 108.

[0056] In some instances, if at least one pair of texts in the discovery indication text 132 in discovery 134 lacks functional / semantic equivalence, response 136 may incorporate a witness provided by equivalence validator 108. This witness may be shared with the user of client device 122 via client application 124. The intention here is to help the user refine or restate the initial prompt 126. Subsequently, the refined or restated prompt, or a prompt derived from it, may be resubmitted to the sampled LLM service 110, thereby facilitating new attempts with the enhanced prompt.

[0057] In contrast, when discovery 134 indicates that all texts 132 are functionally / semantically equivalent, response 136 may contain one or more texts from text 132. The text integrated into response 136 can be displayed to the user of client device 122 via client application 124. For example, the text may be displayed in the graphical user interface (GUI) of an integrated development environment (IDE) application.

[0058] In some scenarios, the equivalence checker 108 is invoked to verify the functional / semantic equivalence of a set of domain-specific texts. To achieve this, the equivalence checker 108 has the ability to transform both texts into corresponding logical formulas. These logical formulas serve as a structured representation of the text, which is easily understood by automated theorem provers such as SMT solvers. This transformation process may include the equivalence checker 108 encoding different aspects (such as control flow, data manipulation, input / output behavior, and other logical attributes) into logical constraints within a formulaic logical formula.

[0059] As an example, text can be transformed into various logical frameworks, including propositional logic, first-order logic, arithmetic, arrays, bit vectors, and other theories used in formal verification and automated reasoning, all conforming to the Satisfiability Modulus Theory Library (SMT-LIB) format and language. SMT-LIB represents a standardized format and language designed to describe logical theories and formulas intended for use by SMT solvers. This standardized format provides a unified platform, enabling users to express logical problems in a consistent and transferable manner, thereby promoting interchangeability between problems and solvers.

[0060] In some instances, domain-specific text undergoes a transformation into a logical framework embedded within the SMT-LIB language. This framework may include undefined uninterpreted functions (QF_UF), undefined arrays and bit vectors (QF_AUFBV), undefined linear integer arithmetic (QF_LIA), undefined linear real arithmetic (QF_LRA), or any other applicable SMT-LIB logic. The choice of which specific logic to employ can vary based on unique considerations for the particular implementation under discussion. This includes factors such as the specific preconditions of the current problem and the elements contained in the logical formulas.

[0061] Once the equivalence checker 108 has encoded the pair of domain-specific texts, it begins to formulate an equivalence query. In some scenarios, this equivalence query takes the form of a logical formula that negates the equivalence between the two texts it encodes. More simply, the equivalence query confirms that the two texts lack functional / semantic equivalence, and it presents the query to the SMT solver, asking if a satisfactory assignment can be identified for the query.

[0062] If the SMT solver does indeed locate a satisfactory assignment for the query, it confirms that the two texts are functionally / semantically equivalent. The located satisfactory assignment can be provided as evidence in the findings returned by the SMT solver. Conversely, if the SMT solver deems the equivalence query unsatisfactory, this result indicates that the two texts are indeed functionally / semantically equivalent.

[0063] In some instances, the equivalence checker 108 triggers the SMT solver action by providing a formulaic query as input. The SMT solver undertakes an exhaustive search for satisfactory assignments, aiming to establish the state of the equivalence query—whether it is satisfiable (indicating that the two texts are not functionally / semantically equivalent) or unsatisfiable (implying that the two texts are functionally / semantically equivalent).

[0064] In some scenarios, the findings 134 returned by the equivalence checker 108 convey whether an equivalence query is determined to be satisfiable or unsatisfiable. When an equivalence query is determined to be unsatisfiable, it confirms the functional / semantic equivalence of the two texts. Conversely, when an equivalence query is identified as satisfiable, the findings may contain a satisfactory assignment serving as evidence of non-equivalence, often referred to as a "witness."

[0065] However, in certain scenarios, the equivalence checker 108 employs alternative automated reasoning tools, different from the SMT solver, to evaluate the functional / semantic equivalence between two texts. This approach can be used independently or as a supplementary method alongside the SMT solver. For example, the equivalence checker 108 may employ a symbolic model checker or other similar semi-automatic or fully automatic techniques specifically designed for equivalence verification.

[0066] In some examples, the equivalence checker 108 is a sequential equivalence checker or a combinatorial equivalence checker. In both cases, the equivalence checker 108 verifies whether two different domain-specific texts are functionally / semantically equivalent. In other words, for the same input, the two texts produce the same output when executed, interpreted, or otherwise processed. It is important to note that functional / semantic equivalence does not require the two texts to implement the function in the same way. In fact, the two texts can be sampled from one or more LLMs and use different corresponding hints designed to implement the function in different ways. For example, one hint might request a function to recursively compute the factorial of 'n', and another hint might request a function to iteratively compute the factorial of 'n'. The two hints can still be functionally / semantically equivalent even if they implement the function in different ways.

[0067] For the sequential equivalence checker 108, the checker 108 can first obtain a sequential logical or functional formal representation of the domain-specific text. For example, the text can be transformed into a state machine (where states represent conditions or stages), and the transformations are driven by rules or instructions contained in the text. The checker 108 can also identify inputs and outputs, where the inputs are the conditions or variables operated on by the text, and the outputs are the resulting actions or conclusions. Specifically, the checker 108 can verify that the texts have the same set of input and output parameters. The checker 108 can define how the state system of the text moves from one state to another based on the inputs and the current state. The checker 108 can then compare the state machines representing the two texts, verifying all possible sequences of inputs and states to ensure that for the same sequence of inputs, both state machines reach equivalent states and produce the same outputs. For large state machines, the checker 108 can use techniques such as abstraction, bidirectional simulation, or symbolic execution to make the equivalence verification problem more tractable. If the checker 108 finds that the two state machines are equivalent, it can produce the finding that the two texts are functionally / semantically equivalent. If not, the validator 108 can generate a discovery that identifies the location of the difference, which can help the user or an automated process determine whether the functional inconsistency between the two texts is due to an ambiguous prompt or due to an illusionary LLM.

[0068] For the combinatorial equivalence checker 108, the checker 108 can analyze two texts by converting them into mathematical representations of the logic described by the texts. By comparing these representations, the checker 108 verifies that the texts describe the same relationship between specific inputs and outputs, regardless of sequences or states, thus verifying that the texts are functionally / semantically equivalent even if they are expressed or structured differently. The checker 108 can compile or convert the texts into forms that capture the described logic or functionality. For example, the checker 108 can convert the texts into logical expressions, logical formulas, or other mathematical representations detailing the relationship between inputs and outputs. Additionally or alternatively, the checker 108 can convert the texts into combinatorial logic representations (such as Boolean functions) that capture the relationship between inputs and outputs. For example, this could be a direct mapping from each possible combination of inputs to the corresponding output. The checker 108 can use methods such as binary decision graphs (BDD), Boolean satisfiability (SAT) solvers, or satisfiability modular theory (SMT) solvers to compare the combinatorial representations of the two texts. For large or complex texts, direct comparison can be computationally difficult. In such cases, advanced techniques and heuristics may be needed to efficiently determine equivalence. If two texts are found to describe the same logic, regardless of order, performance, or timing, then validator 108 can declare the two texts to be functionally / semantically equivalent. Otherwise, validator 108 can find that they are not functionally / semantically equivalent and report the difference between the two texts (e.g., witness).

[0069] Figure 2 An exemplary system and method are illustrated for performing both sampling and concatenation operations on large language models while incorporating equivalence checks. In addition to its ability to sample LLMs, service 110 is also configured to concatenate these LLMs. The financial implications, environmental considerations, and energy consumption associated with obtaining accurate outputs from large language model (LLM) services are significant to the practical value of such services. Unfortunately, it is often difficult to discern in advance which specific LLM service among a set of available alternatives will provide an accurate response at a reasonable cost. Considering cost factors, this underscores the necessity of techniques capable of optimizing the selection of one or more LLM services.

[0070] This requirement for selecting a technology is linked to the necessity of verifying the correctness of the generated response.

[0071] In some instances, service 110 provides a cost-optimized method for selecting LLM services, while employing an equivalence checker 108 to validate the generated answers. In some scenarios, the LLM service arrays 114-1, 114-2, ..., 114-M are ordered in ascending order based on their costs. When faced with a task to process prompt 126, the most economical LLM service candidate will be selected according to... Figure 1 The process shown is used for sampling. This operation produces sample 130 from the selected LLM service in response to prompt 126.

[0072] The equivalence checker 108 then takes action to evaluate the functional / semantic equivalence of the samples 130. If they fail to demonstrate functional / semantic equivalence, the next step requires moving to the next candidate LLM service, characterized by a slightly higher cost. The method for generating samples from this subsequent candidate LLM service reflects... Figure 1 The process outlined in the document is as follows. Subsequently, the equivalence checker 108 is used again to determine whether the new sample set exhibits functional / semantic equivalence.

[0073] This iterative sampling sequence, focused on an ordered array of candidate LLM services, can continue until the equivalence checker 108 verifies the functional / semantic equivalence of samples from a given LLM service, or until a predefined stopping condition is met, such as reaching or exceeding a predetermined cost threshold. If the lower-cost LLM service does successfully generate functionally / semantically equivalent samples, this approach can result in significant cost savings compared to directly using a higher-cost LLM service from the outset.

[0074] By illustrating the issue under discussion, consider a scenario involving high-performance LLM services that charge based on the scope of the prompts, the length of the generated answers, and a fixed fee for utilizing their application programming interface (API). For generating top-quality text, businesses might prefer to resort to high-performance LLM services. However, there is an opportunity to reduce costs if more economical LLM services can deliver the required level of text at a lower expense.

[0075] In some instances, service 110 introduces a method characterized by the economical and certified utilization of LLM services. This method includes a cascading strategy, where, for a specific prompt, one or more services are systematically sampled sequentially from an array of available LLM service options (114-1, 114-2, ..., 114-M). After the sample text generated by the LLM services is validated by equivalence validator 108, those sample texts confirmed to be functionally / semantically equivalent form the basis for generating the answer to prompt 126. In such cases, no additional LLM services need to be invoked to process prompt 126. Subsequent LLM services are sampled only if an earlier sampled LLM service produces sample text that equivalence validator 108 does not determine to be functionally / semantically equivalent.

[0076] When applied to a large number of prompts, this method has the potential to significantly reduce the cost of LLM services. Obtaining validated answers for a large number of prompts with minimal or even more economical LLM services can result in significant cost savings. Furthermore, the implementation of this method also has positive implications for environmental and energy considerations.

[0077] In some scenarios, service 110 undertakes its operation in operation 1 by receiving a set of Large Language Model (LLM) hints. This set of hints may contain various types of hints, including those involving code generation. Each of these hints contains a natural language expression or another high-level code specification designed for processing by the LLM services to generate queries using a domain-specific language or an alternative lower-level language (e.g., Structured Query Language (SQL)). Once obtained, this set of hints is dispatched by service 110 to a series of LLM services, causing these LLM services to generate corresponding answers to the hints.

[0078] As previously indicated, for each prompt, service 110 may employ an iterative sampling method involving one or more LLM services in operation 2. This sequential process continues until one of the LLM services returns a set of samples exhibiting functional / semantic equivalence (operation 3), or until a predetermined stopping criterion is met. The stopping criterion may include factors such as a cost budget allocated to the LLM prompts or the overall set of prompts, and only if the predefined budget is not exceeded will another LLM service be employed.

[0079] In order to determine the functional / semantic equivalence of sample pairs generated from the LLM service in operation 3, service 110 utilizes the equivalence checker 108 within the automatic reasoning service 104 via API 106.

[0080] During operation 3, if discovery 134 of domain-specific text 132 regarding sample answer 130 confirms the functional / semantic equivalence of the domain-specific text, service 110 proceeds to operation 4. In operation 4, the service initiates the provision of response 136. Response 136 includes the action of returning one or more samples 130 or one or more texts 132 extracted from these samples.

[0081] Conversely, at operation 3, when discovery 134 regarding text 132 determines that the text does not have functional / semantic equivalence, service 110 adopts an alternative approach. It continues to systematically sample subsequent LLM services in a sequential manner, including entities such as LLM services 114-2, ..., 114-M.

[0082] During operation 3, equivalence validator 108 determines that text 132 is functionally / semantically equivalent and relays the finding 134 back to service 110. In operation 4, if equivalence validator 108 verifies the functional / semantic equivalence of one or more texts in text 132, response 136 may include those one or more texts. Alternatively, if equivalence validator 108 determines that text 132 is not functionally / semantically equivalent, a different processing path will be initiated.

[0083] In this scenario, sampling of the remaining LLM services continues systematically and sequentially until any of the services returns a set of samples that are determined to be functionally / semantically equivalent, or a predetermined stopping criterion is met. An example of such a stopping criterion could be a budget constraint, such as a predefined cost limit. For instance, a customer might have a specified monthly or daily cost budget or per-query cost budget associated with service 110. This budget serves as an upper limit on the customer's expenses incurred during a specified period by sampling LLM services 114-1, 114-2, ..., 114-M using customer-provided prompts.

[0084] If sampling another LLM service would result in exceeding the budgeted time period, or if the budget has already been exceeded, service 110 may cease sampling LLM services 114-1, 114-2, ..., 114-M for the remainder of the budgeted time period. When the budgeted time period expires and new budget becomes available for subsequent periods, service 110 may restart the sampling process for LLM services 114-1, 114-2, ..., 114-M using prompts provided by the customer. The use of cost budgets proves particularly appropriate when customers submit fluctuating or unpredictable amounts of prompts to service 110 on a daily, weekly, monthly, or otherwise defined basis. For example, a certain amount of prompts may be submitted to service 110 during off-peak times (e.g., early morning) when computation costs are lower.

[0085] LLM services 114-1, 114-2, ..., 114-M demonstrate different performance and cost characteristics, as well as varying advantages and disadvantages. Therefore, making informed choices of the appropriate LLM service from this group (114-1, 114-2, ..., 114-M) can yield benefits such as reduced costs, enhanced performance, and decreased energy consumption and environmental impact. It should be noted that while the examples discussed sampling and cascading across multiple LLM services 114-1, 114-2, ..., 114-M, this technique can be equivalently applied to sampling the same LLM or the same LLM service with multiple different configurations, each chosen for a specific performance-cost trade-off. For example, three different configurations could be used: one offering high performance and high cost, another offering balanced performance and balanced cost, and a third offering low performance and low cost.

[0086] Service 110 samples an array of LLM services (114-1, 114-2, ..., 114-M) by sequentially dispatching prompts received via API 112. If any of these LLM services (114-1, 114-2, ..., 114-M) returns a set of sample answers verified by equivalence checker 108 to be functionally / semantically equivalent, then one or more of these sample answers can constitute a response to the prompt. Subsequently, it is not necessary to sample additional LLM services from this set (114-1, 114-2, ..., 114-M).

[0087] In some instances, if a set (or multiple sets) of sample answers received from previously invoked LLM services (114-1, 114-2, ..., 114-M) fails the equivalence checker 108’s verification for functional / semantic equivalence, service 110 exclusively forwards the prompt to the remaining LLM services (114-1, 114-2, ..., 114-M).

[0088] Service 110 has the ability to cascade a given prompt across various LLM services—namely, LLM services 114-1, 114-2, ..., 114-M—in different orders. One method of ordering the LLM services (114-1, 114-2, ..., 114-M) involves arranging them based on their associated costs, organized sequentially from lowest to highest cost. For each given prompt, the cost associated with each of these LLM services (114-1, 114-2, ..., 114-M) can depend on one or more factors, such as the lexical length of the prompt, the lexical length of the generated answer, and a fixed cost. Therefore, the LLM services (114-1, 114-2, ..., 114-M) can be categorized according to various metrics, including the average cost per prompt, the mean cost per prompt, or any other relevant cost measure per prompt.

[0089] During the cascading process, the initially sampled LLM may provide a set of sample answers that are subsequently determined to lack functional / semantic equivalence. This situation raises the question of whether the lack of functional / semantic equivalence in the sample answers can be attributed to: (1) the inherent ambiguity of the cue; or (2) the LLM's inability to respond accurately to the cue (e.g., exhibiting hallucinatory behavior). In response to this challenge, Service 110 can implement a range of strategies to address the potential concerns.

[0090] One strategy involves employing a machine learning model, such as a regression model, designed to distinguish between two scenarios: (1) the need to refine or rewrite the cue, followed by resampling the same LLM with the modified cue; or (2) advancing the cascading process to different LLMs more likely to produce consistent responses. Using supervised learning methods, this model can be trained on a dataset containing both ambiguous and unambiguous cuees. Once trained, the model can perform classification to determine whether a given cue is likely ambiguous.

[0091] If the evaluation classifies a given prompt as ambiguous, it indicates that the prompt needs refinement or rewriting before being resubmitted to the same LLM. Service 110 can prompt the user to undertake this task, or alternatively, the service itself can undertake the automatic refinement or rewriting of the prompt. In some instances, Service 110 can leverage findings generated by an equivalence checker to determine the best approach to refine or rewrite the prompt. Specifically, witnesses provided in the findings can guide Service 110 in this regard. For example, a witness might highlight ambiguity in the prompt. In cases where user involvement is required to refine or rewrite the prompt, Service 110 can provide witnesses as a reference, enabling the user to detect and correct any ambiguity while refining or rewriting the prompt.

[0092] Conversely, if the model determines that a given prompt is unambiguous, Service 110 can automatically cascade the prompt to another LLM, or ask the user if they wish to initiate such a cascade.

[0093] Alternatively, supervised learning can be used to train the model on a dataset containing both high-quality and low-quality answers, thus eliminating the need for a specific corpus of ambiguous or unambiguous prompts. In this scenario, when a set of sample answers from an LLM lacks functional / semantic equivalence, the model can classify one or more of these sample answers based on their quality. If a sample answer is evaluated as high-quality despite lacking functional / semantic equivalence, it indicates ambiguity in the prompt. Conversely, low-quality answers suggest that the LLM may struggle to respond consistently to prompts.

[0094] In the case of a high-quality answer, Service 110 can prompt the user to refine or rewrite the suggestion, or perform an automated refinement or rewriting process. Afterward, the same LLM can be retried using the improved suggestion. In the case of a low-quality answer, Service 110 can ask the user about their preference for cascading the suggestion to another LLM, or the service can automatically cascade the suggestion to an alternative LLM that is more likely to produce a clear response.

[0095] In some instances, when the LLM generates a set of sample answers lacking functional / semantic equivalence, service 110 can provide a witness to the user. This witness (also known as a counterexample) serves as evidence that there is no functional / semantic equivalence between two domain-specific texts. In this context, equivalence validator 108 can provide a "witness" to this scenario of lack of equivalence.

[0096] In this context, a witness is a specific input or set of inputs that, when input into two texts and executed as a program or otherwise evaluated, produces different outputs. The existence of a witness provides concrete proof that two texts do not share functional / semantic equivalence. For example, a witness could include assigning a value or set of values ​​to a parameter or set of parameters within the corresponding functions described by the two texts in a programming language. The equivalence checker 108 determines that the value or set of values ​​will produce different outputs when executed, thus demonstrating a lack of functional / semantic equivalence between the two texts.

[0097] In certain scenarios, when an LLM generates a set of sample answers lacking functional / semantic equivalence, service 110 can provide a witness provided by equivalence validator 108. This witness, along with two texts lacking equivalence, can be presented to the user. The user can then review the witness and texts to determine if the provided hints are more likely to be ambiguous or if a more capable LLM is needed to provide the correct response. After validating the witness and texts, the user can choose to refine or rewrite the hints and retry the same LLM. Alternatively, the user can request service 110 to cascade the hints to another LLM.

[0098] In some instances, if the prompt has been refined or rewritten multiple times (e.g., more than twice), and the same LLM consistently produces sample answers that lack functional / semantic equivalence in each case, this may indicate that the LLM is insufficient to generate consistent responses. If such a pattern occurs, service 110 may ask the user if they wish to continue cascading to another LLM.

[0099] In a specific scenario, if a prompt has been cascaded across multiple LLMs and none of the sampled LLMs produces a functionally / semantically equivalent response, service 110 can notify the user that the prompt is ambiguous and request them to enhance and restate the prompt. Subsequently, after the user refines and restates the prompt, service 110 can reset the cascading for the modified prompt, starting with the initial LLM in the sequence (e.g., the lowest-cost LLM). This process is performed if the refinement or rewriting of the prompt resolves the ambiguity, enabling the low-cost LLM to provide a consistent response to the modified prompt.

[0100] Figure 3 An equivalence checker designed to establish the functional / semantic equivalence of a pair of domain-specific texts is presented. The equivalence checker 108 accepts a pair of domain-specific texts as input, which is referred to as "first domain-specific text" or simply "first text" and "second domain-specific text" or simply "second text".

[0101] Sample pair 332 is input into semantic converter 342 within equivalence checker 108. Semantic converter 342 presents domain-specific text in a more structured and logical form, such as corresponding sets of logical assertions, while preserving their logical meaning. These sets of assertions provide a more rigorous and logical description of the text content within sample pair 332.

[0102] The semantic codes (e.g., sets of logical assertions) generated by the semantic converter 342 from the text in sample pair 332 are provided as input to the logic constraint generator 344. The logic constraint generator 344 converts these semantic codes into logical formulas and mathematical (arithmetic) constraints, and structures them into a format suitable for interpretation by an automated reasoning solver, such as a satisfiability modular theory (SMT) solver. One format that can be used for this purpose is SMT-LIB. However, other suitable solver encodings can be used.

[0103] Once the solver encodings of the sample pair of 332 texts are generated by the logic constraint generator 344, these solver encodings are combined with additional solver constraints. These constraints confirm that for any possible input, the first domain-specific text produces a different output than the second domain-specific text. If the automated reasoning solver produces an "unsatisfiable" result when processing this combination of constraints, it indicates that no input exists that would cause the first domain-specific text and the second domain-specific text to produce different outputs, meaning they are functionally / semantically equivalent. Conversely, if the solver produces a "satisfiable" result, it indicates that at least one input (witness) exists that would cause the first domain-specific text and the second domain-specific text to produce different outputs, thus confirming that they are not functionally / semantically equivalent.

[0104] In some instances, the equivalence checker 108 integrates multiple automated reasoning solvers, such as 346-1, 346-2, ..., 346-N, which operate in parallel or concurrently on the combined logical constraints generated by the logic constraint generator 344 for the sample pairs 332. Employing different automated reasoning solvers can serve various purposes, including but not limited to the following: First, different automated reasoning solvers can apply different algorithms, heuristics, and optimization strategies. Using multiple automated reasoning solvers for the same problem provides insight into which solvers perform optimally for a specific input. Some automated reasoning solvers may excel in a particular domain or problem category but struggle in others. Employing multiple automated reasoning solvers increases the likelihood that at least one of them will produce effective results. For example, some problems may require complex combinations of constraints or theories, posing a challenge to specific automated reasoning solvers. Employing multiple solvers increases the chance of finding a solver capable of effectively handling this complexity.

[0105] If multiple automated reasoning solvers reach a consensus on a result (e.g., "unsatisfiable"), this can enhance the level of confidence in the result. Running multiple automated reasoning solvers simultaneously on the same problem can speed up the search for solutions. For example, equivalence checker 108 can trigger multiple automated reasoning solvers on combined logical constraints and return the first finding generated by any of these solvers.

[0106] In certain scenarios, methods for sampling and cascading large language models using equivalence checks have facilitated software development within cloud-based integrated development environments (IDEs). Specifically, IDEs can leverage these methods to automatically generate code based on code generation suggestions provided by the IDE user.

[0107] Figure 4 An example of a graphical user interface (GUI) 400 for an intelligent integrated development environment (IDE) is illustrated, which employs the methods described herein to achieve automated code generation. This intelligent IDE may be cloud-based or reside in other software development environments hosted by a provider network (such as provider network 100) and accessible via the Internet or other intermediate networks (such as intermediate network 102). The functionality of the intelligent IDE can be executed through client applications (such as client application 124) on user devices (e.g., client device 122) within the network (e.g., client network 120). This may involve utilizing web browsers, thereby allowing software developers to work on different devices and locations without needing to install and configure software on their respective computing devices.

[0108] One of the features of a smart IDE is code editing. This smart IDE provides a code editor equipped with features such as syntax highlighting, auto-completion, and other coding aids, allowing developers to write, edit, and format code directly within the IDE application or another graphical computing environment, such as a web browser. For example, Figure 4 The graphical user interface 400 depicted includes a code editor. Figure 4 In the example shown, the user has started coding a Python programming language file named "sort.py". On the first line of the file, the user has entered Python programming language comments that explain the Python function they intend to code. In this particular instance, the user's goal is to create a function that sorts a list in ascending order.

[0109] Now go to Figure 5This illustrates a sample graphical user interface (GUI) 500 for an intelligent integrated development environment (IDE) that uses disclosed technologies for automated code generation. The GUI 500 reflects the user's experience as described above. Figure 4 The code editor's state is as depicted in GUI 400 after a comment is entered on line 1 and after the user has entered a large language model code generation suggestion on line 3. In this example, the prefix "LLM:" is interpreted as a command or signal to the intelligent IDE, meaning the text following the prefix is ​​the generation suggestion. In this example, the suggestion instructs LLM to generate a function to sort the list. Note that this suggestion is ambiguous because it does not specify how the list should be sorted (e.g., ascending, descending, etc.).

[0110] In response to a code generation suggestion, the Smart IDE sends a code generation suggestion to service 110 via API 112. The code generation suggestion can be sent from the user's web browser (e.g., at client device 122) to service 110. Alternatively, the code generation suggestion may first be sent to the Smart IDE service in a provider network (e.g., provider network 100), and then from the Smart IDE to service 110. In either case, service 110 can respond as described above regarding... Figure 1 The described sampling of Large Language Models (LLMs), or as mentioned above... Figure 2 The described process involves sampling and cascading a set of LLMs. The response is then returned to the user's web browser for input into a code editor.

[0111] Now go to Figure 6 This example illustrates a graphical user interface (GUI) 600 of a smart IDE that employs the techniques for automatic code generation disclosed herein. The GUI 600 reflects the state of the code editor upon receiving a response. The smart IDE has automatically replaced the code generation suggestion entered into the code editor on line 2 with the response to lines 2 through 11 of the code editor (e.g., ...). Figure 5(As depicted in GUI 500). Here, the equivalence checker 108 finds that the samples of the prompt obtained from the LLM are functionally / semantically inequivalent. Specifically, the LLM generates: one domain-specific text output where the list is sorted in ascending order (lines 2-4); and another domain-specific text output where the list is sorted in descending order (lines 6-8). The response includes both outputs. Additionally, the response includes a witness (counterexample) returned by the equivalence checker 108 in lines 10-11, which proves that the two domain-specific text outputs are functionally / semantically inequivalent. In this example, the two functions will sort the input list [1, 2] differently, one in ascending order ([1, 2]) and the other in descending order ([2, 1]). From the response, the user can easily determine that the original prompt is ambiguous.

[0112] Beneficially, the techniques disclosed in this paper allow users to automatically generate domain-specific text from the code editor of a smart IDE using a Large Language Model (LLM). The generated and returned domain-specific text is likely a clear answer to the user's prompts and is unlikely to be an illusion. In the context of code generation, illusions can take many forms, such as grammatically incorrect, using the wrong programming language, or grammatically correct but functionally incorrect. Furthermore, if verified domain-specific text can be obtained from a relatively low-cost LLM service, the cost of the LLM service can be reduced.

[0113] Figure 7 An example of a method for sampling a large language model using equivalence checks is illustrated. Some or all of operations 700 (or other processes described herein, or variations and / or combinations thereof) are executed under the control of one or more computing devices configured with executable instructions. These operations are implemented as code (e.g., executable instructions, one or more computer programs, or one or more application programs) that executes jointly on one or more processors. The code is stored on a computer-readable storage medium. For example, the code may be in the form of a computer program including instructions executable by one or more processors. The computer-readable storage medium is non-transitory. In some examples, one or more (or all) of operations 700 are performed by... Figure 1 and Figure 2 Service 110 and Automatic Reasoning Service 104 are used to execute.

[0114] Operation 700 includes, at box 702, receiving code generation hints including code specifications. For example, service 110 may receive a high-level description or specification of the code to be generated by LLM. This description or specification may be in natural language or structured form. The code specification defines what the code should achieve, its input and output behavior, any specific libraries, frameworks, or programming languages ​​to be used, and any other relevant details. Upon receiving the code generation hints, cascading LLM service 110 may process the code specification to extract key details such as function names, variable types, control flow structures, and any other important elements.

[0115] Operation 700 also includes, at box 704, selecting a large language model service. For example, service 110 may select the lowest-cost LLM service from a set of available options, or it may select the lowest-cost LLM service from available options, wherein the lowest-cost LLM service generates a predicted probability of generating a functionally / semantically equivalent code specification that exceeds a certain threshold or minimum probability.

[0116] Operation 700 also includes, at box 706, sampling the selected large language model service with code generation suggestions. For example, service 110 may, in several cases, transmit code specifications from code generation suggestions to the selected large language model service. In some instances, the code generation suggestions sent to the large language model service may differ from those received in the operation at box 702. As an example, the code generation suggestions dispatched to the large language model service may contain key details extracted by service 110 from the code generation suggestions received during the operation at box 702, combined with code specifications. Furthermore, in some cases, service 110 may modify the code generation suggestions received in the operation at box 702 and forward the adapted code generation suggestions to the large language model service. For example, service 110 may reduce the size of the received code generation suggestions by removing one or more examples included in the initial code generation suggestions, or service 110 may amplify the size of the received code generation suggestions by including one or more examples.

[0117] Operation 700 also includes, at box 708, receiving samples from the large language model service based on the sampling performed in box 706. Each sample may contain the output of the large language model given a code specification from a code generation suggestion, where the code generation suggestion outlines a specific coding task or instruction. For each sample, the large language model analyzes the code generation suggestion and strives to produce code that satisfies the requirements or intent expressed in the code specification of the code generation suggestion. To achieve this goal, the large language model leverages its understanding of programming languages, syntax, and popular coding patterns to generate consistent and relevant code.

[0118] Operation 700 also includes, at block 710, using an equivalence checker to determine whether each of the one or more sample pairs received at block 706 is functionally / semantically equivalent. This equivalence checker may be a specialized automated theorem-proving tool or software system (e.g., an SMT solver) configured to establish equivalence between two samples, presented in the form of a logical formula or theorem (e.g., as an SMT-LIB constraint). Employing formal logic and mathematical reasoning, the equivalence checker determines whether two samples are functionally / semantically equivalent, indicating that they produce the same output on all potential inputs. The two samples being compared can be described using a formal language (e.g., SMT-LIB) or another variant of first-order logic, propositional logic, or higher-order logic.

[0119] If the equivalence checker indicates that one of the sample pairs is functionally / semantically inequivalent, the method can continue by returning to the operation at box 704 to select a new large language model service. The code generation hint or its processed version received from the operation at box 702 can be sent to this new large language model service. For example, service 110 can select the next lowest-cost LLM service from the available options, or the next lowest-cost LLM service that generates a predicted probability of functionally / semantically equivalent code specifications exceeding a certain threshold or minimum probability. In some scenarios, if the equivalence checker indicates that one of the sample pairs is functionally / semantically inequivalent, the method can prompt the user to refine or restate the hint received at box 702, instead of moving to the next LLM service in the queue. The same LLM service selected in operation 704 can be reused, but this time with the refined or rewritten hint. To help the user resolve the ambiguity of the hint, the user's hint can present evidence returned by the equivalence checker proving that the sample pair is functionally / semantically inequivalent.

[0120] Conversely, if the determination made by the equivalence checker is that each sample pair is functionally / semantically equivalent, then operation 700 also includes providing one or more sample responses from the sample responses received from the current large language model service at box 712 as an action to answer the code generation prompt received from the operation at box 702.

[0121] Some examples discussed in this paper involve sampling a single LLM using one or more prompts. However, multiple LLMs can be sampled using the same prompt (e.g., parallel sampling). For example, LLM A and LLM B can be sampled using prompt P to obtain domain-specific text A' from LLM A and domain-specific text B' from LLM B. The functional / semantic equivalence of texts A' and B' can then be tested (e.g., using equivalence checker 108). If two texts A' and B' are functionally equivalent, then texts A', B', both A' and B', or a combination of A' and B' can be returned as the answer to prompt P, based on the premise that since texts A' and B' are functionally / semantically equivalent, these texts A' and B' are likely consistent or correct answers to prompt P.

[0122] If two texts, A' and B', are not functionally / semantically equivalent, it can be determined whether the prompt P is malformed, incorrect, or ambiguous, or whether either or both of LLM A and LLM B cannot return a consistent or correct answer to the prompt P. In cases where it is suspected that either or both of LLM A and LLM B cannot return a consistent or correct answer to the prompt P, the prompt P can be cascaded to LLM C to obtain text C'. The functional / semantic equivalence of text C' with either or both of texts A' and B' can be tested. If text C' is functionally / semantically equivalent to text A' or text B', then any one, two, or a combination of functionally / semantically equivalent texts can be returned as an answer to prompt P, based on the possibility that either LLM A or LLM B failed to generate a consistent or correct response to prompt P, and on the premise that since two texts from two different LLMs are functionally equivalent, these functionally / semantically equivalent texts are likely to be consistent or correct answers to prompt P. If text C' is not functionally / semantically equivalent to both text A' and text B', then prompt P is more likely to be malformed, incorrect, or ambiguous, because the domain-specific text pairs output by three different LLMs for the same prompt are not functionally / semantically equivalent.

[0123] Alternatively, if it is suspected that either or both of LLM A and LLM B cannot return a consistent or correct answer to the prompt, the prompt P can be cascaded to LLM C and LLM D to obtain domain-specific text C' from LLM C and domain-specific text D' from LLM D. The functional / semantic equivalence of texts C' and D' can then be tested (e.g., using equivalence checker 108). If two texts C' and D' are functionally equivalent, then texts C', D', both C' and D', or combinations of C' and D' can be returned as answers to the prompt P, based on the premise that since texts C' and D' are functionally / semantically equivalent, these texts C' and D' are likely consistent or correct answers to the prompt P.

[0124] If two texts, C' and D', are not functionally / semantically equivalent, it can be determined whether prompt P is malformed, incorrect, or ambiguous, or whether either or both of LLM C and LLM D cannot return a consistent or correct answer to prompt P. If it is suspected that either or both of LLM A and LLM B cannot return a consistent or correct answer to prompt P, prompt P can also be further cascaded to another LLM (if another LLM is available).

[0125] Examples in this article include sampling Large Language Models (LLMs). LLMs are a subset of base models specifically designed for understanding and generating human-like text. They are commonly used for NLP tasks. LLMs are primarily used for tasks involving text, including but not limited to text completion, summarization, translation, question answering, and dialogue. GPT-3, GPT-4, BERT, and RoBERTa are examples of LLMs. Like base models, LLMs are typically pre-trained on a wide range of language modeling tasks and then fine-tuned for specific NLP tasks.

[0126] While the examples in this paper involve sampling LLMs, the sampling and cascading techniques presented here can be applied more generally to base models. Base models and Large Language Models (LLMs) refer to a category of large-scale machine learning models trained on massive amounts of data and capable of being fine-tuned or adapted for a variety of tasks. Base models are pre-trained models that serve as a starting point for building more specialized models. These models are trained on large amounts of data from diverse sources, acquiring a broad range of knowledge and capabilities. Base models can be used in a variety of applications, including Natural Language Processing (NLP), Computer Vision, and more. Base models are not limited to text. GPT-3, BERT, and computer vision models such as ResNet can all be considered base models because they provide a foundation for a wide range of applications. They are typically pre-trained on a broad range of tasks (e.g., language modeling or image classification) and then fine-tuned for specific applications.

[0127] Therefore, in summary, a base model refers to a large, pre-trained model that can be used as the foundation for a variety of applications across different domains (including but not limited to text). A Large Language Model (LLM) refers to a specialized type of base model focused on processing and generating text. While all LLMs can be considered base models, not all base models are LLMs. The sampling and cascading techniques disclosed in this paper can be used with base models, including LLMs.

[0128] Figure 8 An example of a provider network environment is illustrated, where techniques for sampling and concatenating large language models using equivalence checks are employed. Provider network 800 includes the infrastructure and architecture established by a cloud service provider to deliver cloud services and virtualized resources to customers. Provider network 800 can be distributed across multiple data centers in different geographic regions. Each data center houses physical hardware and servers that host virtualized resources, applications, and services provided by the cloud service provider. The purpose of provider network 800 is to enable customers to access and utilize cloud-based services, applications, and virtualized resources from their customer network (e.g., customer network 850) via the Internet, and to use their computing devices (e.g., customer device 852) securely and efficiently. While for the sake of clarity, Figure 8 The document only describes one customer network and one customer device, but the provider network 800 can be accessed by many customer networks and many customer devices.

[0129] In addition to providing sampling and concatenation of large language models using equivalence verification service 830, provider network 800 can provide virtualization service 810, which allows customers to use virtualized resources (e.g., virtualized resource 812) within provider network 800. While for the sake of clarity, in Figure 8 Only a single virtualization resource is described, but provider network 800 may include zero or more virtualization resources. Therefore, virtualization resource 812 generally represents one of many potential virtualization resources in provider network 800. Virtualization resource 812 can be one of various types of virtualization resources present in provider network 800, including, for example, virtual machines, containers, virtual routers, virtual switches, virtual load balancers, virtual firewalls, virtual network interfaces, virtual network services, or any other suitable type of virtualization resource.

[0130] Virtualization service 810 allows multiple virtualized resources (e.g., virtualized resource 812) such as operating systems, servers, storage devices, or networks to reside on a single physical hardware platform (e.g., Figure 9The virtualization service 810 runs on a computing device 900. Virtualization services 810 may use various technologies to implement virtualization, including any or all of the following: a hypervisor or virtual machine monitor or other software or firmware that creates and manages virtual machines; a virtual machine or other emulation of a physical computer capable of running its own operating system and applications independently of the host system and other virtual machines on the same hardware; a host system or physical server or computer that runs the hypervisor / virtual machine monitor and hosts the virtual machines; a guest operating system installed on each virtual machine that behaves as if it were running on a dedicated physical machine; or other suitable virtualization technologies.

[0131] By implementing virtualization via the virtualization service 810, the provider network 800 offers several benefits to customers and cloud service providers. Virtualization saves costs and utilizes computing power more efficiently by running multiple virtual machines on a single physical server, making better use of underlying hardware resources. Each virtual machine is isolated from other virtual machines to a certain extent, providing security and fault tolerance, ensuring that if one virtual machine crashes or encounters a problem, it does not affect other virtual machines on the same host system. Virtualization allows for the easy creation, deletion, and migration of virtual machines, enabling greater flexibility and scalability in managing the provider network environment. Virtualization supports rapid software testing and development because it allows developers to quickly create multiple environments without requiring separate physical hardware.

[0132] Additionally or alternatively, virtualization provided by Virtualization Service 810 includes containerization technology. Containerization is a form of virtualization that allows software applications and their dependencies to be packaged and isolated into self-container units called containers. Each container includes the application, runtime environment, libraries, and other necessary components, ensuring that the application runs consistently and reliably across different environments. Containerization provides a lightweight, portable, and scalable solution for deploying and managing software applications. Containers share the host system's operating system kernel, making them more efficient than virtual machines that require separate guest operating systems. This allows containers to start and stop quickly, using fewer resources and easily scaling.

[0133] Provider network 800 uses public network addresses (e.g., public network address 814) and local IP addresses (e.g., local network address 814) to provide virtualized resources to clients. Provider network 800 allows public network address 814 and local network address 816 to be associated with virtualized resources 812 provided to clients via virtualization service 810. Public network address 814 can be one of many public network addresses used by provider network 800. The same applies to local network address 816. Therefore, public network address 814 and local network address 816 generally represent the public network address and local network address used by provider network 800 to provide virtualized resources, respectively. Both public network address 814 and local network address 814 can be Internet Protocol (IP) network addresses, such as, for example, IPv4 or IPv6 network addresses. Local network address 816 can be a private network address located in the address block reserved in Internet Engineering Task Force (IETF) Request for Comments (RFC) 1918, a private network address in the address format specified by IETF RFC 4193, or another type of private network address.

[0134] Using resource 812 and public network address 814, a customer can implement a customer-specific application and present that application on intermediate network 840 (e.g., the Internet). Network traffic originating outside provider network 800 is not directly routed to local network address 816. Instead, network traffic uses public network address 814, which is mapped to local IP address 816. Provider network 800 may include networking devices or equipment that provide Network Address Translation (NAT) or similar functionality to perform forward mapping from public IP address 814 to local IP address 816. Another network entity 820 on intermediate network 840 may generate network traffic (e.g., Internet Protocol (IP) packets) destined for public network address 814. Network traffic destined for public network address 814 is routed to provider network 800 via intermediate network 840. Received network traffic is routed within provider network 800 to local network address 816 and virtualization resource 812 that processes the network traffic. Network traffic generated by virtualization resource 812 may be routed onto intermediate network 840, reaching network entity 820.

[0135] Figure 9 Examples of computing devices for implementing techniques for sampling and concatenating large language models using equivalence checks are illustrated. In some examples, systems implementing some or all of the techniques described herein may include general-purpose computer systems, such as... Figure 9The illustrated computing device 900 (also referred to as a computing system or electronic device) includes or is configured to access one or more computer-accessible media. In the illustrated example, the computing device 900 includes one or more processors 910 coupled to system memory 920 via an input / output (I / O) interface 930. The computing device 900 also includes a network interface 940 coupled to the I / O interface 930. Although Figure 9 The computing device 900 is shown as a single computing device, but in various examples, the computing device 900 may include a single computing device or any number of computing devices configured to work together as a single computing device 900.

[0136] In various examples, computing device 900 may be a single-processor system including one processor 910 or a multiprocessor system including several processors 910 (e.g., two, four, eight, or another suitable number). Processor 910 may be any suitable processor capable of executing instructions. For example, in various examples, processor 910 may be a general-purpose or embedded processor implementing any of a variety of instruction set architectures (ISAs), such as x86, ARM, PowerPC, SPARC, or the MIPS ISA or any other suitable ISA. In a multiprocessor system, each of the processors 910 may typically, but not necessarily, implement the same ISA.

[0137] System memory 920 may store instructions and data accessible by processor 910. In various examples, system memory 920 may be implemented using any suitable memory technology, such as random access memory (RAM), static RAM (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory. In the illustrated example, program instructions and data (such as the methods, techniques, and data described above) that implement one or more desired functions are shown as service code 925 (e.g., executable to fully or partially implement...). Figure 1 or Figure 2 The services 110 and automatic reasoning service 104) and data 926 are stored in the system memory 920.

[0138] In some examples, I / O interface 930 may be configured to coordinate I / O traffic between processor 910, system memory 920, and any peripheral devices within the device, including network interface 940 and / or other peripheral interfaces (not shown). In some examples, I / O interface 930 may perform any necessary protocol, timing, or other data transformations to convert data signals from one component (e.g., system memory 920) into a format suitable for use by another component (e.g., processor 910). In some examples, for instance, I / O interface 930 may include devices that support attachment via various types of peripheral buses, such as the Peripheral Component Interconnect (PCI) bus standard or variants of the Universal Serial Bus (USB) standard. In some examples, for instance, the functionality of I / O interface 930 may be split into two or more separate components, such as a northbridge and a southbridge. Additionally, in some examples, some or all of the functionality of I / O interface 930 (such as an interface to system memory 920) may be directly incorporated into processor 910.

[0139] For example, network interface 940 may be configured to allow computing device 900 to communicate with other computing devices 960 attached to one or more networks 950 (such as, e.g., Figure 1 The network interface 940 can exchange data with other computer systems or devices shown. In various examples, for instance, the network interface 940 can support communication via any suitable wired or wireless general-purpose data network (such as various types of Ethernet networks). Additionally, the network interface 940 can support communication via telecommunications / telephone networks (such as analog voice networks or digital fiber optic communication networks), via storage area networks (SANs) (such as Fibre Channel SANs), and / or via any other suitable type of network and / or protocol.

[0140] In some examples, computing device 900 includes one or more offload cards 970A or 970B (including one or more processors 975 and possibly one or more network interfaces 940), which are connected using I / O interfaces 930 (e.g., a version implementing the Peripheral Component Interconnect Fast (PCI-E) standard or another interconnect bus such as Fast Path Interconnect (QPI) or Hyper Path Interconnect (UPI). For example, in some examples, computing device 900 may act as a host electronic device hosting computing resources such as computing instances (e.g., operating as part of a hardware virtualization service), and one or more offload cards 970A or 970B act as a virtualization manager that manages the computing instances running on the host electronic device. As an example, in some examples, offload cards 970A or 970B may perform computing instance management operations such as pausing and / or unpausing computing instances, starting and / or terminating computing instances, performing memory transfer / copy operations, etc. In some examples, these management operations may be performed by the offload card 970A or 970B in cooperation with a hypervisor (e.g., upon request from the hypervisor) executed by other processors 910A to 910N of the computing device 900. However, in some examples, the virtualization manager implemented by the offload card 970A or 970B may adapt to requests from other entities (e.g., from the computing instance itself) and may not cooperate with (or serve) any individual hypervisor.

[0141] In some examples, system memory 920 may be one example of a computer-accessible medium configured to store program instructions and data as described above. However, in other examples, program instructions and / or data may be received, transmitted, or stored on different types of computer-accessible media. Computer-accessible media may include any non-transitory storage medium or memory medium, such as magnetic or optical media, for example, a disk or DVD / CD coupled to computing device 900 via I / O interface 930. Non-transitory computer-accessible storage media may also include any volatile or non-volatile medium that may be included as system memory 920 or another type of memory in some examples of computing device 900, such as RAM (e.g., SDRAM, Double Data Rate (DDR) SDRAM, SRAM, etc.), read-only memory (ROM), etc. Furthermore, computer-accessible media may include transmission media or signals transmitted via communication media (such as networks and / or wireless links), such as electrical signals, electromagnetic signals, or digital signals, such as those implemented via network interface 940.

[0142] The various examples discussed or presented herein can be implemented in a wide variety of operating environments, in some cases including one or more user computers, computing devices, or processing devices that can be used to operate any of many applications. User devices or client devices can include any of several general-purpose personal computers, such as desktop or laptop computers running standard operating systems, and cellular, wireless, and handheld devices running mobile software and capable of supporting many networking and messaging protocols. Such systems may also include numerous workstations running any of a variety of commercially available operating systems and other known applications for purposes such as development and database management. These devices may also include other electronic devices, such as virtual terminals, thin clients, gaming systems, and / or other devices capable of communicating via a network.

[0143] Most examples use at least one network familiar to those skilled in the art to support communication using any of a wide range of widely available protocols such as Transmission Control Protocol / Internet Protocol (TCP / IP), File Transfer Protocol (FTP), Universal Plug and Play (UPnP), Network File System (NFS), Public Internet File System (CIFS), Extensible Messaging and Field Protocol (XMPP), AppleTalk, etc. The network may include, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Virtual Private Network (VPN), the Internet, an intranet, an extranet, the Public Switched Telephone Network (PSTN), an infrared network, a wireless network, and any combination thereof.

[0144] In examples using a web server, the web server can run any of a variety of server or middleware applications, including HTTP servers, File Transfer Protocol (FTP) servers, Common Gateway Interface (CGI) servers, data servers, Java servers, business application servers, etc. The server can also respond to requests from user devices, such as by executing one or more web applications that can be implemented as one or more scripts or programs written in any programming language (such as Java®, C, C#, or C++) or any scripting language (such as Perl, Python, PHP, or TCL) and combinations thereof. The server may also include a database server, including but not limited to commercially available database servers from Oracle®, Microsoft®, Sybase®, IBM®, etc. The database server can be relational or non-relational (e.g., "NoSQL"), distributed or non-distributed, etc.

[0145] The environment disclosed herein may include a variety of data storage areas and other memories and storage media as discussed above. These may reside in a variety of locations, such as on (and / or in) storage media local to one or more computers in a computer, or on storage media of any or all computers remotely on a network. In a particular example set, information may reside in a storage area network (SAN) familiar to those skilled in the art. Similarly, any necessary files for performing functions belonging to a computer, server, or other network device may be stored locally and / or remotely as appropriate. Where the system includes computerized devices, each such device may include hardware elements electrically coupled via a bus, including, for example, at least one central processing unit (CPU), at least one input device (e.g., mouse, keyboard, controller, touchscreen, or keypad), and / or at least one output device (e.g., display device, printer, or speaker). Such a system may also include one or more storage devices, such as hard disk drives, optical storage devices, and solid-state storage devices such as random access memory (RAM) or read-only memory (ROM), as well as removable media devices, memory cards, flash memory cards, etc.

[0146] Such devices may also include computer-readable storage medium readers, communication devices (e.g., modems, (wireless or wired) network interface cards, infrared communication devices, etc.), and working memory as described above. Computer-readable storage medium readers may be connected to or configured to receive computer-readable storage media, which represent remote, local, fixed, and / or removable storage devices and media for temporarily and / or more permanently accommodating, storing, transmitting, and retrieving computer-readable information. Systems and various devices will also typically include numerous software applications, modules, services, or other elements, including operating systems and applications such as client applications or web browsers, residing within at least one working memory device. It should be understood that alternative examples may have many variations different from those described above. For example, custom hardware may also be used, and / or specific elements may be implemented in hardware, software (including portable software such as applets), or both. Furthermore, connections to other computing devices, such as network input / output devices, may be employed.

[0147] Storage media and computer-readable media used to contain code or code portions may include any suitable media known or used in the art, including storage media and communication media, such as, but not limited to, volatile and non-volatile media, removable and non-removable media implemented in any way or technology to store and / or transmit information (such as computer-readable instructions, data structures, program modules or other data), including RAM, ROM, electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc-read-only memory (CD-ROM), digital universal disc (DVD) or other optical storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by system devices. Based on this disclosure and the teachings provided herein, those skilled in the art will appreciate other ways and / or methods for implementing the various examples.

[0148] In the foregoing description, various examples have been described. Specific configurations and details have been elaborated for illustrative purposes to provide a thorough understanding of the examples. However, it will be apparent to those skilled in the art that the examples can be practiced without these specific details. Furthermore, well-known features may be omitted or simplified to avoid obscuring the described examples.

[0149] In this document, parenthesized text and boxes with dashed borders (e.g., large dashes, small dashes, dot dashes, and dots) are used to exemplify optional aspects for adding additional features to some examples. However, this notation should not be interpreted as implying that these are the only options or optional operations, and / or that in some examples, boxes with solid borders are not optional.

[0150] The reference numerals with suffix letters (e.g., 114-1 to 114-N) can be used to indicate that one or more instances of the mentioned entity may exist in various examples, and when multiple instances exist, each instance need not be identical, but may share some general characteristics or function in a common form. Furthermore, unless explicitly indicated to the contrary, the suffixes used do not imply the existence of a specific number of entities. Thus, in various examples, two entities using the same or different suffix letters may have or not have the same number of instances.

[0151] References to "an example," "example," etc., indicate that the described example may include a particular feature, structure, or characteristic, but each example may not necessarily include that particular feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same example. Additionally, when a particular feature, structure, or characteristic is described in conjunction with an example, it should be assumed that, whether explicitly described or not, implementing such a feature, structure, or characteristic in conjunction with other examples is within the knowledge of those skilled in the art.

[0152] Furthermore, in the various examples described above, unless otherwise specifically indicated, the disjunctive linguistic intent of phrases such as “at least one of A, B, or C” is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Similarly, the linguistic intent of phrases such as “at least one or more of A, B, and C” (or “one or more of A, B, and C”) is understood to mean A, B, or C, or any combination thereof (e.g., A, B, and / or C). Therefore, disjunctive language is neither intended nor should be understood to imply that a given example requires the existence of at least one of A, at least one of B, and at least one of C.

[0153] As used herein, the term "based on" (or similar) is an open-ended term used to describe one or more factors that influence a determination or other action. It should be understood that this term does not exclude additional factors that may influence a determination or action. For example, a determination may be based solely on the listed factors or on said factors and one or more additional factors. Therefore, if action A is "based on" B, it should be understood that B is a factor influencing action A, but this does not preclude the action from also being based on one or more other factors, such as factor C. However, in some cases, action A may be entirely based on B.

[0154] Unless otherwise expressly stated, articles such as “a / an” should generally be interpreted as including one or more of the described items. Therefore, phrases such as “a device configured to…” or “computing device” are intended to include one or more of the described devices. Such one or more described devices may be collectively configured to perform the described operations. For example, “a processor configured to perform operations A, B, and C” may include a first processor configured to perform operation A working in conjunction with a second processor configured to perform operations B and C.

[0155] Furthermore, the words “may” or “may” are used in a permissive sense (i.e., implying a possibility) rather than a mandatory sense (i.e., implying a requirement). The words “include,” “including,” and “includes” are used to indicate an open relationship and therefore imply, including but not limited to. Similarly, the words “have,” “having,” and “has” also indicate an open relationship and therefore imply, having but not limited to. The terms “first,” “second,” “third,” etc., as used herein, serve as markers for the nouns that follow them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.) unless otherwise explicitly indicated. Similarly, the values ​​of such numerical markers are generally not used to indicate the required quantity of a particular noun in a claim set forth herein, and therefore, the element “fifth” generally does not imply the presence of four other elements unless those elements are explicitly included in the claim or their presence is otherwise sufficiently clear.

[0156] At least some embodiments of the disclosed technology may be described in light of the following terms: 1. A computer-implemented method for sampling and concatenating a large language model (LLM) using equivalence checks, the method being performed by one or more computing devices, the method comprising: Receive notification; Based on the prompts, the first large language model (LLM) is sampled to obtain a first set of sample answers generated by the first LLM, each sample answer in the first set of sample answers including the corresponding domain-specific text; An equivalence checker is used to determine that the first pair of corresponding domain-specific texts in the first set of sample answers are not equivalent; Based on the prompts, the second LLM is sampled to obtain a second set of sample answers generated by the second LLM, each of the sample answers in the second set of sample answers including the corresponding domain-specific text; Use an equivalence checker to determine whether the second pair of corresponding domain-specific texts in the second set of sample answers are equivalent; Based on the domain-specific text of the second pair, determine the specific domain-specific text to be displayed in the graphical user interface; and Provide the specific domain-specific text for display in the graphical user interface.

[0157] 2. The computer-implemented method as described in Clause 1, wherein the prompts are expressed in natural language.

[0158] 3. A computer-implemented method as described in any one of Clauses 1 to 2, wherein each sample answer in the first set of answers includes a corresponding definition of a function in a programming language.

[0159] 4. A computer-implemented method executed by one or more computing devices, the method comprising: Receive the first notification; Based on the first prompt, a large language model (LLM) is sampled to obtain a first set of sample answers generated by the first LLM, each sample answer in the first set of sample answers including the corresponding domain-specific text; An equivalence checker is used to determine that the first pair of corresponding domain-specific texts in the first set of sample answers are not equivalent; Receive a second prompt, wherein the second prompt is a refinement of the first prompt; Based on the second prompt, the LLM is sampled to obtain a second set of sample answers generated by the second LLM, each of the sample answers in the second set of sample answers including corresponding domain-specific text; Use an equivalence checker to determine whether the second pair of corresponding domain-specific texts in the second set of sample answers are equivalent; Based on the domain-specific text of the second pair, determine the specific domain-specific text to be displayed in the graphical user interface; and Provide the specific domain-specific text for display in the graphical user interface.

[0160] 5. The computer-implemented method as described in Clause 4, wherein the first prompt and the second prompt are expressed in natural language.

[0161] 6. The computer-implemented method as described in Clause 4, wherein each sample answer in the first set of answers includes a corresponding definition of a function in a programming language.

[0162] 7. The computer-implemented method as described in Clause 4, wherein each sample answer in the first set of answers includes a corresponding Structured Query Language (SQL) query.

[0163] 8. A computer-implemented method as described in any one of Clauses 4 to 7, wherein the equivalence checker includes a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are unsatisfiable.

[0164] 9. A computer-implemented method as described in any one of Clauses 4 to 7, wherein the equivalence checker includes a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the first pair of corresponding domain-specific texts of the first set of sample answers are not equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are satisfiable.

[0165] 10. The computer-implemented method as described in any one of clauses 4 to 7, further comprising: Obtain witnesses from the equivalence checker for the first pair of corresponding domain-specific texts, the witnesses including counterexamples to the equivalence of the first pair of texts; and The counterexamples to the equivalence of the first pair of texts are presented in the graphical user interface.

[0166] 11. A computer-implemented method as described in any one of clauses 4 to 7, wherein the determination that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent using the equivalence checker is based on: The first domain-specific text of the second pair and the second domain-specific text of the second pair are encoded according to the Satisfiability Model Theory Library (SMT-LIB) standard to generate a first SMT-LIB encoding of the first domain-specific text and a second SMT-LIB specification of the second domain-specific text; Combine the first SMT-LIB code and the second SMT-LIB code to generate a combined SMT-LIB code; and The combined SMT-LIB code is input into the SMT solver.

[0167] 12. The computer-implemented method as described in Clause 11, wherein the combined SMT-LIB encoding is input into each of a plurality of SMT solvers.

[0168] 13. A computer-implemented method as described in any one of Clauses 4 to 7, wherein the first prompt is received by a service in a provider network from a client device in a client network; and wherein the domain-specific text is returned by the service in the provider network to the client device in the client network.

[0169] 14. A computer-implemented method as described in any one of clauses 4 to 7, wherein the LLM is a first LLM, and whether the method further comprises: Before sampling the first large language model based on the first prompt: Based on the first prompt, the second LLM is sampled to obtain a third set of sample answers generated by the second LLM, each of the sample answers in the third set of sample answers including corresponding domain-specific text; The equivalence checker is used to determine that the third pair of corresponding domain-specific texts in the third set of sample answers are not equivalent; and Based on the determination that the third pair of corresponding domain-specific texts of the third set of sample answers are not equivalent, the first LLM is sampled based on the first prompt.

[0170] 15. A system comprising: A first or more computing devices are configured to implement a sampled large language model service in a multi-tenant provider network, the sampled large language model service including instructions that, when executed, cause the sampled large language model service to: Receive the first notification; Based on the first prompt, a large language model (LLM) is sampled to obtain a first set of sample answers generated by the first LLM, each sample answer in the first set of sample answers including the corresponding domain-specific text; Receive a second prompt, wherein the second prompt is a refinement of the first prompt; and Based on the second prompt, the LLM is sampled to obtain a second set of sample answers generated by the second LLM, each sample answer in the second set of sample answers including corresponding domain-specific text; and A second or more computing device, configured to implement an automated inference service in the multi-tenant provider network, the automated inference service including instructions that, when executed, cause the automated inference service to: An equivalence checker is used to determine that the first pair of corresponding domain-specific texts in the first set of sample answers are not equivalent; and The equivalence checker is used to determine that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent.

[0171] 16. The system as described in Clause 15, wherein the first prompt and the second prompt are expressed in natural language.

[0172] 17. The system as described in Clause 15, wherein each sample answer in the first set of answers includes a corresponding definition of a function in a programming language.

[0173] 18. The system as described in Clause 15, wherein each sample answer in the first set of answers includes a corresponding Structured Query Language (SQL) query.

[0174] 19. A system as described in any of 15 to 18, wherein the equivalence checker includes a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are unsatisfiable.

[0175] 20. A system as described in any one of clauses 15 to 18, wherein the equivalence checker includes a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the first pair of corresponding domain-specific texts of the first set of sample answers are not equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are satisfiable.

[0176] The specification and drawings should therefore be considered illustrative rather than restrictive. However, it will be apparent that various modifications and alterations may be made therein without departing from the broader scope of this disclosure as set forth in the claims.

Claims

1. A computer-implemented method executed by one or more computing devices, the method comprising: Receive the first notification; Based on the first prompt, a large language model (LLM) is sampled to obtain a first set of sample answers generated by the first LLM, each sample answer in the first set of sample answers including the corresponding domain-specific text; An equivalence checker is used to determine that the first pair of corresponding domain-specific texts in the first set of sample answers are not equivalent; Receive a second prompt, wherein the second prompt is a refinement of the first prompt; Based on the second prompt, the LLM is sampled to obtain a second set of sample answers generated by the second LLM, each of the sample answers in the second set of sample answers including corresponding domain-specific text; Use an equivalence checker to determine whether the second pair of corresponding domain-specific texts in the second set of sample answers are equivalent; Based on the domain-specific text of the second pair, determine the specific domain-specific text to be displayed in the graphical user interface; as well as Provide the specific domain-specific text for display in the graphical user interface.

2. The computer-implemented method of claim 1, wherein the first prompt and the second prompt are expressed in natural language.

3. The computer-implemented method of claim 1, wherein each sample answer in the first set of answers includes a corresponding definition of a function in a programming language.

4. The computer-implemented method of claim 1, wherein each sample answer in the first set of answers includes a corresponding Structured Query Language (SQL) query.

5. The computer-implemented method of any one of claims 1 to 4, wherein the equivalence checker comprises a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are unsatisfiable.

6. The computer-implemented method of any one of claims 1 to 4, wherein the equivalence checker comprises a satisfiability modulo theory (SMT) solver; and wherein using the equivalence checker to determine that the first pair of corresponding domain-specific texts of the first set of sample answers are not equivalent is based on the SMT solver determining that a set of logical constraints representing the first pair of corresponding domain-specific texts are satisfiable.

7. The computer-implemented method as described in any one of claims 1 to 4, further comprising: Witnesses for the first pair of corresponding domain-specific texts are obtained from the equivalence checker, the witnesses including counterexamples to the equivalence of the first pair of texts; as well as The counterexamples to the equivalence of the first pair of texts are presented in the graphical user interface.

8. The computer-implemented method of any one of claims 1 to 4, wherein determining that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent using the equivalence checker is based on: The first domain-specific text of the second pair and the second domain-specific text of the second pair are encoded according to the Satisfiability Model Theory Library (SMT-LIB) standard to generate a first SMT-LIB encoding of the first domain-specific text and a second SMT-LIB specification of the second domain-specific text; The first SMT-LIB code and the second SMT-LIB code are combined to produce a combined SMT-LIB code; as well as The combined SMT-LIB code is input into the SMT solver.

9. The computer-implemented method of claim 8, wherein the combined SMT-LIB code is input into each of the plurality of SMT solvers.

10. The computer-implemented method of any one of claims 1 to 4, wherein the first prompt is received by a service in a provider network from a client device in a client network; and wherein the domain-specific text is returned by the service in the provider network to the client device in the client network.

11. The computer-implemented method of any one of claims 1 to 4, wherein the LLM is a first LLM, and whether the method further comprises: Before sampling the first large language model based on the first prompt: Based on the first prompt, the second LLM is sampled to obtain a third set of sample answers generated by the second LLM, each of the sample answers in the third set of sample answers including corresponding domain-specific text; The equivalence checker is used to determine that the third pair of corresponding domain-specific texts in the third set of sample answers are not equivalent; and Based on the determination that the third pair of corresponding domain-specific texts of the third set of sample answers are not equivalent, the first LLM is sampled based on the first prompt.

12. A system comprising: A first or more computing devices are configured to implement a sampled large language model service in a multi-tenant provider network, the sampled large language model service including instructions that, when executed, cause the sampled large language model service to: Receive the first notification; Based on the first prompt, a large language model (LLM) is sampled to obtain a first set of sample answers generated by the first LLM, each sample answer in the first set of sample answers including the corresponding domain-specific text; Receive a second prompt, wherein the second prompt is a refinement of the first prompt; as well as Based on the second prompt, the LLM is sampled to obtain a second set of sample answers generated by the second LLM, each of the sample answers in the second set of sample answers including corresponding domain-specific text; as well as A second or more computing device, configured to implement an automated inference service in the multi-tenant provider network, the automated inference service including instructions that, when executed, cause the automated inference service to: An equivalence checker is used to determine that the first pair of corresponding domain-specific texts in the first set of sample answers are not equivalent; and The equivalence checker is used to determine that the second pair of corresponding domain-specific texts of the second set of sample answers are equivalent.

13. The system of claim 12, wherein the first prompt and the second prompt are expressed in natural language.

14. The system of any one of claims 12 to 13, wherein each sample answer in the first set of answers includes a corresponding definition of a function in a programming language.

15. The system of any one of claims 12 to 13, wherein each sample answer in the first set of answers comprises a corresponding Structured Query Language (SQL) query.

Images