Instruction compliance large language model to reduce computing resource consumption

Abstract

Implementations relate to using instruction decomposition, self-evaluation, and optional progressive perfection to improve instruction compliance of large language models (LLMs). A processor of the system may obtain a natural language (NL)-based input, generate a plurality of candidate responses using the LLM and evaluate the candidate responses based on instructions included in the NL-based input, and progressively perfect the candidate responses until it is determined that one or more termination criteria are met. In some implementations, NL-based input may be received from a client device. In these implementations, a progressively perfected given candidate response may be rendered for presentation at a client device, and the given candidate response is responsive to an NL-based input. In additional or alternative implementations, NL-based inputs may be obtained from a database. In these implementations, a given candidate response that is progressively perfected may be utilized in fine tuning of the LLM.

Description

Background Technology

[0001] Large Language Models (LLMs) are a specific type of machine learning model that performs a variety of Natural Language Processing (NLP) tasks, such as language generation, machine translation, and question answering. These LLMs are typically trained on massive amounts of diverse data, including, but not limited to, web pages, ebooks, software code, e-news articles, and machine-translated data. Therefore, these LLMs leverage the underlying data on which they were trained to perform these diverse NLP tasks. For example, when performing a language generation task, these LLMs can process natural language (NL)-based input received from a client device and generate NL-based output that is in response to the NL-based input and is to be rendered at the client device.

[0002] In some cases, LLMs can include hundreds of millions, billions, or even hundreds of billions or more parameters. Therefore, given the large number of parameters included in an LLM, using an LLM to perform NLP tasks can consume relatively large amounts of resources (e.g., in terms of computational resources used to complete the NLP task, the time spent executing the NLP task, the energy consumed to complete the NLP task, etc.). Furthermore, also due to the size of the LLM, it may be difficult to adequately train it to reliably perform the task under the corresponding constraints of a given NLP task, as the LLM may be prone to generating illusions (e.g., generating responses to NL-based inputs that are inaccurate and / or meaningless). Therefore, in terms of computational resource usage, it is advantageous for an LLM to generate responses to NL-based inputs that do not require additional follow-up responses due to initially generating illusory responses. Summary of the Invention

[0003] The implementations described herein can be used to reduce the number of successive NL-based inputs that can be received by an LLM by mitigating and / or eliminating instances where the LLM produces illusions. While any given user may decide to provide successive NL-based inputs, any “average” reduction in the number of successive NL-based inputs can be extremely beneficial in terms of computational resource usage. More specifically, some implementations described herein involve using self-evaluation when leveraging an LLM to generate responses to NL-based inputs. Some of these implementations described herein involve using a progressive refinement process in conjunction with self-evaluation when leveraging an LLM to generate responses to NL-based inputs. Some additional or alternative implementations described herein involve using self-evaluation to identify NL-based inputs that cannot be adequately responded to using an LLM.

[0004] For example, in some implementations, an LLM can be used to process NL-based input to generate multiple responses, and a critique of these responses can be generated by comparing them to the instructions included in the NL-based responses. For instance, the NL-based input might request the LLM to generate a response consisting of six lines and conforming to a particular writer's style. Thus, the LLM can be used to generate multiple responses and determine whether each of these responses follows each of the instructions (e.g., based on determining the number of lines in the response and whether the response conforms to the style of that particular writer). Therefore, if a response is determined to follow all (or at least more than a certain threshold number) of the instructions in the NL-based input, the response can be considered "high-quality." For example, it can be assumed that the probability of a given response leading to subsequent NL-based input corresponds to the degree to which the response conforms to the set of instructions included in the NL-based input. Multiple responses can then be progressively refined until it is determined that a response follows all (or at least a threshold number) of the instructions included in the NL-based response. Such techniques can result in responses that reduce the number of subsequent NL-based inputs (at least on average across a user population).

[0005] Incremental refinement can be a repetitive (or in other words, iterative or looping) process, in which at each iteration, candidate responses generated so far are selected for refinement, and each refined response is added to a candidate response "pool". In some implementations, the candidate response with the highest potential (e.g., highest quality) is selected for refinement.

[0006] Because LLM operates probabilistically, the quality of the initial candidate responses generated using LLM based on NL-based input can vary. For example, if the average quality of the candidate responses is taken, some candidate responses may be considered below average quality, and some above average quality. By evaluating the quality of candidate responses against instructions included in the NL-based input, the candidate responses considered to be of the highest quality can be identified. This can then be used, for example, to determine the most promising candidate responses to be refined. Thus, at each iteration for incremental refinement, the likelihood of generating responses of higher quality than those generated so far can be increased.

[0007] In some implementations, the self-evaluation and progressive refinement processes described herein can be used during the utilization of NL-based response systems, including LLM, to generate responses to NL-based inputs associated with a user via a client device (e.g., provided by the user via the client device). In this way, the likelihood of generating all (or at least a threshold number) of the instruction responses that conform to the NL-based input provided by the user and are to be rendered at the client device is increased.

[0008] In some additional or alternative implementations, the self-evaluation and progressive refinement processes described herein can be used to generate synthetic training data for fine-tuning the LLM for subsequent use by, for example, a NL-based response system. For example, NL-based inputs can be obtained (e.g., from a database of previously submitted NL-based inputs provided by one or more users) and provided as input to the NL-based response system. The NL-based response system can then provide high-quality responses as outputs by utilizing the self-evaluation and progressive refinement processes described herein. The NL-based inputs and high-quality responses can then be stored as training instances to be used for fine-tuning the LLM. In some implementations, another candidate response from the candidate responses can also be stored as an example of a “low-quality” response, and fine-tuning can be based on both the “high-quality” and “low-quality” responses.

[0009] In some additional or alternative implementations, the self-evaluation process described herein can be used (e.g., from a database of previously submitted NL-based inputs provided by one or more users) to identify “hard” NL-based inputs. In this context, “hard” NL-based inputs can be defined as NL-based inputs for which the LLM (e.g., prior to fine-tuning) has a low probability of generating high-quality responses (e.g., due to violations of one or more instructions present in the NL-based input). Since the performance of an LLM (e.g., its instruction compliance ability) is generally expected to be improved through fine-tuning (or training) based on “hard” NL-based inputs, it is beneficial to identify “hard” NL-based inputs and provide them along with corresponding “high-quality” examples of responses to these inputs. Therefore, once “hard” NL-based inputs have been identified, the implementations described herein involve generating training data based on the identified NL-based inputs. For example, in some implementations, the progressive refinement process described herein can be used to generate the corresponding high-quality responses. In some additional or alternative implementations, the identified NL-based inputs or corresponding responses (which can be generated or retrieved from a database) can be modified so that the resulting NL-based inputs and responses correspond.

[0010] By fine-tuning the LLM based on examples of "high-quality" responses (especially for "hard" NL-based inputs), the average quality of the response generated using the fine-tuned LLM can be higher than the corresponding response generated using the LLM before fine-tuning. This process is repeatable, allowing the average quality of the response generated using the LLM to improve at each iteration.

[0011] In these and other ways, responses generated using LLM can reliably have higher quality. This is true whether the self-evaluation process (and optional progressive refinement process) described herein is used during response generation or to generate training data used to fine-tune the LLM before generating responses. Therefore, instances of subsequent (e.g., follow-up) NL-based inputs provided by the user, for example to improve the quality of the initial response, which the LLM would otherwise process, can be reduced. For example, if the initial response does not conform to the instruction from the initial NL-based input, the user can provide further NL-based input that explicitly requests compliance with the instruction to force the LLM to generate a response that conforms to the instruction. For example, if subsequent responses do not adequately conform to the instruction, or if there are further instructions that the response does not conform, the user can repeat this process multiple times. As described herein, the implementation of the self-evaluation process (and optional progressive refinement process) ensures that resources that would otherwise be consumed in these repetitive interactions with the LLM are saved.

[0012] Furthermore, as described herein, a mechanism for self-evaluation of responses generated using LLM is provided. In this way, responses and corresponding NL-based inputs can be stored as training data, where the self-evaluation process provides a way to label the training data with minimal human intervention and / or guided by the self-evaluation process. Additionally, a progressive refinement process can be used to ensure that high-quality examples are included in the training data. Therefore, the implementation described herein provides a relatively low-cost and time-efficient way to label training data compared to, for example, manually labeling training data by humans (e.g., manually indicating the relative or absolute quality of responses generated using LLM).

[0013] Furthermore, in some cases, self-evaluation and progressive refinement processes can be used as part of a NL-based response system for dialogue with human users (e.g., involving multiple inputs and responses). For example, NL-based response systems can be provided as part of dialogue with automated assistants, chatbots, etc. In some cases, the user can provide one or more commands to be performed as part of the dialogue (e.g., to control a smart device, generate code, generate commands to control a robot, assist in vehicle navigation, etc.). Therefore, the use of the processes described herein can also assist users in performing technical tasks through continuous and guided human-computer interaction processes. Furthermore, since responses generated using LLM can reliably have higher quality, the human-computer interaction process can be completed quickly and efficiently.

[0014] Furthermore, the implementation described herein allows users to interact with and control NL-based response systems more easily and intuitively, systems that are themselves technical systems. For example, because NL-based response systems can evaluate and refine responses themselves, users do not need to explicitly provide further NL-based input to perform these processes. As discussed herein, determining the content of such further NL-based input by humans may require trial and error, or may require a high level of skill, training, and / or familiarity with a particular LLM. Therefore, the implementation described herein mitigates these obstacles.

[0015] In other words, the implementation described herein provides a mechanism by which instructions included in NL-based input can be effectively utilized by the LLM when processing that NL-based input, without any additional interaction from the user, to generate higher-quality responses and thus provide more efficient access to information stored in the LLM. This enhances the effectiveness of NL-based input to the LLM (e.g., using the structured requests described herein to perform the described processes) and thus objectively improves the performance of information retrieval performed by the LLM.

[0016] As mentioned, LLMs are typically trained using data from sources such as web pages, ebooks, software code, e-news articles, and machine-translated data. When generating responses to specific NL-based inputs, for example, the LLM utilizes information distilled from the underlying data upon which it was trained. In this way, an LLM can be considered a database structure in which information is stored within the parameters of the LLM. Since, as described herein, NL-based inputs to be processed by an LLM can be enhanced using the processes described herein, this can be considered an improved database query that produces more efficient information retrieval.

[0017] The above description serves as an overview of some implementations of this disclosure. Further details of those implementations and others will follow. Attached Figure Description

[0018] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0019] Figure 2A Example process flows for evaluating candidate responses to NL-based inputs generated using NL-based response systems are described, based on various implementations.

[0020] Figure 2B Example process flows for generating responses to NL-based inputs using NL-based response systems are described, based on various implementations.

[0021] Figure 2C Example process flows are described, based on various implementations, for generating responses to NL-based inputs associated with client devices using NL-based response systems, and causing the display of those responses.

[0022] Figure 2D Example process flows for generating training instances by using a NL-based response system to generate responses to NL-based inputs, based on various implementations, are described.

[0023] Figure 3 Example layouts depict a progressive improvement process based on various implementation methods.

[0024] Figure 4A Example process flows for generating training instances by modifying NL-based inputs for NL-based response systems, based on various implementations, are described.

[0025] Figure 4B Example process flows for generating training instances by modifying responses to NL-based inputs generated by an NL-based response system, based on various implementations, are described.

[0026] Figure 5 Example process flows for fine-tuning large language models, based on various implementations, are described.

[0027] Figure 6A A flowchart is depicted illustrating example methods for generating responses to NL-based inputs associated with client devices using NL-based response systems, according to various implementations.

[0028] Figure 6BA flowchart is depicted illustrating example methods for generating training instances by utilizing a NL-based response system to generate responses to NL-based inputs, based on various implementations.

[0029] Figure 7A A flowchart is depicted illustrating example methods for generating training instances by modifying the NL-based input to a NL-based response system, based on various implementations.

[0030] Figure 7B A flowchart is depicted illustrating example methods for generating training instances by modifying the responses to NL-based inputs generated by an NL-based response system, based on various implementations.

[0031] Figure 8 A flowchart is depicted illustrating example methods for generating responses to NL-based inputs associated with client devices using NL-based response systems, according to various implementations.

[0032] Figure 9 Example architectures of computing devices based on various implementations are described. Detailed Implementation

[0033] Turn now Figure 1 The diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0034] The example environment includes a client device 110 and an NL-based response system 120. In some implementations, all or some aspects of the NL-based response system 120 may be implemented locally at the client device 110. In additional or alternative implementations, all or some aspects of the NL-based response system 120 may be obtained from, for example... Figure 1 The depicted client device 110 is implemented remotely (e.g., at a remote server). In these implementations, the client device 110 and the NL-based response system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet)).

[0035] Client device 110 may be one or more of the following: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart home appliance (such as a smart TV), and / or user's wearable device including a computing device (e.g., user's watch with computing device, user's glasses with computing device, virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0036] Client device 110 may execute one or more software applications via application engine 115, through which NL-based input can be submitted, and / or NL-based output and / or other outputs in response to the NL-based input can be rendered (e.g., audibly and / or visually). Application engine 115 may execute one or more software applications that are separate from the operating system of client device 110 (e.g., installed "on top" of the operating system), or alternatively implemented directly by the operating system of client device 110. For example, application engine 115 may execute a web browser or automation assistant installed on top of the operating system of client device 110. As another example, application engine 115 may execute a web browser software application or automation assistant software application that is integrated as part of the operating system of client device 110. Application engine 115 (and one or more software applications executed by application engine 115) may interact with NL-based response system 120.

[0037] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of client device 110. Additionally, or alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in the field of view of one or more visual components. Additionally, or alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch or typed input directed at client device 110.

[0038] Some examples of NL-based input described herein can be queries formulated based on user input provided by the user of client device 110 and detected by user input engine 111 in response to NL. For example, the query can be a typing query typed via a physical or virtual keyboard, a suggested query selected via the touchscreen or mouse of client device 110, a spoken voice query detected via the microphone of client device 110 (and optionally directed to an automated assistant performing at least partially at client device 110), or an image or video query based on visual data captured by the vision component of client device 110 (or based on NL input generated using, for example, an object detection model, a captioning model, etc.). Other examples of NL-based input described herein can be prompts for NL content formulated based on user input provided by the user of client device 110 and detected by user input engine 111. For example, the prompt can be a typing query typed via a physical or virtual keyboard, a suggested prompt selected via the touchscreen or mouse of client device 110, a spoken prompt detected via the microphone of client device 110, or an image prompt based on an image captured by the vision component of client device 110.

[0039] In various implementations, client device 110 may include a rendering engine 112 configured to render content (e.g., NL-based responses) for audible and / or visual presentation to a user of client device 110 using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers enabling the content to be provided for audible presentation to a user via client device 110. Additionally, or alternatively, client device 110 may be equipped with a display or projector enabling the content to be provided for visual presentation to a user via client device 110.

[0040] In various implementations, client device 110 may include a context engine 113 configured to determine the context (e.g., current or recent context) of client device 110 and / or its users (e.g., active users of client device 110 when client device 110 is associated with multiple users). In some of these implementations, context engine 113 may determine the context based on data stored in a selectively utilized data database 110A. The data stored in the selectively used data database 110A may include, for example, user interaction data characterizing the current or recent interactions of the client device 110 and / or the user of the client device 110, location data characterizing the current or recent location of the client device 110 and / or the user of the client device 110, user attribute data characterizing one or more attributes of the user of the client device 110, user preference data characterizing one or more preferences of the user of the client device 110, user profile data characterizing the profile of the user of the client device 110 (e.g., whether the user has a subscription to the use of the NL-based response system 120, and if so, optionally characterizing the level of the subscription to the use of the NL-based response system 120) and / or any other data accessible to the scene engine 113 via the selectively used data database 110A or otherwise.

[0041] For example, the context engine 113 may determine the current context based on the current state of the conversation (e.g., considering one or more recent inputs provided by the user during the conversation), profile data, and / or the current location of the client device 110. For example, the context engine 113 may determine the current context of “best landmarks to visit in London” based on recently issued queries, profile data, and / or the current or expected future location of the client device 110 (e.g., based on calendar information associated with the user that the context engine 113 can access). As another example, the context engine 113 may determine the current context based on which software application is active in the foreground of the client device 110, the current or recent state of the active software application, and / or the content currently or recently rendered by the active software application. The context determined by the context engine 113 can be used, for example, to supplement or rewrite NL-based inputs specified based on user input, generate implicit NL-based inputs (e.g., implicit queries or prompts specified independently of any explicit NL-based inputs provided by the user of the client device 110), and / or determine the submission of implicit NL-based inputs and / or render results for implicit NL-based inputs (e.g., NL-based outputs).

[0042] In various implementations, client device 110 may include an implicit input engine 114 configured to: generate implicit NL-based input independently of any user-explicitly NL-based input provided by a user of client device 110; submit implicit NL-based input, optionally independently of any user-explicitly NL-based input requesting the submission of implicit NL-based input; and / or cause rendering to respond to implicit NL-based input, optionally independently of any explicitly NL-based input requesting the rendering of such response. For example, implicit input engine 114 may use one or more past or current contexts from context engine 113 to generate implicit NL-based input, determine the submission of implicit NL-based input, and / or determine the response causing rendering to respond to implicit NL-based input. For example, implicit input engine 114 may automatically generate and automatically submit implicit queries or implicit hints based on the one or more past or current contexts. Furthermore, the implicit input engine 114 may automatically push responses generated in response to implicit queries or implicit prompts, causing the responses to be rendered automatically, or may automatically push notifications of the responses (such as optional notifications), which, when selected, cause the responses to be rendered. Additionally, or alternatively, the implicit input engine 114 may submit corresponding implicit NL-based input at regular or irregular intervals, causing corresponding responses to be provided automatically (or notifications of corresponding responses to be provided automatically). For example, implicit NL-based input may be “Automation Assistant News” based on one or more past or current contexts indicating the user’s general interest in the automation assistant; implicit NL-based input or variations thereof may be submitted periodically, and corresponding responses may be provided automatically (or notifications of corresponding responses to be provided automatically). It should be noted that, given the existence of new / recent search result documents over time, for example, the corresponding responses may vary over time.

[0043] Furthermore, the client device 110 and / or the NL-based response system 120 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components that facilitate communication via one or more networks in network 199. In some implementations, one or more software applications may be locally installed at the client device 110, while in other implementations, one or more software applications may be remotely hosted (e.g., by one or more servers) and may be accessible from the client device 110 via one or more networks in network 199.

[0044] although Figure 1The aspects shown or described herein relate to a single client device with a single user, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form a device ecosystem that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., user group, home, workplace, hotel, etc.).

[0045] The NL-based response system 120 in Figure 1 The engine is shown as including a fine-tuning engine 130, a request generation engine 141, an LLM engine 142, an instruction decomposition engine 151, a comment response generation engine 152, a response revision engine 161, a refinement engine 162, an input modification engine 171, and a response modification engine 172. Some of these engines may be combined and / or omitted in various implementations. Furthermore, these engines may include various sub-engines. For example, the fine-tuning engine 130 in... Figure 1 The diagram is shown to include training instance engine 131 and training engine 132. Similarly, sub-engines may be combined and / or omitted in various implementations. Therefore, it should be understood that... Figure 1 The various engines and sub-engines of the NL-based response system 120 shown are depicted to describe specific functionality and are not intended to impose limitations.

[0046] Furthermore, the NL-based response system 120 in Figure 1 The system is shown interfacing with various databases, such as the selective utilization data database 110A, the LLM log database 131A, the training instance database 132A, the request generation data database 141A, the LLM database 142A, and the example input data database 151A. While specific engines and / or sub-engines are depicted as having access to specific databases, it should be understood that this is for illustrative purposes and not intended to be limiting. For example, in some implementations, each of the various engines and / or sub-engines of the NL-based response system 120 may have access to every database in the various databases. Furthermore, some of these databases may be combined and / or omitted in various implementations. Therefore, it should be understood that... Figure 1 The various databases that the NL-based response system 120 interfaces with are depicted to describe the specific data that the NL-based response system 120 can access and are not intended to be restrictive.

[0047] As this article (for example, regarding...) Figures 2A to 2D , Figure 3 , Figure 4A , Figure 4B , Figure 5 , Figure 6A , Figure 6B , Figure 7A , Figure 7B and Figure 8 In more detail, the NL-based response system 120 can be used to evaluate NL-based input-response pairs and, in some implementations, refine initial candidate responses generated based on the NL-based inputs. For example, the NL-based inputs can be processed by an LLM engine 142 using an LLM stored in an LLM database 142A to generate initial candidate responses. The instruction set included in the NL-based input data can be obtained by an instruction decomposition engine 141, for example, by processing the NL-based inputs using an LLM stored in an LLM database 142A. The initial candidate responses and the instructions included in the corresponding NL-based inputs can be processed by a comment response generation engine 152, for example, using an LLM stored in an LLM database 142A, to generate comment responses. The comment responses can indicate a comparison between the initial candidate responses and the instruction set (e.g., including an indication of whether the corresponding candidate response conforms to each instruction in the instruction set, and optionally including reasoning for that determination).

[0048] In some implementations, a refinement engine 162 can be used to progressively refine the response. To refine the response, it can be revised based on the initial candidate responses and corresponding comment responses processed by the response revision engine 161, for example, using the LLM stored in the LLM database 142A. This can be repeated multiple times using the refinement engine 162 to ensure that one or more of the initial candidate responses are refined to conform to the instruction set.

[0049] Instruction generation, comment response generation, and / or response refinement can be initiated by utilizing the corresponding request generated by the request generation engine 141 through LLM processing and optionally by using request generation data stored in the request generation data database 141A.

[0050] In some implementations, the NL-based input to be processed by the NL-based response system 120 may be associated with the client device 110 (e.g., explicitly provided by the user of client device 110 via user input engine 111, implicitly provided by the user of client device 110 via implicit input engine 114, etc.), and self-evaluation and progressive refinement may be used to generate a specific response to be rendered at the client device 110 (e.g., using rendering engine 112) (e.g., regarding...). Figure 2B and Figure 6A (As described).

[0051] In additional or alternative implementations, the NL-based input can be obtained from example input data stored in the LLM log database 131A and / or the example input data database 151A. In some of these implementations, self-evaluation and progressive refinement can be used to generate labeled training instances, including the NL-based input and refined responses to the NL-based input (e.g., as per...). Figure 2C and Figure 6B (As described). In some other implementations of these methods, self-evaluation can be used to generate labeled training instances, including modified NL-based inputs modified using input modification engine 171 and responses to the modified NL-based inputs (e.g., as described regarding...). Figure 4A and Figure 7A (As described). In some further implementations of these methods, self-evaluation can be used to generate labeled training instances, including NL-based inputs and using response modification engines 172 (e.g., as described in...). Figure 4B and Figure 7B The training instances (as described) are modified and respond to the modified response based on the NL-based input. Training instances may be stored in a training instance database 132A, for example, using a training instance engine 131. The LLMs stored in the LLM database 142A can be fine-tuned using the training engine 132 based on the training instances stored in the training instance database 132A (e.g., as described). Figure 5 (As described). This article is about Figures 2A to 2D , Figure 3 , Figure 4A , Figure 4B , Figure 5 , Figure 6A , Figure 6B , Figure 7A , Figure 7B and Figure 8 Additional descriptions of the various engines and / or sub-engines of the NL-based response system 120 are provided.

[0052] Turn now Figure 2A It describes a method for evaluating response systems using NL-based approaches (e.g., Figure 1 An example process flow is provided for the generation of candidate responses to NL-based inputs by an NL-based response system 120. As discussed herein, NL-based input 210 is available. NL-based input 210 may be provided to the NL-based response system to obtain a response in response to NL-based input 210. For example, NL-based input 210 may include a query or prompt. In some implementations, NL-based input 210 may include an intent to complete a specific task, such as that to be performed by an automated assistant or chatbot communicatively coupled to the NL-based response system (e.g., via network 199).

[0053] The NL-based input 210 can use an NL-based response system (such as regarding...). Figure 1 The NL-based response system 120 described herein processes the input 210. The NL-based response system 120 can generate an LLM output by processing the NL-based input 210 based on (e.g., using an LLM stored in an LLM database 142A by an LLM engine 142). Using the LLM output, multiple candidate responses 220 can be determined. Specifically, candidate responses 220 can be candidate LLM responses. Since an LLM can be described as probabilistic, candidate responses 220 can be different from each other, even though they are based on the same NL-based input 210 and generated using the same LLM. Therefore, it can be assumed that the actual degree of response of the candidate responses 220 to the NL-based input 210 will differ.

[0054] In some implementations, an instruction set 212 may also be obtained from the NL-based input 210. In some implementations, the instruction set 212 may be extracted from the NL-based input 210 by an NL-based response system 120 (e.g., using an instruction decomposition engine 151). For example, the instruction set 212 may be generated based on processing the NL-based input 210 using an LLM (e.g., the same LLM used to generate candidate responses 220, or a different LLM stored in an LLM database 142A).

[0055] In some versions of these implementations, the NL-based response system 120 may extract instruction set 212 based on a request to extract instruction set 212 from NL-based input 210. The content of this request may be automatically generated during the utilization of the NL-based response system 120, or the content of this request may be generated prior to the utilization and retrieved when needed (e.g., from the request generation data database 141A). In either case, the request to extract instruction set 212 may be generated based on the obtained content.

[0056] In some implementations, the request does not include any instruction samples (e.g., the request uses a so-called "zero-shot" approach). In this way, the instruction set 212 extracted using LLM can be less constrained and biased (e.g., the properties of any samples provided).

[0057] In some implementations, the output format and / or style can be defined in the request. This ensures consistency in the output format and / or style and prevents paraphrasing by the model. Therefore, the parsing of the output can be made more reliable.

[0058] As an example, a request for this purpose can be represented as: "Given a 'query', your task is to provide a set of all the instruction constraints specified in the 'query' Output: { “instruction_constraint”: List all the instruction constraints in the query by breaking it down into individual verifiable components } Query: {query} Response: {response} Given a 'query', your task is to provide a set of all instruction constraints specified in the 'query'. Output: { "Instructions_Constraints": Lists all instructions and constraints in a query by breaking it down into individual verifiable components. } Query: {query} Response: {response}”)

[0059] In various implementations, the NL-based response system 120 may generate a comment response 230 based on (e.g., using a comment response generation engine 152) processing candidate responses 220 and instruction set 212 (and / or NL-based input 210). For example, an LLM (e.g., the same or a different LLM used to generate candidate responses 220 and / or instruction set 212) may be used to generate the comment response 230 based on processing candidate responses 220. A corresponding comment response in the comment response 230 may be generated for each candidate response in the candidate responses 220. Therefore, each comment response in the comment response 230 may indicate the degree to which one or more candidate responses in the corresponding candidate responses 220 conform to instruction set 212. For example, a given comment response 230 may include an indication of whether the corresponding candidate response 220 conforms to each instruction in instruction set 212. The comment response 230 may also include reasoning for why it has been determined that candidate response 220 conforms to or does not conform to each instruction in instruction set 212. Therefore, the score for each candidate response in candidate response 220 can be determined (or provided in the corresponding comment response) based on the number of instructions in instruction set 212 that are determined to match (e.g., as indicated by the corresponding comment response). This score can, for example, be in the range of 0 to 1. For example, and not limitingly, if a particular candidate response in candidate response 220 is determined to match three of the four instructions included in instruction set 212, then the score for that particular candidate response 220 can be provided as 0.75.

[0060] In some implementations, the comment response 230 can be generated based on a request to generate a comment response 230 for the LLM. For example, the request can be generated and processed using the LLM along with candidate responses 220 and instruction set 212. The content of the request to generate a comment response 230 for the LLM can be predefined or can be generated using an NL-based response system 120.

[0061] As mentioned above regarding the request to extract instructions from NL-based input, in some implementations, this request does not include any sample of the LLM comment response. In this way, the comment response 230 generated using LLM can be less constrained and biased (e.g., the properties of any sample provided).

[0062] Similar to requests to extract instructions from NL-based input, some implementations allow the output format and / or style to be defined within the request. This ensures consistency in the output format and / or style of the request and prevents reinterpretations by the model. Consequently, the parsing of the output becomes more reliable.

[0063] As an example, a request for this purpose can be represented as: "Given a 'response' and 'instruction constraints', your task is to evaluate how well the 'response' follows each of the 'instructionconstraints'. Output: { For each instructional constraint in the 'instruction_constraints'output the following keys "instruction_constraint": "selected instruction constraint", "reasoning": "provide comprehensive analysis on whether the aboveconstraint is satisfied by 'response' by referencing specific portions ofresponse" "follows_instruction": Output 'false' if response does not satisfy the above instruction constraint based on 'reasoning', otherwise output 'true' } Instruction constraints: {instruction_constraints} Response: {response} (Given a 'response' and 'instruction constraints', your task is to evaluate how well the 'response' conforms to each of the 'instruction constraints'.) Output: { For each instruction constraint in 'Instruction_Constraint', output the following key "Instruction_Constraint": "Selected instruction constraint" "Reasoning": "Providing a comprehensive analysis of whether the 'response' satisfies the above constraints by referencing specific parts of the response." "Follow _Instruction": Output 'False' if the response based on 'reasoning' does not satisfy the above instruction constraints; otherwise, output 'True'. } Instruction constraints: {Instruction_Constraint} Response: {response}”)

[0064] In some implementations, a comment response generation engine 152 can be used to generate a comment response 230 based on processing candidate responses 220 and NL-based input 210. In these implementations, an instruction set can be extracted from the NL-based input 210, and the candidate responses 220 can then be evaluated against the instruction set 212 in a single step (e.g., using an LLM based on processing a single request in a so-called "one-shot" method). In this way, later parts of the request can reference earlier parts, thus creating order dependencies in the LLM output. This reduces compounding errors. Furthermore, the computational requirements are reduced because the number of inferences performed by the LLM to generate the comment response 230 can be decreased. This is especially true compared to, for example, using an LLM to process the request for each instruction in the instruction set.

[0065] As an example, a request for this purpose can be represented as: "Given a 'query' and 'response', your task is to evaluate how well the 'response' follows each of the instruction constraints specified in the 'query'. Output: { “instruction_constraints”: List all of the instructional constraints in query by breaking it down into individual verifiable components, For each instructional constraint in the 'instruction_constraints'output the following keys "instruction_constraint": "selected instruction constraint", "reasoning": "provide comprehensive analysis on whether the aboveconstraint is satisfied by 'response' by referencing specific portions ofresponse" "follows_instruction": Output 'false' if response does not satisfy the above instruction constraint based on 'reasoning', otherwise output 'true' } Query: {query} Response: {response} ("Given a 'query' and a 'response', your task is to evaluate how well the 'response' conforms to each of the directive constraints specified in the 'query'.) Output: { "Instructions_Constraints": Lists all instructions and constraints in a query by breaking it down into individual verifiable components. For each instruction constraint in 'Instruction_Constraint', output the following key "Instruction_Constraint": "Selected instruction constraint" "Reasoning": "Providing a comprehensive analysis of whether the 'response' satisfies the above constraints by referencing specific parts of the response." "Follow _Instruction": Output 'False' if the response based on 'reasoning' does not satisfy the above instruction constraints; otherwise, output 'True'. } Query: {query} Response: {response}”)

[0066] In some implementations, at least some of the instructions in instruction set 212 may be related to the statistics of candidate responses (e.g., the instructions may specify the maximum number of characters, words, sentences, lines, paragraphs, etc.). In these implementations, the comment response generation engine 152 may employ a mathematical engine to determine the statistics of candidate responses (and whether the candidate responses meet the required statistics). In this way, errors in the evaluation performed by the LLM due to the illusion created by the LLM regarding the statistics of candidate responses can be mitigated.

[0067] In some implementations, the operations of process flow 200A can be repeated multiple times. In other words, multiple comment responses 230 can be generated for each candidate response in candidate response 220. Since LLM can be probabilistic as described herein, even when processing the same input data (e.g., candidate response 220 and / or other input data described herein) using LLM, each comment response in the comment responses 230 for the corresponding candidate response 220 can differ (e.g., in terms of which instruction in instruction set 212 the corresponding response 220 conforms to, or the reasoning provided for the reason why this occurs). Therefore, the comment responses 230 for the corresponding candidate response 220 (or their determined scores) can be aggregated (e.g., averaged, summed, joined, the most consistent response determined by evaluating "self-consistency," etc.). In this way, noise in the evaluation of candidate response 220 can be reduced, and the robustness of the evaluation of candidate response 220 can be improved.

[0068] As described with respect to process flow 200A, each instruction in instruction set 212 can be identified (or extracted) and evaluated individually against candidate response 220 (e.g., via a so-called "decomposition" method). In this way, the evaluation of candidate response 220 can be reliable even when a particular NL-based input includes many and / or complex instructions. For example, by forcing the LLM to evaluate the response against each instruction sequentially, the LLM can be prevented from ignoring or combining some instructions, or otherwise paraphrasing the comments on the response.

[0069] Turn now Figure 2B It describes a system for generating response systems using NL-based approaches (e.g., Figure 1 An example process flow 200B is generated by an NL-based response system 120 in the example. Initially, a determination 240 can be made (e.g., using a refinement engine 162) regarding whether one or more termination criteria are met. Termination criteria may include, for example, a threshold degree to which a given candidate response conforms to a set of instructions. For example, a termination criterion can be determined to be met if a given candidate response (e.g., from an initial candidate response 220, or from a refined candidate response 260 as described below) conforms to a greater number of instructions in instruction set 212 than the number of instructions indicated by the termination criterion. In this case, determination 240 may be a termination criterion met (e.g., as indicated by a "yes" path), where a given candidate response is selected as the selected candidate response 224. In some implementations, the termination criterion (e.g., a threshold degree) may include the determination that a given candidate response conforms to all instructions in instruction set 212.

[0070] Additionally, or alternatively, termination criteria may include one or more of the following: threshold repetition count, threshold execution time, threshold processor loop count, threshold memory utilization, etc. In this way, the maximum time and / or resource allocation for generating and / or refining the selected candidate response 224 can be defined. Furthermore, situations where process flow 200B will repeat indefinitely can be avoided (e.g., situations where the NL-based response system 120 cannot generate candidate responses with a degree exceeding the threshold). If it is determined that the termination criteria are met because processing time and / or computational resources have exceeded the defined threshold, when processing time and / or computational resources have exceeded the defined threshold, a candidate response (e.g., from candidate response 220 and / or the refined candidate response 260 described below) can be selected as the selected candidate response 224 based on determining which candidate response best conforms to the instruction set (e.g., based on the corresponding candidate response score).

[0071] If a determination 240 is made that one or more termination criteria are not met, the “No” path may be followed, and the next node may be selected 245.

[0072] In many implementations, each initial candidate response in the initial candidate response 220 can be considered a "root node" in a tree structure. As described herein, the initial candidate response 220 can be refined to generate refined candidate responses 260. Therefore, each refined candidate response in the refined candidate response 260 can also be considered a "node" in the tree structure and a "child node" of the initial candidate response 220 used in the generation of the refined candidate responses. Further, the refined candidate response 260 can then be refined to generate further refined candidate responses 260, which themselves can be considered a "node" in the tree structure and a "child node" of the refined candidate response 260 used in the generation of the refined candidate responses. This can be repeated multiple times (e.g., until one or more termination criteria are determined to be met).

[0073] Therefore, to determine which of the 245 nodes should be selected, the next "node" can be selected, for example, using the refinement engine 162. The next node of the 245 nodes can be selected in any suitable manner. For example, in some implementations, the next node (or candidate response) of the 245 nodes can be selected based on the A* algorithm. Furthermore, in some implementations, the next node (or candidate response) of the 245 nodes can be selected based on a recursive search algorithm (e.g., process flow 200B can typically be recursive). In some other implementations, the next node of the 245 nodes can be selected sequentially or randomly.

[0074] In some implementations, the next node (or candidate response) can be selected based on determining which of the nodes represents the most promising next node (e.g., for generating a refined candidate response that better conforms to the instruction set). This can be determined based on determining which candidate response best conforms to the instruction set (e.g., the candidate response associated with the highest score). Nodes that have already been expanded can be excluded from further expansion (e.g., to avoid wasting computational resources).

[0075] In some implementations, each node (or candidate response) in the node (or candidate response) can be arranged as a priority queue. Therefore, each node (or candidate response) in the node (or candidate response) can be associated with a priority. The priority can be based on the comment response corresponding to the node (or candidate response). For example, the priority can be a score of the candidate response or can be determined based on the score of the candidate response. In these implementations, the next node (or candidate response) can therefore be selected based on searching the priority queue. The node with the highest priority (or in other words, the candidate response determined to best conform to instruction set 212) can be identified and selected as the next node.

[0076] Although this selection 245 has been generally described with respect to tree search algorithms, it should be understood that any suitable method may be used. Furthermore, although each of the initial candidate responses 220 and the refined responses 260 is described as a node in a tree structure, it should be understood that they may also be collectively described as a set of candidate responses, with each new generation of refined responses being added to the candidate response set.

[0077] By intelligently selecting the next node 245 in the manner described herein, the number of repetitions required to generate a refined candidate response 260 that meets the termination criteria can be reduced. Therefore, the time and computational resources required to generate a given refined candidate response 260 that meets the termination criteria can be reduced. Furthermore, even if no given refined candidate response is generated before the termination criteria are met (e.g., due to exceeding the maximum number of repetitions), the selected candidate response 224 can be considered to conform to the instruction set to a higher degree compared to selected candidate responses 224 in other cases.

[0078] Once the next node 245 has been selected, one or more improved (or replaced) candidate responses 260 can be generated (e.g., using the response revision engine 161). For example, an LLM stored in the LLM database 142A (e.g., the same LLM used previously, or a different LLM) can be used to process the candidate responses 222 corresponding to the selected node and the corresponding comment responses 232.

[0079] In some implementations, a refined candidate response 260 can be generated based on a request to refine a candidate response 222 for LLM. For example, the request can be generated and processed using LLM along with the candidate response 222 and the corresponding comment response 232 (and optional instruction set 212). The content of the request to generate the refined candidate response 260 for LLM can be predefined or can be generated using an NL-based response system 120.

[0080] In some implementations, the output format and / or style can be defined in the request. This ensures consistency in the output format and / or style of the request and prevents reinterpretations by the model. Consequently, the parsing of the output can also be made more reliable.

[0081] In some implementations, the response revision engine 161 may be guided (e.g., via a request) to ensure that the generated improved candidate response 260 conforms to the instructions in the instruction set 212 for which candidate response 222 was determined to be non-compliant (e.g., as indicated by comment response 232). For example, each instruction in the instructions for which candidate response 222 was determined to be violated may be identified (e.g., using comment response 232). These instructions may then be provided (e.g., in the request) to increase the likelihood that the improved candidate response 260 will better conform to instruction set 212. In some implementations, inference about why candidate response 222 was determined to violate these instructions may also be provided (e.g., from comment response 232). This may further increase the likelihood that the improved candidate response will better conform to the instruction set, for example, by providing additional information in the request for LLM processing. In implementations where multiple comment responses 232 are generated for candidate response 222, inference from one or more comment responses for a specific instruction may be aggregated and provided. This can also further increase the likelihood that the refined candidate response better conforms to the instruction set, for example, by providing even more additional information in the request to be processed by the LLM.

[0082] The refined response 260 can then be evaluated against instruction set 212 to generate a comment response 234. This can be achieved using, for example, a comment response generation engine 152, as described above. Figure 2A The process is executed in a manner similar to that described in Flowchart 200A.

[0083] The incremental refinement process can then be repeated. For example, a determination can be made as to whether any of the improved candidate responses 260 generated in this iteration meet the threshold instruction number at determination 240 (e.g., based on comment response 234). If so, the improved candidate response 260 can be selected as the selected candidate response 224. Otherwise, the next node 245 can be selected, where the improved candidate response 260 has been added to the "tree structure" as an additional node and as a child node of candidate response 222 that has been used in the generation of improved candidate responses (or in other words, added to the candidate response set).

[0084] As described in this article, this incremental improvement process can be repeated (or in other words, cyclically), and in some implementations, it can be performed recursively. For example, temporarily switching... Figure 3 This describes an example layout of a progressive refinement process 300. For example... Figure 3 The described progressive improvement process 300 may include a self-assessment process 310 (e.g., as about...). Figure 2A (As described), request generation 320 and response generation 330. Similarly, as... Figure 3 As described, the self-assessment process 310 may include using a mathematical engine 314 to determine response statistics 312, and determining a validator 316 based on instruction decomposition 318.

[0085] As described in more detail in this article, a request can be generated at request generation 320 (e.g., using...). Figure 1 (Request generation engine 141). This could be a request for a refined candidate response (which could be an initial candidate response or a candidate response that has been refined in a previous iteration). The request could be processed at response generation 330 (e.g., using response revision engine 161) to generate a refined candidate response.

[0086] The refined response can be evaluated using a self-evaluation process 310. In some implementations, the NL-based input to which the refined response is addressed may include one or more instructions associated with statistics related to the response. Therefore, various statistics (e.g., the number of characters, words, sentences, lines, paragraphs, etc.) can be calculated for the response. Response statistics can be calculated using, for example, a mathematical engine 314 (e.g., instead of using the LLM itself, which may provide unreliable statistical information).

[0087] The validator 316 can then determine the degree to which the response conforms to the instructions included in the NL-based input. For example, the validator 316 can generate a comment response (e.g., using a comment response generation engine 152). The comment response can indicate which instruction in the instructions the refined response was determined to conform to or violate, and in some implementations, it indicates the reasoning for how these determinations were made. Instructions can be provided to the validator 316, or they can be determined using instruction decomposition 318 (e.g., based on instruction decomposition engine 151) based on processing the NL-based input. Where the instruction is related to statistics of the response, the validator 316 can use response statistics 312, which are calculated using a math engine 314. The comment response can then be provided to request generation 320 to generate a request for the next repetition (or loop).

[0088] Turn now Figure 2C An example process flow 200C is described for generating a response to an NL-based input associated with a client device using an NL-based response system, and causing the display of that response. Figure 2C Example process flow 200C is largely related to... Figure 2A The example process flow described is the same as 200A. However, as Figure 2C As shown in the process flow 200C, the input 210 based on NL can be connected with... Figure 1 The client device 110 (or another client device) is associated with it. Furthermore, the selected response 224 (e.g., according to...) Figure 2B Example process flow 200B (generated) can be provided for rendering at client device 110 (e.g., via rendering engine 112).

[0089] For example, NL-based input 210 may be provided based on user input made by a user of client device 110. The user may provide user input, for example, by typing on a virtual or physical keyboard of client device 110; providing speech captured by one or more microphones of client device 110; selecting suggested input (e.g., via tapping on a touchscreen display, voice commands, using a pointing device, etc.); providing gestures captured by one or more sensors of client device 110, etc. Information indicating user input may be used to determine NL-based input 210. For example, the information may include entered text, selected text, or text determined based on the user's speech through speech recognition processing. This text may then be provided as NL-based input 210. As another example, the information may include one or more lexical units that may be used to determine NL-based input 210 (e.g., by client device 110 or NL-based response system 120). The information may be provided by client device 110 to NL-based response system 120, for example, via a wireless network (such as network 199).

[0090] Similarly, the selected response 224 (or information indicating the selected response 224) may be provided to the client device 110 by the NL-based response system 120 (e.g., via network 199). Commands may also be sent to the client device 110 to render the selected response 224 (e.g., via the display of the client device 110, via the speakers of the client device 110, etc.). However, in some implementations, it is contemplated that the client device 110 renders the selected response 224 upon receipt, without receiving any explicit command to do so.

[0091] Although it is generally described that the client device 110 associated with the NL-based input 210 and the client device 110 rendering the selected response 224 are the same client device 110, this may not be the case in some implementations. In other words, the client device 110 rendering the selected response 224 may be a different client device 110 from the client device 110 that provided the NL-based input 210. For example, and not limitingly, the selected response 224 may be rendered on a display that is separate from (but possibly associated with) the smart speaker receiving the NL-based input 210, for example, because the user account is logged in on both devices.

[0092] In this way, the NL-based response system 120, and in particular the self-evaluation mechanism described herein, can be used to generate a response to the NL-based input 210 associated with the client device 110, and cause the client device 110 to render the selected response 224. In other words, the NL-based response system 120 can be used to provide a response to the user based on the processing performed by the LLM. Therefore, resources required to process repetitive interactions with the LLM (e.g., subsequent NL-based inputs) that might otherwise occur to refine the initial response can be saved. Furthermore, the expertise and experience required to formulate NL-based inputs to retrieve a specific response (which may include, for example, evaluation and refinement of intermediate responses) can be reduced and / or completely eliminated.

[0093] In some implementations, a determination 242 is made regarding whether incremental improvement should be performed. This determination may be based on the context data and / or the content of the NL-based input 210 (e.g., based on the type or number of instructions included in the NL-based input 210, based on keywords or phrases included in the NL-based input 210, etc.). In some implementations, the context data may be associated with the client device 110 and / or the user of the client device 110. For example, the context data may include one or more of the following: time, location of the client device 110, preference data associated with the user account (e.g., subscription service data), applications running on the client device, and the subject and / or recipient of messages displayed on the client device. The context data may be retrieved, for example, from a selectively utilized data database 110A.

[0094] In some implementations, determination 242 may be based on whether an initial response generated based on processing NL-based input 210 conforms to a threshold number of instructions included in NL-based input 210 (e.g., all instructions included in NL-based input 210). For example, this determination may include generating the initial response based on processing the NL-based input using an LLM (e.g., an LLM stored in an LLM database 142A). A corresponding comment response may then be generated using an LLM (e.g., a comment response generation engine 152) based on processing the initial response. The corresponding comment response may include an indication of the degree to which the LLM response conforms to the set of instructions included in the NL-based input. The degree to which the initial response conforms to the set of instructions may be higher or lower than a threshold degree may then be determined based on the comment response. In response to determining that the initial response conforms to the instructions to a degree higher than the threshold, it may be determined that incremental improvement is not required (e.g., and therefore prevented). However, in response to determining that the initial response conforms to the instructions to a degree lower than the threshold, it may be determined that incremental improvement is performed.

[0095] If incremental improvement is not performed, a response system 120 based on NL (e.g., using its LLM) can be used to generate a response 226 based on processing the NL-based input 210. The response 226 can then be rendered on the client device 110 (e.g., instead of the selected response 224). In this way, computational resources that would otherwise be consumed can be saved when incremental improvement is not performed (e.g., if the LLM can generate a satisfactory response to the NL-based input 210 without using incremental improvement).

[0096] Turn now Figure 2D An example process flow 200D is described for generating training instances by using an NL-based response system 120 to generate responses to NL-based inputs 210. Figure 2D Example process flow 200D is largely related to... Figure 2A The example process flow described is the same as 200A. However, as Figure 2D As shown in process flow 200D, NL-based input 210 can be obtained from LLM log database 131A and / or example input data database 151A. Furthermore, the selected response 224, the NL-based input 210, and optional corresponding comment responses can be stored in training instance database 132A.

[0097] In some implementations, a determination 244 can be made before storage regarding whether the selected candidate response 224 conforms to instruction set 212 at least to a threshold level. If determination 244 is negative, the response can be prevented from being stored as a training instance. In this way, if a candidate response 224 is selected because it meets a termination criterion (e.g., due to exceeding the number of repetitions) but does not conform to instruction set 212 to a threshold level, the storage of the selected candidate response 224 can be avoided. Therefore, the quality of training instances in the training instance database 132A can be ensured.

[0098] In some implementations, NL-based inputs stored in the example input data database 151A may be provided by the user, for example, as an example of a "hard" prompt. In some additional or alternative implementations, the LLM log database 131A may include a historical log of NL-based inputs provided as input by one or more users to the LLM (e.g., the LLM stored in the LLM database 131A), and optionally include corresponding responses generated in response to the NL-based inputs 210.

[0099] In some implementations, NL-based input 210 can be identified from NL-based input stored in LLM log database 131A and / or sample input data database 151A. Identification of NL-based input 210 can be based on one or more selection criteria. Selection criteria can be based on, for example, the content of NL-based input 210 (e.g., specific words or phrases present in NL-based input 210). Additionally, or alternatively, selection criteria may include the determination of a quality metric based on the corresponding response generated from processing NL-based input 210 having a quality metric below a threshold. The corresponding response may have been generated at a previous time and retrieved from LLM log database 131A and / or sample input data database 151A. Alternatively, NL-based response system 120 can be used to generate the corresponding response in the same or similar manner as any of the methods described herein.

[0100] In some implementations, a quality metric may include the degree to which a corresponding response conforms to a set of instructions included in a particular NL-based input. This may be determined based on evaluating the NL-based input 210 and the corresponding candidate response 220. For example, a corresponding comment response 230 may be generated by processing the corresponding candidate response 220 (e.g., using a comment response generation engine 152). The corresponding comment response 230 may include an indication of the degree to which the corresponding candidate response 220 conforms to a set of instructions 212 included in the NL-based input 210, which may then be used as (or used to determine) a quality metric.

[0101] In some additional or alternative implementations, the quality metric for the corresponding response can be user-generated. For example, at a previous time, an NL-based input 210 (or instruction set 212) and a corresponding response may be presented to the user. The user can then provide feedback on the degree to which the corresponding response conforms to the instructions included in the NL-based input 210. This data, along with the NL-based input 210 and the response, may be stored in an LLM log database 131A and / or an example input data database 151A, and retrieved accordingly.

[0102] Training instance database 132A may include, for example, information about... Figure 2D The described generation and optionally further acquisition in any other way (e.g., to relate to...) Figure 4A Training instances (in the same or similar manner as described in 4B, such as manual generation of training instances, etc.). In this way, labeled training data can be generated without manual human labeling. This can significantly reduce the resources required to obtain training data for fine-tuning the LLM (e.g., by the NL-based response system 120).

[0103] Once training instances have been generated in this manner, the NL-based response system 120 (or its LLM) can be fine-tuned (or otherwise referred to as trained) using the training instances stored in the training instance database 132A (e.g., using training engine 132). This can be performed in any suitable manner (e.g., supervised learning, reinforcement learning, etc.).

[0104] For example, for temporary reference Figure 5 This describes an example process flow 500 for fine-tuning an LLM (e.g., utilized by an NL-based response system 120). Figure 5 As shown, NL-based input 512 and training instance response 514 (which can be retrieved, for example, from the training instance database 132A) can be obtained from a specific training instance 510. For example, as per [reference to...] Figure 2A and Figure 2B As described, a selected response 520 can be generated using a NL-based response system 120 (or its LLM) based on processing NL-based input 512. The selected response 520 can be compared with training instance responses 514 to generate a training loss 530. Comparing the selected response 520 with the training instance responses 514 can include, for example, lexicalization, natural language understanding (NLU), natural language processing (NLP), etc. For example, instead of comparing the responses themselves, embeddings generated based on the responses (in any suitable manner) can be compared (e.g., in the embedding space or another lower-level latent space representation) to generate the training loss 530. Furthermore, the LLM can be updated based on the training loss 530.

[0105] In some implementations, additional data, such as a comment response 516 corresponding to the training instance response 514, may also be obtained from the training instance 510. The comment response 516 can be included in the comparison between the selected response 520 and the training instance response 514. The comment response 516 can be used to provide weighting to the comparison when generating the training loss 530. For example, if the comment response 516 indicates that the training instance response 514 has very high quality (e.g., if, for example, the training instance response is determined to conform to all the instructions identified in the NL-based input 512 based on the training instance response score), the difference between the selected response 520 and the training instance response 514 can be propagated to a greater extent (e.g., by determining a larger training loss 530, by giving the training loss 530 greater significance during training, etc.), and vice versa. As another example, the comment response 516 can be used to train a separate reward model for use when fine-tuning an LLM using reinforcement learning (e.g., utilized by the NL-based response system 120).

[0106] Once the LLM (e.g., utilized by the NL response system 120) has been fine-tuned, the fine-tuned LLM can be deployed for use in generating responses to NL-based inputs. In some cases, the fine-tuned LLM can be used to update the NL-based response system 120 for inference (e.g., regarding...). Figure 2C (as described) or used for further generating training data and fine-tuning (e.g., to describe) Figure 2D (The manner described).

[0107] Although it is generally described that training instances can be generated using self-evaluation and progressive refinement together (e.g., as discussed in...), Figure 2D (As described), but it should be understood that training instances can also be generated in other ways, for example, those that do not include progressive refinement. For example, now turning to Figure 4A The example process flow 400A is described according to various implementations for generating training instances by modifying the NL-based input 410 for the NL-based response system 120.

[0108] It can be done in any suitable way (e.g., with regard to) Figure 2D The NL-based input 410 and corresponding response 420 are obtained in the same or similar manner as described. For example, the NL-based input 410 (and optionally the response 420) can be retrieved from an LLM log database 131A and / or an example input data database 151A. The response 420 can be evaluated against a first set of instructions 412 included in the NL-based input 410 to generate a comment response 430 (e.g., to comment on...). Figure 2A or Figure 2D (In the same or similar manner as described). Comment response 430 may indicate whether response 420 conforms to each instruction in the first instruction set 412. For example, comment response 430 may include an indication that response 420 has been determined to have violated at least one instruction in the first instruction set 412.

[0109] Based on comment response 430, NL-based input 410 can be modified, for example, using input modification engine 171 to generate a modified NL-based input 414 including a second instruction set 416. Specifically, NL-based input 410 can be modified such that response 420 conforms to each instruction in the second instruction set. This modification can be performed in any suitable manner. For example, in some implementations, NL-based input 410 can be modified to simply remove any instruction in the first instruction set 412 that response 420 is determined to violate (e.g., based on comment response 430). In some additional or alternative implementations, response 420 can be determined to violate instructions related to response statistics. In this case, NL-based input can be modified such that the second instruction set specifies the response statistics exhibited by response 420. As a non-limiting example, NL-based input 410 can include instructions for generating a response that includes up to five words. It can be determined (e.g., using comment response 430) that response 420 includes six words and therefore does not conform to the instructions for including up to five words. Therefore, the NL-based input 410 can be modified to include instructions for including up to six words instead.

[0110] In some implementations, a comment response generation engine 152 can be used, for example, to work with the "About" section. Figure 2A The response 420 is evaluated against the second instruction set in the same or similar manner as described. In this way, it can be verified that the response 420 actually conforms to each instruction in the second instruction set.

[0111] Once the NL-based input 410 has been modified (and optionally validated), the modified NL-based input 410 can be stored as a training instance in the training instance database 132A along with the response 420 (and optionally also with the corresponding comment response generated for validation, for example).

[0112] In some implementations, the LLM can be fine-tuned at box 470 using training instances stored in the training instance database 132A. This can be done in any suitable manner, for example, with reference to... Figure 5 It is performed in the same or similar manner as described. Similarly, as regarding Figure 5 As described, after fine-tuning, the fine-tuned LLM can be deployed for inference or for generating further training data.

[0113] Turning Figure 4B An example process flow 400B is described for generating training instances by modifying the response 420 to an NL-based input 410 generated by an NL-based response system. Figure 4B The operation is largely related to... Figure 4AThe operations described are similar. However, instead of executing NL-based inputs, the response 420 can be modified, for example, using a response modification engine 172.

[0114] Specifically, response 420 can be modified based on identifying which instruction in the first instruction set 412 the response violates. Then, response 420 can be modified such that the modified response 422 conforms to each instruction in the first instruction set 412. For example, in some cases, NL-based input 410 may include instructions related to statistics of response 420. Therefore, comment response 430 may indicate that response 420 does not conform to instructions related to response statistics. Therefore, response 420 can be modified such that the modified response 422 conforms to instructions related to response statistics. As a non-limiting example, suppose NL-based input 410 includes an instruction that response 420 should include no more than ten words. Further suppose response 420 includes twenty words, and comment response 430 therefore indicates that response 420 does not conform to that instruction. In this case, the response can be modified such that any words after the tenth word are deleted, so that the modified response 422 will conform to the instruction to include no more than ten words.

[0115] In some implementations, a comment response generation engine 152 can be used, for example, to work with the "About" section. Figure 2A The modified response 422 is evaluated against the first instruction set 412 in a manner similar to or the manner described. In this way, it can be verified that the modified response 422 actually conforms to each instruction in the first instruction set 412.

[0116] Once the response has been modified (and optionally validated), the NL-based input 410 can be stored as a training instance in the training instance database 132A along with the modified response 422 (and optionally also with the corresponding comment response generated for validation, for example).

[0117] In this way, due to the fact that... Figure 4A and Figure 4B The described method uses self-evaluation to generate high-quality examples of responses to a given NL-based input for use as training data, without, for example, manually labeling or creating training data, or manually modifying the NL-based input and / or response. Furthermore, regarding... Figure 4A and Figure 4B The described method can generate labeled training data in a relatively low-cost manner in terms of computational resource consumption. This is especially true for relatively minor modifications that can produce corresponding NL-based input-response pairs without relatively simple instructions that require, for example, further inference using LLM.

[0118] Turn now Figure 6AA flowchart illustrating an example method 600A for generating a response to an NL-based input associated with a client device using a NL-based response system is depicted. For convenience, the operation of method 600A is described with reference to a system performing the operation. Such a system of method 600A includes a computing device (e.g., Figure 1 Client device 110 Figures 1 to 5 NL-based response system 120 Figure 9 The computing device 910, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 600A are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0119] At box 602, the system receives NL-based input associated with the client device (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0120] At box 604, the system uses LLM-based processing of NL-based input to generate multiple candidate LLM responses, where the multiple candidate LLM responses together form a candidate LLM response set (e.g., with respect to...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0121] At box 606, the system generates a corresponding comment response for each candidate LLM response among multiple candidate LLM responses (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner described herein, and / or other manner as described herein). Corresponding comment responses may be generated using LLM based on processing multiple candidate LLM responses. Each corresponding comment response may include an indication of the degree to which the corresponding candidate LLM response conforms to the set of instructions included in the NL-based input.

[0122] At box 608, the system progressively refines the candidate LLM responses until one or more termination criteria are met (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C(In the same or similar manner as described herein, and / or in other manner as described herein). For example, the system may determine whether one or more termination criteria are met. When it is determined that one or more termination criteria are not met, the system may progressively refine the candidate LLM response set using LLM based on the candidate LLM responses in the candidate response set until it is determined that one or more termination criteria are met. When it is determined that one or more termination criteria are met (e.g., before or during refinement), the system may proceed to box 610.

[0123] At box 610, the system selects candidate LLM responses from the candidate response set based on the corresponding comment response (e.g., based on the comment response). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0124] At box 612, the system causes the selected candidate LLM response to be rendered on the client device (to reflect the context). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0125] Turn now Figure 6B A flowchart illustrating an example method 600B for generating training instances by utilizing a response system based on NL to generate responses to NL-based inputs is depicted. For convenience, the operation of method 600B is described with reference to a system performing this operation. Such a system of method 600B includes a computing device (e.g., Figure 1 Client device 110 Figures 1 to 5 NL-based response system 120 Figure 9 The computing device 910, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 600B are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0126] At box 630, the system receives NL-based input (e.g., in terms of...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0127] At box 632, the system uses LLM-based processing of NL-based input to generate multiple candidate LLM responses, where the multiple candidate LLM responses together form a candidate LLM response set (e.g., with respect to...). Figure 2A , Figure 2B , Figure 2C(The same or similar manner as described herein, and / or other manner as described herein).

[0128] At box 634, the system generates a corresponding comment response for each candidate LLM response among multiple candidate LLM responses (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner described herein, and / or other manner as described herein). Corresponding comment responses may be generated using LLM based on processing multiple candidate LLM responses. Each corresponding comment response may include an indication of the degree to which the corresponding candidate LLM response conforms to the set of instructions included in the NL-based input.

[0129] At box 636, the system progressively refines the candidate LLM responses until one or more termination criteria are met (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner described herein, and / or other manner as described herein). For example, the system may determine whether one or more termination criteria are met. When it is determined that one or more termination criteria are not met, the system may progressively refine the candidate LLM response set using LLM based on the candidate LLM responses in the candidate response set until it is determined that one or more termination criteria are met. When it is determined that one or more termination criteria are met (e.g., before or during refinement), the system may proceed to box 638.

[0130] At box 638, the system selects candidate LLM responses from the candidate response set based on the corresponding comment response (e.g., based on the comment response). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0131] At box 640, the system stores the NL-based input along with the second candidate LLM response as instances of training data for fine-tuning the LLM (e.g., with respect to...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0132] Turn now Figure 7A A flowchart illustrating an example method 700A for generating training instances by modifying the NL-based input to a NL-based response system is depicted. For convenience, the operation of method 700A is described with reference to a system performing the operation. Such a system for method 700A includes a computing device (e.g., Figure 1 Client device 110 Figures 1 to 5 NL-based response system 120 Figure 9 The computing device 910, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 700A are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0133] At box 702, the system receives NL-based input (e.g., in terms of...). Figure 4A (The same or similar manner as described herein, and / or other manner as described herein).

[0134] At box 704, the system obtains an LLM response, which has been generated using LLM-based processing of NL-based input (e.g., with respect to...). Figure 4A (The same or similar manner as described herein, and / or other manner as described herein).

[0135] At box 706, the system generates a comment response for the LLM response. The comment response for the LLM response is generated using LLM-based processing of the LLM response (e.g., regarding...). Figure 4A (In the same or similar manner as described herein, and / or in other manner as described herein). NL-based input may include a first instruction set, and the comment response may include an indication that the LLM response has been determined to violate at least one instruction in the first instruction set.

[0136] At box 708, the system can modify the NL-based input such that the modified NL-based input includes a second instruction set (e.g., regarding...). Figure 4A (in the same or similar manner as described herein, and / or in other manner as described herein), wherein the LLM response conforms to each instruction in the second instruction set.

[0137] At box 710, the system can store the modified NL-based input and LLM response as instances of the training data (e.g., to match with...). Figure 4A (The same or similar manner as described herein, and / or other manner as described herein).

[0138] Turn now Figure 7B A flowchart illustrating an example method 700B for generating training instances by modifying the response to an NL-based input generated by a NL-based response system is depicted. For convenience, the operation of method 700B is described with reference to a system performing this operation. Such a system of method 700B includes a computing device (e.g., Figure 1 Client device 110 Figures 1 to 5 NL-based response system 120 Figure 9The computing device 910, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 700B are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0139] At box 720, the system receives NL-based input (e.g., in terms of...). Figure 4B (The same or similar manner as described herein, and / or other manner as described herein).

[0140] At box 722, the system obtains an LLM response, which has been generated using LLM-based processing of NL-based input (e.g., with respect to...). Figure 4B (The same or similar manner as described herein, and / or other manner as described herein).

[0141] At box 724, the system generates a comment response for the LLM response. The comment response for the LLM response is generated using LLM-based processing of the LLM response (e.g., regarding...). Figure 4B (In the same or similar manner as described herein, and / or in other manner as described herein). NL-based input may include a set of instructions, and a comment response may include an indication that the LLM response has been determined to have violated at least one instruction in the instruction set.

[0142] At box 726, the system can modify the LLM response such that the modified LLM response conforms to at least one instruction (e.g., to match the instruction regarding...). Figure 4B (The same or similar manner as described herein, and / or other manner as described herein).

[0143] At box 728, the system can store the NL-based input and the modified LLM response as instances of training data (e.g., to match...). Figure 4B (The same or similar manner as described herein, and / or other manner as described herein).

[0144] Turn now Figure 8 A flowchart illustrating an example method 800 for generating a response to an NL-based input associated with a client device using a NL-based response system is depicted. For convenience, the operation of method 800 is described with reference to a system performing the operation. Such a system of method 800 includes a computing device (e.g., Figure 1 Client device 110 Figures 1 to 5 NL-based response system 120 Figure 9The computing device 910, one or more servers and / or other computing devices, and one or more processors, memories and / or other components. Furthermore, although the operations of method 800 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0145] At box 802, the system receives NL-based input associated with the client device (e.g., regarding...). Figure 2A , Figure 2B , Figure 2C (The same or similar manner as described herein, and / or other manner as described herein).

[0146] At block 804, the system determines the set of instructions to be generated in response to receiving an NL-based input based on the content included in the NL-based input.

[0147] At box 804, the system uses LLM-based processing of NL-based inputs to generate LLM outputs.

[0148] At box 806, the system determines candidate LLM responses to NL-based inputs based on the LLM output.

[0149] At box 808, the system determines whether the candidate LLM response satisfies each instruction included in the instruction set for LLM responses.

[0150] In some implementations, to determine whether a candidate LLM response satisfies each instruction included in the instruction set for an LLM response, the system can use the LLM-based processing of NL-based inputs and instruction sets to generate a comment response indicating whether the candidate LLM response satisfies each instruction included in the instruction set for an LLM response. The system can also determine whether a candidate LLM response satisfies each instruction included in the instruction set for an LLM response based on the comment response.

[0151] In some implementations, the system may cause the candidate LLM response to be rendered at the client device in response to determining that the candidate LLM response satisfies each of the instructions included in the instruction set for the LLM response.

[0152] At box 810, the system determines that the candidate LLM response does not satisfy each instruction in the instruction set included for LLM responses.

[0153] At box 812, in response to determining that a candidate LLM response fails to satisfy one or more instructions included in the instruction set for LLM responses, the system uses the LLM to generate additional LLM output based on at least processing one or more instructions for which the candidate LLM response failed to satisfy.

[0154] In some implementations, LLM is used to generate additional LLM outputs based on candidate LLM responses that fail to satisfy one or more instructions included in the instruction set for LLM responses.

[0155] At box 814, the system determines alternative candidate LLM responses that also respond to NL-based inputs based on the additional LLM output.

[0156] At box 816, the system determines whether the alternative candidate LLM response satisfies each instruction included in the instruction set for the LLM response.

[0157] At box 818, in response to determining that the alternative candidate LLM response satisfies each of the instructions included in the instruction set for the LLM response, the system causes the alternative candidate LLM response to be rendered at the client device.

[0158] In some implementations, the system may determine additional candidate LLM responses that also respond to NL-based inputs based on the LLM output. In these implementations, the system may determine whether the additional candidate LLM responses satisfy each instruction included in the instruction set for the LLM response. In response to determining that the additional candidate LLM responses satisfy each instruction included in the instruction set for the LLM response, the system may suppress the generation of additional LLM outputs. The system may also cause the additional candidate LLM responses to be rendered at the client device instead of the candidate LLM responses.

[0159] In some of these implementations, generating additional LLM output further responds to the system determining that the additional candidate LLM response fails to satisfy one or more instructions included in the instruction set for LLM responses.

[0160] In some implementations, the system may determine whether to continue incremental refinement based on the NL-based input before determining the set of instructions to be generated in response to the received NL-based input for the LLM response. In these implementations, determining the set of instructions to be generated in response to the received NL-based input is in response to determining to continue incremental refinement.

[0161] Although method 800 is generally described in relation to generating responses to NL-based inputs associated with a client device using an NL-based response system, it should be understood that many operations in method 800 can also be used when generating training data for fine-tuning the LLM. For example, not the operations described in box 802, but rather those that can be described, for example, in relation to Figure 2CThe NL-based input is obtained in the same or similar manner as described herein and / or in any other manner as described herein. Furthermore, not regarding the operations described in box 818, but rather the possibility of storing alternative candidate LLM responses along with the NL-based input as instances of training data for fine-tuning the LLM (e.g., in relation to…). Figure 2C (In the same or similar manner as described herein, and / or in other manner as described herein).

[0162] Turn now Figure 9 This diagram depicts a block diagram of an example computing device 910 that can be optionally utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automation assistant component, or other cloud-based software application component and / or other components may include one or more components of the example computing device 910.

[0163] The computing device 910 typically includes at least one processor 914 that communicates with a plurality of peripheral devices via a bus subsystem 912. These peripheral devices may include a storage subsystem 924 (including, for example, a memory subsystem 925 and a file storage subsystem 926), a user interface output device 920, a user interface input device 922, and a network interface subsystem 916. The input and output devices allow user interaction with the computing device 910. The network interface subsystem 916 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0164] User interface input device 922 may include a keyboard, pointing devices (such as a mouse, trackball, touchpad, or graphics tablet), scanner, touchscreen integrated into a display, audio input devices (such as a voice recognition system, microphone), and / or other types of input devices. Generally, the term "input device" is intended to include all possible types of means and methods for inputting information into computing device 910 or into a communication network.

[0165] User interface output device 920 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is intended to encompass all possible types of means and methods for outputting information from computing device 910 to a user or another machine or computing device.

[0166] Storage subsystem 924 stores the functional programming and data constructs of some or all of the modules described herein. For example, storage subsystem 924 may include selected aspects and implementations for performing the methods disclosed herein. Figure 1 The logic of the various components described in the document.

[0167] These software modules are typically executed by processor 914 alone or in conjunction with other processors. The memory 925 used in storage subsystem 924 may include multiple memories, including main random access memory (RAM) 930 for storing instructions and data during program execution and read-only memory (ROM) 932 for storing fixed instructions therein. File storage subsystem 926 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing certain functionalities of the implementation may be stored in file storage subsystem 926 within storage subsystem 924, or in other machines accessible to processor 914.

[0168] Bus subsystem 912 provides mechanisms for enabling various components and subsystems of computing device 910 to communicate with each other as intended. Although bus subsystem 912 is schematically shown as a single bus, alternative implementations of bus subsystem 912 may use multiple buses.

[0169] The computing device 910 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, [the following applies]. Figure 9 The description of the computing device 910 depicted herein is intended only as a specific example to illustrate some implementation methods. Many other configurations of the computing device 910 are possible, and these configurations are related to… Figure 9 The computing devices described in the document have more or fewer components compared to those in the document.

[0170] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be provided with the opportunity to control whether programs or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content more relevant to the user is received from content servers. Furthermore, certain data may be processed in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or, where geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level) to the point that the user's specific geographic location cannot be determined. Therefore, users can control how information about themselves is collected and / or used.

[0171] In some implementations, a method implemented by one or more processors is provided, the method comprising: receiving NL-based input associated with a client device; generating a plurality of candidate LLM responses using LLM-based processing of the NL-based input. The plurality of candidate LLM responses together form a candidate LLM response set. The method further comprises: generating a corresponding comment response for each of the plurality of candidate LLM responses, the corresponding comment response being generated using LLM-based processing of the plurality of candidate LLM responses. Each corresponding comment response includes an indication of the degree to which the corresponding candidate LLM response conforms to the instruction set included in the NL-based input. The method further comprises: determining whether one or more termination criteria are met. When it is determined that one or more termination criteria are not met, the method further comprises: progressively refining the candidate LLM responses using LLM-based processing of the candidate response set until it is determined that one or more termination criteria are met. When it is determined that one or more termination criteria are met, the method further comprises: selecting a candidate LLM response from the candidate response set based on the corresponding comment response; and causing the selected candidate LLM response to be rendered at the client device.

[0172] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0173] In some implementations, progressively refining candidate LLM responses may include: repeatedly performing the following operations until one or more termination criteria are determined to be met: selecting a first candidate LLM response from the candidate LLM response set; generating multiple refined candidate LLM responses using LLM-based processing of the first candidate LLM response and its corresponding comment response; adding the multiple refined candidate LLM responses to the candidate LLM response set; and generating a corresponding comment response for each of the multiple refined candidate LLM responses, wherein the corresponding comment response for each of the multiple refined candidate LLM responses is generated using LLM-based processing of the multiple refined candidate LLM responses. Each corresponding comment response for each of the multiple refined candidate LLM responses may include an indication of the degree to which the corresponding refined candidate LLM response conforms to the set of instructions included in the NL-based input.

[0174] In some versions of these implementations, an indicator of the degree to which a candidate LLM response conforms to the instruction set can indicate which instruction in the instruction set the corresponding LLM response is determined to conform to.

[0175] In additional or alternative versions of these implementations, the first candidate LLM response may be selected from the set of candidate LLM responses based on: determining that the first candidate LLM response best conforms to the instruction set in the set of candidate LLM responses; and determining that the first candidate LLM response has not been previously selected as the first candidate LLM response.

[0176] In additional or alternative versions of these implementations, the first candidate LLM response can be selected from a set of candidate LLM responses based on a search priority queue. The priority queue may include a set of candidate LLM responses, and each candidate LLM response in the priority queue may be associated with a priority based on the corresponding comment response.

[0177] In some implementations, one or more termination criteria may include one or more of the following: threshold repetition count, threshold execution time, threshold processor cycle, and threshold memory utilization. When one or more termination criteria are met, the selected candidate LLM response may be selected based on determining that the selected candidate LLM response best conforms to the instruction set among the candidate LLM response set.

[0178] In additional or alternative versions of these implementations, one or more termination criteria may include a threshold degree to which a given candidate LLM in the candidate LLM response set conforms to the instruction set. When a particular candidate LLM in the candidate LLM response set is determined to conform to the instruction set to a degree exceeding the threshold degree based on the corresponding comment response, that particular candidate LLM response may be selected as the chosen candidate LLM response.

[0179] In some implementations, generating a corresponding comment response for each candidate LLM response may include: generating a request for the LLM to identify the set of instructions included in the NL-based input and to determine the degree to which each candidate LLM response conforms to the set of instructions; and using the LLM processing request and the NL-based input to identify the set of instructions included in the NL-based input and generate a corresponding comment response.

[0180] In some implementations, generating multiple refined candidate LLM responses may include: generating a request to refine a first candidate LLM response based on a corresponding comment response for the LLM. The request to refine the first candidate LLM response for the LLM may include instructions from a set of instructions that the first candidate LLM response is determined to have violated based on the corresponding comment response; and the method may further include processing the request using the LLM to generate multiple refined LLM responses.

[0181] In some implementations, the method may further include determining whether to continue incremental refinement based on one or both of the following: context data associated with the client device and / or the user of the client device, and NL-based input.

[0182] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining NL-based input; generating a plurality of candidate LLM responses using LLM-based processing of the NL-based input. The plurality of candidate LLM responses together form a candidate LLM response set. The method may further comprise: generating a corresponding comment response for each of the plurality of candidate LLM responses, the corresponding comment response being generated using LLM-based processing of the plurality of candidate LLM responses. Each corresponding comment response includes an indication of the degree to which the corresponding candidate LLM response conforms to the instruction set included in the NL-based input. The method further comprises: determining whether one or more termination criteria are met. When it is determined that one or more termination criteria are not met, the method further comprises: progressively refining the candidate LLM responses using LLM-based processing of the candidate response set until it is determined that one or more termination criteria are met. When it is determined that one or more termination criteria are met, the method further comprises: selecting a candidate LLM response from the candidate response set based on the corresponding comment response; and storing the NL-based input together with the selected candidate LLM response as instances of training data for fine-tuning the LLM.

[0183] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0184] In some implementations, an indicator of the degree to which a candidate LLM response conforms to the instruction set can indicate which instruction in the instruction set the corresponding LLM response is determined to conform to.

[0185] In some additional or alternative implementations, progressively refining candidate LLM responses may include: repeatedly performing the following operations until one or more termination criteria are determined to be met: selecting a first candidate LLM response from the candidate LLM response set; generating multiple refined candidate LLM responses using LLM-based processing of the first candidate LLM response and its corresponding comment response; adding the multiple refined candidate LLM responses to the candidate LLM response set; and generating a corresponding comment response for each of the multiple refined candidate LLM responses, wherein the corresponding comment response for each of the multiple refined candidate LLM responses is generated using LLM-based processing of the multiple refined candidate LLM responses. Each corresponding comment response for each of the multiple refined candidate LLM responses may include an indication of the degree to which the corresponding refined candidate LLM response conforms to the set of instructions included in the NL-based input.

[0186] In some implementations, the first candidate LLM response may be selected from the set of candidate LLM responses based on: determining that the first candidate LLM response best conforms to the instruction set in the set of candidate LLM responses; and determining that the first candidate LLM response has not been previously selected as the first candidate LLM response.

[0187] In some implementations, the first candidate LLM response can be selected from a set of candidate LLM responses based on a search priority queue. The priority queue may include a set of candidate LLM responses, and each candidate LLM response in the priority queue may be associated with a priority based on the corresponding comment response.

[0188] In some implementations, one or more termination criteria may include one or more of the following: threshold repetition count, threshold execution time, threshold processor cycle, and threshold memory utilization. When one or more termination criteria are met, the selected candidate LLM response may be selected based on determining that the selected candidate LLM response best conforms to the instruction set among the candidate LLM response set.

[0189] In some additional or alternative implementations, one or more termination criteria may include a threshold degree to which a given candidate LLM in the candidate LLM response set conforms to the instruction set. When the degree to which a particular candidate LLM in the candidate LLM response set conforms to the instruction set exceeds the threshold degree based on the corresponding comment response, that particular candidate LLM response may be selected as the chosen candidate LLM response.

[0190] In some implementations, generating a comment response corresponding to a set of candidate LLM responses may include: generating a request to identify the set of instructions included in the NL-based input for the LLM and to determine the degree to which each candidate LLM response conforms to the set of instructions; and using the LLM processing request and the NL-based input to identify the set of instructions included in the NL-based input and to generate a corresponding comment response.

[0191] In some implementations, generating multiple refined candidate LLM responses may include: generating a request to refine a first candidate LLM response based on a corresponding comment response for the LLM. The request to refine the first candidate LLM response for the LLM may include instructions from a set of instructions that the first candidate LLM response is determined to have violated based on the corresponding comment response; and the method may further include processing the request using the LLM to generate multiple refined LLM responses.

[0192] In some implementations, obtaining NL-based input may include identifying a specific NL-based input from one or more databases based on: obtaining the specific NL-based input and an LLM response; and using the LLM response to generate a corresponding comment response. The corresponding comment response may include an LLM response that violates the instruction of at least one instruction in the instruction set included in the specific NL-based input.

[0193] In some implementations, the method may further include: fine-tuning the LLM based on training data.

[0194] In some versions of the implementation, the method may further include: after fine-tuning the LLM, receiving NL-based input associated with a client device; generating an LLM response using the fine-tuned LLM based on processing the NL-based input associated with the client device; and causing the LLM response to be rendered at the client device.

[0195] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining NL-based input; obtaining an LLM response generated using LLM-based processing of the NL-based input; and generating a comment response for the LLM response, the comment response being generated using LLM-based processing of the LLM response. The NL-based input includes a first instruction set, and the comment response includes an indication that the LLM response is determined to violate at least one instruction in the first instruction set. The method further comprises: modifying the NL-based input such that the modified NL-based input includes a second instruction set. The LLM response conforms to each instruction in the second instruction set. The method further comprises: storing the modified NL-based input and the LLM response as instances of training data.

[0196] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0197] In some implementations, the method may further include: generating additional comment responses for the LLM responses before storing the modified NL-based input and LLM response as instances of training data, the additional comment responses being generated using LLM-based processing of the LLM responses. The additional comment responses may include indications that the LLM response conforms to each instruction in a second set of instructions included in the modified NL-based input. The method may further include: verifying the modified NL-based input based on the indications; and determining, based on the verification, whether to store the modified NL-based input and LLM response as instances of training data.

[0198] In some implementations, at least one instruction in the first instruction set may include a first requirement for the LLM response, comprising a first threshold number of characters, sentences, lines, and / or paragraphs. The LLM response does not meet the first requirement, and a second instruction set may include a second requirement for the LLM response, comprising a second threshold number of characters, sentences, lines, and / or paragraphs, and the LLM response meets the second requirement.

[0199] In some implementations, a method implemented by one or more processors is provided, the method comprising: obtaining NL-based input; obtaining an LLM response generated using LLM-based processing of the NL-based input; generating a comment response for the LLM response, the comment response being generated using LLM-based processing of the LLM response, wherein the NL-based input includes an instruction set, and the comment response includes an indication that the LLM response is determined to have violated at least one instruction in the instruction set. The method further comprises: modifying the LLM response such that the modified LLM response conforms to at least one instruction; and storing the NL-based input and the modified LLM response as instances of training data.

[0200] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0201] In some implementations, the method may further include: before storing the NL-based input and the modified LLM response as instances of training data: generating an additional comment response for the modified LLM response, wherein the additional comment response for the modified LLM response may be generated using LLM-based processing of the modified LLM response, and the additional comment response may include an indication that the modified LLM response conforms to at least one instruction. The method may further include: verifying the modified NL-based input based on the indication; and determining, based on the verification, whether to store the NL-based input and the modified LLM response as instances of training data.

[0202] In some implementations, at least one instruction may include a requirement for the LLM response, including a maximum number of characters, sentences, lines, and / or paragraphs. If the LLM response does not meet the first requirement, modifying the LLM response may include deleting at least a portion of the LLM response to make the modified LLM response conform to the requirement.

[0203] In some implementations, a method implemented by one or more processors is provided, the method comprising: receiving NL-based input associated with a client device; determining, based on the contents included in the NL-based input, a set of instructions for an LLM response to be generated in response to the received NL-based input; generating an LLM output using the LLM based on processing the NL-based input; determining, based on the LLM output, candidate LLM responses in response to the NL-based input; determining whether the candidate LLM responses satisfy each instruction included in the instruction set for the LLM responses; in response to determining that the candidate LLM responses fail to satisfy one or more instructions included in the instruction set for the LLM responses: generating additional LLM output using the LLM based on at least processing one or more instructions that the candidate LLM responses failed to satisfy; and determining, based on the additional LLM output, alternative candidate LLM responses that also respond to the NL-based input; determining whether the alternative candidate LLM responses satisfy each instruction included in the instruction set for the LLM responses; and in response to determining that the alternative candidate LLM responses satisfy each instruction included in the instruction set for the LLM responses: causing the alternative candidate LLM responses to be rendered at the client device.

[0204] These and other implementations of the techniques disclosed herein may optionally include one or more of the following features.

[0205] In some implementations, determining whether a candidate LLM response satisfies each instruction included in the instruction set for an LLM response may include: using the LLM-based processing of NL-based inputs and instruction sets to generate a comment response indicating whether the candidate LLM response satisfies each instruction included in the instruction set for an LLM response; and determining whether the candidate LLM response satisfies each instruction included in the instruction set for an LLM response based on the comment response.

[0206] In some implementations, the method may further include: determining additional candidate LLM responses based on the LLM output, which also respond to the NL-based input; determining whether the additional candidate LLM responses satisfy each instruction included in the instruction set for the LLM response; and in response to determining that the additional candidate LLM responses satisfy each instruction included in the instruction set for the LLM response: suppressing the generation of additional LLM output; and causing the additional candidate LLM responses to be rendered at the client device in place of the candidate LLM responses.

[0207] In some versions of these implementations, generating additional LLM output may further respond to determining that an additional candidate LLM response fails to satisfy one or more instructions included in the instruction set for LLM responses.

[0208] In some implementations, the method may further include: before determining the set of instructions for an LLM response to be generated in response to receiving the NL-based input based on the NL-based input: determining whether to continue incremental improvement based on the NL-based input, and determining that the set of instructions for an LLM response to be generated in response to receiving the NL-based input may be in response to determining to continue incremental improvement.

[0209] In some implementations, using LLM to generate additional LLM output is further based on processing candidate LLM responses that fail to satisfy one or more instructions included in the instruction set for LLM responses.

[0210] In some implementations, the method may further include: in response to determining that a candidate LLM response satisfies each of the instructions included in the instruction set for an LLM response: causing the candidate LLM response to be rendered at the client device.

[0211] Furthermore, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some implementations also include one or more computer-readable storage media (e.g., transient and / or non-transient) storing computer instructions executable by one or more processors to perform any of the methods described above. Some implementations also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

Claims

1. A method implemented by one or more processors, the method comprising: Receive natural language (NL) based input associated with the client device; Large language model LLM is used to generate multiple candidate LLM responses based on processing NL-based input, wherein the multiple candidate LLM responses together form a candidate LLM response set; For each of the plurality of candidate LLM responses, a corresponding comment response is generated, wherein the corresponding comment response is generated using the LLM based on processing the plurality of candidate LLM responses, and wherein each of the corresponding comment responses includes an indication of the degree to which the corresponding candidate LLM response conforms to the instruction set included in the NL-based input; Determine whether one or more termination criteria are met; When it is determined that one or more of the termination criteria are not met: The LLM is used to progressively refine the candidate LLM response based on processing the candidate LLM responses in the candidate response set until it is determined that one or more termination criteria are met. as well as When it is determined that one or more of the termination criteria are met: Candidate LLM responses are selected from the candidate response set based on the corresponding comment responses; as well as This causes the selected candidate LLM response to be rendered at the client device.

2. The method of claim 1, wherein progressively refining the candidate LLM response comprises: Repeat the following operations until one or more of the termination criteria are determined to be met: Select the first candidate LLM response from the candidate LLM response set; The LLM is used to generate multiple refined candidate LLM responses based on the processing of the first candidate LLM response and the corresponding comment response; Add the plurality of refined candidate LLM responses to the candidate LLM response set; as well as A corresponding comment response is generated for each of the plurality of refined candidate LLM responses. The corresponding comment response for each of the plurality of refined candidate LLM responses is generated using the LLM based on processing the plurality of refined candidate LLM responses. Each corresponding comment response for each of the plurality of refined candidate LLM responses includes an indication of the degree to which the corresponding refined candidate LLM response conforms to the instruction set included in the NL-based input.

3. The method of claim 1 or claim 2, wherein the indication of the degree to which the corresponding candidate LLM response conforms to the instruction set indicates that the corresponding LLM response in the instruction set is determined to conform to the instruction.

4. The method of claim 2, wherein the first candidate LLM response is selected from the candidate LLM response set based on the following operation: Determine that the first candidate LLM response best conforms to the instruction set within the candidate LLM response set; and It is determined that the first candidate LLM response has not been previously selected as the first candidate LLM response.

5. The method of claim 2, wherein the first candidate LLM response is selected from the candidate LLM response set based on a search priority queue, wherein the priority queue includes the candidate LLM response set, and wherein each candidate LLM response in the candidate LLM response set in the priority queue is associated with a priority based on the corresponding comment response.

6. The method of claim 2, wherein the one or more termination criteria include one or more of the following: threshold repetition count, threshold execution time, threshold processor loop count, and threshold memory utilization, and wherein when the one or more termination criteria are met, the selected candidate LLM response is selected based on determining that the selected candidate LLM response best conforms to the instruction set in the set of candidate LLM responses.

7. The method of claim 2, wherein the one or more termination criteria include a threshold degree to which a given candidate LLM in the candidate LLM response set conforms to the instruction set, and wherein when it is determined, based on the corresponding comment response, that a particular candidate LLM in the candidate LLM response set conforms to the instruction set to a degree exceeding the threshold degree, the particular candidate LLM response is selected as the selected candidate LLM response.

8. The method of claim 2, wherein generating the plurality of refined candidate LLM responses comprises: Generate a request to improve the first candidate LLM response based on the corresponding comment response for the LLM, wherein the request to improve the first candidate LLM response for the LLM includes an indication of an instruction in the instruction set that the first candidate LLM response is determined to have violated based on the corresponding comment response; as well as The request is processed using the LLM to generate the plurality of refined LLM responses.

9. The method of any preceding claim, wherein generating the corresponding comment response for each of the plurality of candidate LLM responses comprises: The request to generate identification for the LLM includes a set of instructions in the NL-based input and to determine the degree to which each candidate LLM response conforms to the set of instructions. as well as The LLM is used to process the request and the NL-based input to identify the set of instructions included in the NL-based input and generate the corresponding comment response.

10. The method of any of the preceding claims, further comprising: Whether to continue incremental improvement will be determined based on one or both of the following: Context data associated with the client device and / or the user of the client device, and The input based on NL.

11. A method implemented by one or more processors, the method comprising: Obtain input based on natural language (NL); Large language model LLM is used to generate multiple candidate LLM responses based on processing NL-based input, wherein the multiple candidate LLM responses together form a candidate LLM response set; For each of the plurality of candidate LLM responses, a corresponding comment response is generated, wherein the corresponding comment response is generated using the LLM based on processing the plurality of candidate LLM responses, and wherein each of the corresponding comment responses includes an indication of the degree to which the corresponding candidate LLM response conforms to the instruction set included in the NL-based input; Determine whether one or more termination criteria are met; When it is determined that one or more of the termination criteria are not met: The LLM is used to progressively refine the candidate LLM response based on processing the candidate LLM responses in the candidate response set until it is determined that one or more termination criteria are met. as well as When it is determined that one or more of the termination criteria are met: Candidate LLM responses are selected from the candidate response set based on the corresponding comment responses; as well as The NL-based input, along with the selected candidate LLM responses, is stored as an instance of training data for fine-tuning a large language model, LLM.

12. The method of claim 11, wherein progressively refining the candidate LLM response comprises: Repeat the following operations until one or more of the termination criteria are determined to be met: Select the first candidate LLM response from the candidate LLM response set; The LLM is used to generate multiple refined candidate LLM responses based on the processing of the first candidate LLM response and the corresponding comment response; Add the plurality of refined candidate LLM responses to the candidate LLM response set; as well as A corresponding comment response is generated for each of the plurality of refined candidate LLM responses. The corresponding comment response for each of the plurality of refined candidate LLM responses is generated using the LLM based on processing the plurality of refined candidate LLM responses. Each corresponding comment response for each of the plurality of refined candidate LLM responses includes an indication of the degree to which the corresponding refined candidate LLM response conforms to the instruction set included in the NL-based input.

13. The method of claim 11 or claim 12, wherein the indication of the degree to which the corresponding candidate LLM response conforms to the instruction set indicates that the corresponding LLM response is determined to conform to an instruction in the instruction set.

14. The method of claim 12, wherein the first candidate LLM response is selected from the candidate LLM response set based on the following: Determine that the first candidate LLM response best conforms to the instruction set within the candidate LLM response set; and It is determined that the first candidate LLM response has not been previously selected as the first candidate LLM response.

15. The method of claim 12, wherein the first candidate LLM response is selected from the candidate LLM response set based on a search priority queue, wherein the priority queue includes the candidate LLM response set, and wherein each candidate LLM response in the candidate LLM response set in the priority queue is associated with a priority based on the corresponding comment response.

16. The method of claim 12, wherein the one or more termination criteria include one or more of the following: threshold number of repetitions, threshold execution time, threshold number of processor loops, and threshold memory utilization, and wherein when the one or more termination criteria are met, the selected candidate LLM response is selected based on determining that the selected candidate LLM response best conforms to the instruction set in the set of candidate LLM responses.

17. The method of claim 12, wherein the one or more termination criteria include a threshold degree to which a given candidate LLM in the candidate LLM response set conforms to the instruction set, and wherein when it is determined, based on the corresponding comment response, that a particular candidate LLM in the candidate LLM response set conforms to the instruction set to a degree exceeding the threshold degree, the particular candidate LLM response is selected as the selected candidate LLM response.

18. The method of claim 12, wherein generating the plurality of refined candidate LLM responses comprises: Generate a request to improve the first candidate LLM response based on the corresponding comment response for the LLM, wherein the request to improve the first candidate LLM response for the LLM includes an indication of an instruction in the instruction set that the first candidate LLM response is determined to have violated based on the corresponding comment response; as well as The request is processed using the LLM to generate the plurality of refined LLM responses.

19. The method of claim 11, wherein generating the corresponding comment response for each of the plurality of candidate LLM responses comprises: The request to generate identification for the LLM includes a set of instructions in the NL-based input and to determine the degree to which each candidate LLM response conforms to the set of instructions. as well as The LLM is used to process the request and the NL-based input to identify the set of instructions included in the NL-based input and generate the corresponding comment response.

20. The method of claim 11, wherein obtaining the NL-based input comprises identifying a specific NL-based input from one or more databases based on: Obtain the specific NL-based input and the LLM response; The LLM is used to generate a corresponding comment response based on the processing of the LLM response, wherein the corresponding comment response includes an indication that the LLM response violates at least one instruction in the instruction set included in the particular NL-based input.

21. The method of claim 11, further comprising fine-tuning the LLM based on the training data.

22. The method of claim 21, further comprising: After fine-tuning the LLM, NL-based input associated with the client device is received; The finely tuned LLM is used to generate an LLM response based on the NL-based input associated with the client device. as well as This causes the LLM response to be rendered at the client device.

23. A method implemented by one or more processors, the method comprising: Obtain input based on natural language (NL); Obtain the large language model LLM response, which has been generated using LLM based processing of NL-based input; A comment response is generated for the LLM response, wherein the comment response is generated using the LLM based on processing the LLM response, wherein the NL-based input includes a first instruction set, and the comment response includes an indication that the LLM response is determined to have violated at least one instruction in the first instruction set; Modify the NL-based input such that the modified NL-based input includes a second instruction set, wherein the LLM response conforms to each instruction in the second instruction set; as well as The modified NL-based input and the LLM response are stored as instances of training data.

24. The method of claim 23, further comprising, before storing the modified NL-based input and the LLM response as instances of training data: An additional comment response is generated for the LLM response, the additional comment response being generated using the LLM based on processing the LLM response, wherein the additional comment response includes an indication that the LLM response conforms to each instruction in the second instruction set included in the modified NL-based input; The modified NL-based input is validated based on the indicated instructions; as well as Based on the verification, it is determined that the modified NL-based input and the LLM response will be stored as instances of training data.

25. The method of claim 23, wherein the at least one instruction in the first instruction set includes a first requirement for the LLM response including a first threshold number of characters, sentences, lines and / or paragraphs, wherein the LLM response does not meet the first requirement, wherein the second instruction set includes a second requirement for the LLM response including a second threshold number of characters, sentences, lines and / or paragraphs, and wherein the LLM response meets the second requirement.

26. A method implemented by one or more processors, the method comprising: Obtain input based on natural language (NL); Obtain the large language model LLM response, which has been generated using LLM based processing of NL-based input; A comment response is generated for the LLM response, wherein the comment response is generated using the LLM based on processing the LLM response, wherein the NL-based input includes a set of instructions, and the comment response includes an indication that the LLM response is determined to have violated at least one instruction in the set of instructions; Modify the LLM response such that the modified LLM response conforms to at least one instruction; as well as The NL-based input and the modified LLM response are stored as instances of training data.

27. The method of claim 26, further comprising, before storing the NL-based input and the modified LLM response as instances of training data: An additional comment response is generated for the modified LLM response, the additional comment response being generated using the LLM based on processing the modified LLM response, wherein the additional comment response includes an indication that the modified LLM response conforms to at least one instruction; The modified NL-based input is validated based on the indicated instructions; as well as Based on the verification, it is determined that the NL-based input and the modified LLM response will be stored as instances of training data.

28. The method of claim 26, wherein the at least one instruction includes a requirement for the LLM response including a maximum number of characters, sentences, lines and / or paragraphs, wherein the LLM response does not meet the first requirement, and wherein modifying the LLM response includes deleting at least a portion of the LLM response such that the modified LLM response conforms to the requirement.

29. A method implemented by one or more processors, the method comprising: Receive natural language (NL) based input associated with the client device; The set of instructions for a large language model (LLM) response to be generated in response to the received NL-based input is determined based on the content included in the NL-based input. The LLM output is generated by processing the NL-based input; Candidate LLM responses to the NL-based inputs are determined based on the LLM output; Determine whether the candidate LLM response satisfies each instruction included in the instruction set for the LLM response; In response to determining that the candidate LLM response fails to satisfy one or more instructions included in the instruction set for the LLM response: The LLM is used to generate additional LLM output based on at least one or more instructions that fail to satisfy the candidate LLM response; as well as Based on the additional LLM output, alternative candidate LLM responses that also respond to the NL-based input are determined; Determine whether the alternative candidate LLM response satisfies each instruction included in the instruction set for the LLM response; as well as In response to determining that the alternative candidate LLM response satisfies each of the instructions included in the instruction set for the LLM response: This causes the alternative candidate LLM response to be rendered at the client device.

30. The method of claim 29, wherein determining whether the candidate LLM response satisfies each instruction included in the instruction set for the LLM response comprises: The LLM is used to generate a comment response based on processing the NL-based input and the instruction set, indicating whether the candidate LLM response satisfies each instruction included in the instruction set for the LLM response; and Based on the comment response, determine whether the candidate LLM response satisfies each instruction included in the instruction set for the LLM response.

31. The method of claim 29, further comprising: Based on the LLM output, additional candidate LLM responses that also respond to the NL-based input are determined; Determine whether the additional candidate LLM response satisfies each instruction included in the instruction set for the LLM response; and In response to determining that the additional candidate LLM response satisfies each of the instructions included in the instruction set for the LLM response: Suppress the generation of the additional LLM output; as well as This causes the additional candidate LLM response to be rendered at the client device in place of the candidate LLM response.

32. The method of claim 31, wherein generating the additional LLM output is in further response to determining that the additional candidate LLM response fails to satisfy one or more instructions included in the instruction set for the LLM response.

33. The method of claim 29, further comprising: Before determining the set of instructions to be generated in response to receiving the NL-based input: Based on the NL-based input, it is determined whether to continue with incremental improvement, and The instruction set determined for generating the LLM response in response to receiving the NL-based input is in response to determining to continue incremental improvement.

34. The method of claim 29, wherein using the LLM to generate the additional LLM output is further based on processing the candidate LLM response that fails to satisfy one or more instructions included in the instruction set for the LLM response.

35. The method of claim 29, further comprising: In response to determining that the candidate LLM response satisfies each of the instructions included in the instruction set for the LLM response: This causes the candidate LLM response to be rendered at the client device.

36. A system comprising: One or more hardware processors; as well as A memory that stores instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform the method as described in any one of claims 1 to 35.

37. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations according to any one of claims 1 to 35.

Images