Zoom utilization of large language models

Abstract

The present disclosure relates to efficiently receiving and processing input tasks in a scalable manner and in a manner that reduces the amount of tokens processed by a base model (e.g., LLM) and the number of API calls made when processing the input tasks. The system batch processes the set of inputs to provide it as a single batch of inputs to the LLM. The system generates one or more arrangements of the batch inputs to determine an output based on a variable order in which input data is provided within the respective arrangements of the batch inputs. The system may also eliminate one or more data inputs within a respective batch to facilitate a smaller batch of inputs without sacrificing the accuracy of an output set generated by the LLM in response to batch arrangement.

Description

Cross-references to related applications

[0001] This application claims the benefit and priority of provisional application number 63 / 579,730, filed on August 30, 2023, the entire contents of which are incorporated herein by reference. Background Technology

[0002] In recent years, the popularity and application of artificial intelligence (AI) and machine learning (ML) have increased significantly. Furthermore, as cloud-hosted services become increasingly available to end users and other organizations, access to more complex and robust computational models, such as large language models (LLMs), has become more common. These foundational models can be trained to perform a variety of tasks, such as chatbots, providing answers to general questions, generating code and other programming scripts, and, in some cases, providing specific information on specific topics.

[0003] While base models (such as ChatGPT and other large language models) provide useful tools for performing a variety of tasks using substantial computational resources, there are significant computational and processing costs associated with training these large language models to perform these tasks accurately and quickly. Furthermore, these LLMs typically require expensive, state-of-the-art infrastructure (e.g., GPUs) to host them. As these models scale upwards and as the token budget that queries and context can be fed into the LLM increases, the costs associated with running applications that utilize these LLM resources can become inefficient and computationally expensive, especially with the increasing number of application programming interface (API) calls.

[0004] These and other issues relate to the utilization and scaling of LLM and other underlying model resources. Attached Figure Description

[0005] Figure 1 An example environment including a batch prompt generation system according to one or more embodiments is shown.

[0006] Figure 2A-2C A portion of an example workflow according to one or more embodiments is shown, in which a batch prompt generation system generates a batch output set in response to multiple input prompts.

[0007] Figure 3 An example workflow illustrating the features of a batch prompt generation system according to one or more embodiments is shown.

[0008] Figure 4 A series of example actions for extracting and classifying data from digital content items, according to one or more embodiments, are shown.

[0009] Figure 5It shows some components that may be included within a computer system. Detailed Implementation

[0010] This disclosure relates to systems, methods, and computer-readable media for efficiently receiving and processing input tasks in a scalable manner and in a way that reduces the amount of tokens processed by the underlying model (e.g., LLM) and the number of application programming interface (API) calls made when processing input tasks. For example, as will be discussed in further detail below, a batch suggestion generation system batches a set of inputs to provide it as a single batch of input to an LLM (or other underlying model). The batch suggestion generation system additionally generates one or more permutations of the batch inputs to determine the output based on a variable order in which input data is provided within the respective permutations of the batch inputs. In one or more embodiments, the batch suggestion generation system also eliminates one or more data inputs within a respective batch to facilitate smaller batches of input without sacrificing the accuracy of the output set generated by the LLM in response to the batch permutation.

[0011] As an illustrative example, a batch suggestion generation system can perform (or cause to perform) a series of actions for processing batch inputs using one or more LLMs. In one or more embodiments, the batch suggestion generation system generates batch suggestions that include task inputs and multiple data inputs associated with the task, wherein the data inputs have a specific order. The batch suggestion generation system can also generate any number of batch permutations, wherein the data inputs from a first batch suggestion are (e.g., randomly) reordered. The batch suggestion generation system can apply an LLM (or other underlying model) to the batch permutations (and initial batch suggestions) to generate a set of outputs for each batch suggestion (and / or permutation). The batch suggestion generation system can consider characteristics of the outputs (e.g., confidence values, consistency across batch suggestions) to determine the set of outputs (e.g., the final output) based on combinations of outputs associated with the respective batch suggestions.

[0012] This disclosure provides numerous practical applications that offer the benefits and / or solve the associated problems associated with applying LLM and other underlying models to a set of input prompts, particularly where the input prompts are related to similar types of tasks. By way of example and not limitation, some of these features and corresponding benefits will be discussed in conjunction with some example problems and drawbacks of conventional LLM systems.

[0013] Many modern LLMs have token limits associated with the number of words or characters the LLM is configured to process relative to a single query or a series of queries. As the size and complexity of the LLM increases, the token limit also increases, allowing for the processing of more complex queries; however, this also significantly increases the computational budget spent on processing queries and associated hints. In fact, as LLMs become more capable of handling larger queries, including more robust contexts and larger inputs, the token budget is quickly exhausted, and the LLM often cannot provide sufficient service to respond to a series of multiple queries associated with performing one or more related tasks.

[0014] Besides its inability to scale complexity, conventional LLMs also suffer from inefficient token budgeting when receiving a series of related prompts. In fact, in response to receiving a series of inputs associated with a specific task, many LLMs process each input and its associated task individually, leading to significant inefficiencies and inaccuracies. For example, processing individual input prompts that include both the task and its corresponding input can become quite inefficient when dealing with a large number of inputs with similar tasks. Furthermore, because conventional LLMs typically provide outputs that consider both the input and the outputs of previous prompts and the context, processing individual prompts individually will eventually result in less accurate responses over time.

[0015] As will be discussed further in detail herein, the batch suggestion generation system batches input suggestions in a way that reduces the number of tokens spent processing a series of related suggestions. In fact, by batching input suggestions according to one or more embodiments, the number of tokens used is reduced because multiple instances of a task statement (e.g., a task specification) are removed when processing a series of related suggestions.

[0016] Batch suggestion generation systems also prevent inaccuracies in several ways, such as by reordering the data input from batch suggestions. For example, as will be discussed in further detail below, a batch suggestion generation system generates multiple permutations of batch suggestions, in which the data input is reordered. This reordering of batch suggestions provides enhanced accuracy by enabling the LLM to evaluate data input in different orders and consider the context that different orders provide when performing tasks with respect to each data input. By comparing the output from different permutations, batch suggestion generation systems promote higher accuracy with fewer batches performed than in conventional systems.

[0017] Besides inefficiently utilizing tokens and causing potential accuracy issues, conventional methods of processing individual prompts result in a large number of application programming interface (API) calls. These numerous API calls not only consume significant computational resources but also bandwidth resources associated with devices and components interfacing with each other (e.g., via a network). By batch processing prompts according to one or more embodiments described herein, batch prompt generation systems reduce the number of API calls performed by conventional systems when leveraging LLMs. Furthermore, as will be discussed below, batch prompt generation systems implement a process in which confident outputs are removed from subsequent batches, which further reduces the number of API calls (and tokens consumed) without sacrificing the accuracy of the LLM output.

[0018] In addition to the features and associated benefits described above, it should be understood that recent advances in Large Language Models (LLMs) have enabled the processing of long texts consisting of tens of thousands of tokens, sufficient for many conventional Natural Language Processing (NLP) tasks. Many LLMs are trained or otherwise fine-tuned to perform zero-shot or few-shot inference using instruction-based prompts. Constructing prompts for these LLMs typically involves the user providing a detailed task description, context, and a completed example (demonstration), along with a single example of the context used for inference. This conventional prompt baseline is referred to herein as the prompt input or single prompt. For NLP tasks where each data point used for inference is not necessarily verbose, the token count for the instruction and few-shot example in the prompt can be significantly larger than the token count for the data point, resulting in lower token resource utilization compared to encoder-based models (e.g., fine-tuned BERT). This cost-efficiency issue impacts inference speed and computational budget, offsetting many of the benefits that LLMs must provide. As described above, and as will be discussed further in detail below, the features and functionality of the batch prompt generation system described herein are designed to mitigate efficiency and scalability issues by batch processing multiple data points into a single prompt (batch prompt). This strategy increases the "density" of data points, which in turn improves token utilization.

[0019] As will be discussed in further detail, applying batch hints is inherently challenging due to performance degradation. Furthermore, performance variations arise from the different positions of data points within the hints, leading to inaccuracies when batch processing is performed without considering the order of data points. To address quality issues while maintaining high token resource utilization, the implementation of the batch hint generation system described in this paper introduces batch permutations and integrations, as well as filtering or removing certain data points between subsequent permutations of the batch input. As discussed in this paper, the batch hint generation system can provide improved performance for a wide range of popular NLP tasks, including Question Answering (Boolq), Real-Time Entailment (RTE), and Repeat Question Identification (QQP). Based on metrics of utilizing fewer LLM calls, input tokens, and / or API calls when performing batch hints, these performance improvements are even more competitive than SinglePrompt, and in some cases, even provide better performance.

[0020] As discussed above, this disclosure utilizes various terms to describe the features and advantages of one or more embodiments of the batch prompt generation system. Additional details regarding the meaning of some of these terms will now be provided. Further terms will also be discussed in detail in conjunction with one or more embodiments and specific examples below.

[0021] As used herein, a “large language model” or simply “LLM” refers to an AI or ML model trained to generate outputs in response to inputs based on a large dataset. In one or more embodiments described herein, LLM may more generally refer to the base model. An LLM may include a neural network with a large number of parameters (e.g., billions of parameters) that the LLM may take into account when performing a task or otherwise generating outputs based on inputs. In one or more embodiments, an LLM is trained to generate responses to queries or prompts. LLMs can be trained in pattern recognition and text prediction. For example, an LLM may be trained to predict the next word of a particular sentence or phrase. As another example, an LLM may be trained to predict whether a particular sentence contains any errors. As another example, an LLM may be trained to select an output from a predetermined number of outputs. In practice, an LLM may be trained to generate any output from a variety of outputs based on any input prompts from a variety of input prompts. In one or more embodiments, an LLM is a version or generation of a GPT (e.g., GPT 3.5, GPT 4.0), or another brand or variant of an LLM that accepts and processes natural language queries (or other types of input queries). In fact, while one or more embodiments described herein involve features associated with determining the context for an LLM, similar features can be applied to generate the context and use other types of base models to determine the output.

[0022] As used herein, an input prompt refers to a statement that includes data and a task to be performed on that data. In one or more embodiments, the input prompt is generated based on a query provided to the LLM as input. However, in one or more embodiments described herein, the input prompt refers to a task (e.g., a task specification) and associated data input (e.g., a data specification), which includes data to be processed or labeled by the LLM according to a context that takes into account the knowledge base, the task, and any other data provided to the LLM in conjunction with the input query.

[0023] In one or more embodiments, a batch prompt includes or otherwise combines multiple input prompts. In one or more embodiments described herein, a batch prompt is a combination of a task (or multiple similar tasks) and multiple data inputs. For example, in response to multiple input prompts, a batch prompt generation system can generate a batch prompt that includes a combined task based on a task with multiple data inputs and data inputs from the multiple input prompts. The resulting batch prompt may include a single task and any number of data inputs on which the single task is performed. Examples will be discussed below with reference to the accompanying drawings.

[0024] In one or more embodiments, the batch suggestion generation system generates one or more permutations of batch suggestions. As used herein, a permutation or batch arrangement of batch suggestions refers to a batch suggestion in which instances of input data have been reordered in a unique sequence. For example, a batch arrangement may include a task similar to a corresponding batch suggestion and a similar set of data inputs; however, an arrangement may include data inputs reordered within a batch arrangement. In one or more implementations, an arrangement is a random permutation in which the order of the input data has been randomized.

[0025] As used herein, the output of an LLM can include any of the various outputs trained on the LLM. For example, the output can include an answer or execution of a task prompted with respect to any number of input data sets. In addition to the answer or execution of the task, the output can include a confidence value determined by the LLM, which is a measure of the confidence or probability that the answer or execution of the task was correctly completed, and represents the accurate output in response to the task and data input. The output can include any answer, result, response, execution, or any other output of the task.

[0026] Additional details about the batch suggestion generation system implemented based on one or more examples will now be provided. For example, Figure 1A block diagram of environment 100 is shown, having one or more server devices 102 (or other computing devices) on which a bulk prompt generation system 110 is implemented. Environment 100 further includes one or more client devices 104 that communicate with the server devices(s) 102(s) via network 106. As further shown, the server devices(s) 102(s) include or otherwise access LLM 108 for performing one or more tasks in conjunction with data input. Figure 1 As shown, the batch prompt generation system 110 includes several components, which will be discussed in further detail below.

[0027] like Figure 1 As shown, client device 104 and server devices(s) 102 (and devices(s) implementing LLM 108 thereon) can communicate with each other directly or indirectly via network 106. Network 106 may include one or more networks and may use one or more communication platforms or technologies suitable for transmitting data. Network 106 may refer to any data link capable of transmitting electronic data between devices and / or modules in environment 100. Network 106 may refer to a hardwired network, a wireless network, or a combination of a hardwired network and a wireless network. In one or more embodiments, network 106 includes the Internet.

[0028] Client device 104 and server devices(s) 102 can refer to various types of computing devices. For example, in one or more embodiments, client device 104 may include mobile devices such as mobile phones, smartphones, PDAs, tablets, or desktop computers. In one or more embodiments, client device 104 may include non-mobile devices such as desktop computers, server devices, or other non-portable devices. In one or more embodiments, client device 104 refers to a client of a cloud computing system (e.g., an internal or external client). In one or more embodiments described herein, server devices(s) 102 refer to one or more server devices of a cloud computing system accessible to client devices (e.g., user-operated consumer devices). In one or more implementations, server devices(s) 102 refer to one or more third-party server devices independent of client device 104. Each of client device 104 and server devices(s) 102 may include the following combinations Figure 5 The described features and functions.

[0029] As stated above, and as Figure 1As shown, the batch suggestion generation system 110 includes multiple components for performing actions and providing the functionality described herein. As an example, the batch suggestion generation system 110 may include a batch manager 112, a sorting manager 114, a self-reflective guided early stop (SEAS) manager 116 (or simply "early stop manager"), an output generator 118, and a data storage device 120 that, in performing the features and functions described herein, provides any component of the batch suggestion generation system 110 with access to data stored or accessible on the data storage device 120.

[0030] As further shown, the batch prompt generation system 110 can communicate with the LLM 108 (e.g., one or more devices on which the LLM 108 is implemented). In one or more embodiments, the LLM 108 is stored or contained within the computing environment of the server device(s) 102. In some implementations, the LLM 108 is external to the batch prompt generation system 110 (such as a third-party LLM). In one or more embodiments, the LLM 108 is a combination of multiple models that collectively perform one or more tasks related to data input.

[0031] It should be understood that, although Figure 1 An example is shown where each component of the batch prompt generation system 110 is implemented entirely on server device(s) 102; however, other implementations may include one or more components (or subcomponents) implemented on different devices across environment 100. As a non-limiting example, one or more of the batch manager 112, arrangement manager 114, SEAS manager 116, and / or output generator 118 may be implemented on different computing devices (e.g., on different server nodes of a cloud computing system or together across different cloud computing platforms). One or more components may be implemented wholly or partially on client device 104, such as batch processing prompts on client device 104 and other steps performed on server device(s) 102. Furthermore, although LLM 108 is shown external to server device(s) 102, in one or more embodiments, LLM 108 is included as a component of the batch prompt generation system 110 and may be combined with… Figure 1 The other components shown are implemented on server devices (multiple) 102.

[0032] Additional details regarding the components of the batch prompt generation system 110 will be discussed in conjunction with the accompanying drawings. For example, Figure 2AAn example workflow 200 (e.g., a portion of a workflow) is illustrated, in which a batch manager 112 receives multiple input prompts 122a-n. As described above, input prompts 122a-n may include a task 126 and an associated data input 128. In one or more implementations, each of the input prompts 122a-n includes a task and an associated data input. In one or more implementations, input prompts 122a-n are associated with performing a task of the same or similar type. For example, each of the tasks 126a-n may refer to the same (e.g., equivalent) or similar type of task associated with a unique or different data input 128a-n.

[0033] In one or more implementations, the batch suggestion generation system 110 receives multiple input suggestions 122a-n. The batch suggestion generation system 110 can group the input suggestions 122a-n into corresponding groups. For example, the batch manager 112 can receive multiple input suggestions 122a-n and categorize them based on included tasks 126a-n. For example, the batch manager 112 can categorize input suggestions 122a-n within a first group based on (e.g., the same or similar) a first task, and categorize a second group based on (e.g., the same or similar) a second task. The batch manager 112 can group the input suggestions 122a-n based on any of various criteria or characteristics of the input suggestions 122a-n. In one or more implementations, the batch manager 112 groups the input suggestions 122a-n based on programming to the LLM 108 to process the corresponding input suggestions 122a-n within a single batch.

[0034] like Figure 2A As shown, the batch manager 112 generates a batch prompt 124 based on content included within the received input prompts 122a-n. In one or more embodiments, the batch manager 112 generates the batch prompt 124 based on a set of input prompts 122a-n determined to be relevant (e.g., based on similar or identical tasks 126a-n associated with corresponding data inputs 128a-n). Figure 2A As shown, tasks with batch prompt 124 are usually referred to as batch task 126. For example... Figure 2A As shown, batch manager 112 generates batch prompt 124, which includes batch tasks 126 (or a similar set of tasks) and an associated batch dataset 130, which includes data inputs 128a-n from each of (e.g., associated) input prompts 122a-n. Batch manager 112 can provide batch prompt 124, including batch tasks 126 and batch dataset 130, to arrangement manager 114.

[0035] The permutation manager 114 can generate any number of batch permutations 134a-c of the batch prompts 124. As described above, batch permutations can include variations of the batch prompts 124, wherein the data inputs 128a-n for a given task are reordered into a specific and unique sequence or sort. Figure 2A In the example shown, the multiple batch permutations 134a-c include a first batch permutation 134a, which includes (e.g., batch hint 124) batch tasks 126 and a first batch dataset 130a sorted as data sequence 1. The multiple batch permutations 134 also include a second batch permutation 134b, which includes tasks(s) 126 and a second batch dataset 130b sorted as data sequence 2. The multiple batch permutations 134 also include a third batch permutation 134c, which includes tasks(s) 126 and a third batch dataset 130c sorted as data sequence 3. Each of the batch datasets 130a-c may include similar combinations of data inputs 128a-n reordered according to a corresponding data sequence (e.g., randomly). In this way, each of the multiple batch permutations 134 may include all, but in a unique order, the data inputs 128a-n of tasks 126 and batch prompts 124 (and input prompts 122a-n).

[0036] Although Figure 2A Examples of multiple batch permutations 134a-c including three permutations are shown, but more or fewer permutations can be used. For example, in one or more implementations, permutation manager 114 generates a single batch permutation in addition to the original batch hint 124 to provide to LLM 108. In one or more implementations, permutation manager 114 generates two or more batch permutations. In one or more embodiments, permutation manager 114 generates batch permutations and determines whether one or more additional batch permutations should be generated. For example, permutation manager 114 may provide batch permutations 134a-c to LLM 108 and may determine a confidence score based on the corresponding output from LLM 108. Permutation manager 114 may generate one or more additional batch permutations based on the confidence score (e.g., if the confidence score is not high enough).

[0037] It should be understood that although the data sequences for the batch datasets 130a-c in batch permutations 134a-c are randomized (or simply reordered) versions of the original batch hint 124, in one or more embodiments described herein, LLM 108 can be applied to any batch permutation in a manner similar to when LLM 108 is applied to the original batch hint 124. Therefore, in the following example where LLM 108 is applied to the first batch permutation 134a and the second batch permutation 134b, this can mean that LLM 108 is applied to any batch permutation of the original batch hint 124 and batch permutations 134a-c. Alternatively, this can mean that LLM 108 is applied to any one of batch permutations 134a-c, followed by the original batch hint 124. Alternatively, this can mean that LLM is applied to both of batch permutations 134a-c (either the original batch hint 124 is applied first or none is applied).

[0038] Although Figure 2A An example is shown in which batch manager 112 generates batch hints 124 and permutation manager 114 generates a set of batch permutations 134; however, it should be understood that this is an example implementation provided as an explanation of a possible workflow. In one or more embodiments, batch manager 112 and permutation manager 114 are implemented as a single component for generating multiple batch hints, which include any number of batch permutations of different data sequences having a batch dataset. Thus, while one or more examples described herein involve generating an original batch hint 124 and then creating permutations 134a-c based on the original batch hint 124, batch hint generation system 110 can generate one or more batch hints 124 that include batch hints of different ordered sets 128 of data input (e.g., different data sequences of batch dataset 130). As an example, batch hint generation system 110 can generate a first batch hint with a first ordered set of data input 128 and a second batch hint with a second ordered set of data input 128. Batch hint generation system 110 can generate any number of batch hints 124 including different ordered sets of data input 128.

[0039] continue, Figure 2B It shows Figure 2A The next part of the workflow described in section 200. For example... Figure 2B As shown, multiple batches can be arranged in 134a-c (e.g., including such batches). Figure 2AThe batch permutations 134a-c and / or the original batch hints 124 shown are provided as input to the LLM 108. The LLM 108 can analyze each batch permutation 134 relative to the batch datasets 130a-c of task 126. As described above, the LLM 108 can evaluate the batch datasets 130a-c in a specific order or data sequence, wherein the data input is sorted within the corresponding batch datasets 130a-c within each corresponding batch permutation.

[0040] For example, in the case where the first batch permutation 134a includes a batch dataset 130a sorted by a first data sequence (e.g., data sequence 1), the LLM 108 may evaluate the data inputs 128 in the stated specific order and update the context used by the LLM 108 as iteratively evaluates the data inputs 128 (e.g., when performing or evaluating task 126 relative to each data input 128). In one or more embodiments, the LLM 108 uses the evaluation of each data input 128 to update the context (e.g., a prompt or query context used by the LLM 108 to generate output based on the corresponding input). In one or more embodiments, the LLM 108 uses the evaluation of every two or three data inputs 128 (or other predetermined number of data inputs 128) to update the context.

[0041] After evaluating the first batch permutation 134a out of multiple batch permutations 134a-c, LLM 108 can be used to evaluate the second batch permutation 134b again in a similar manner. In this example, LLM 108 can start from an initial context and apply LLM 108 to each data input 128 of the batch dataset 130 in the second ordered sequence (e.g., data sequence 2) of the second batch permutation 134b. Similar to evaluating the first batch permutation 134a, LLM 108 can evaluate the second batch dataset 130b in a specific order of the second data sequence and update the context used by LLM 108 while processing data input 128 and while performing task 126 with respect to data input 128. LLM 108 can iteratively update the context in a similar manner to that described above.

[0042] In some embodiments, LLM 108 evaluates data inputs 128 of the batch dataset in the associated data sequence order for each batch permutation 134 as described above, and generates outputs 144 for each data input 128 of the corresponding batch dataset. For example, LLM 108 may evaluate each data input 128 relative to task 126 and may determine the corresponding answer, result, response, or other output as output 144 for that data input 128. For example, as Figure 2BAs shown, LLM 108 can evaluate each data input 128 and determine a correct or incorrect output 144 for each data input 128. LLM 108 can generate a set of permutation outputs 140 in this manner for each of the batch permutations 134a-c. Based on the multiple sets of permutation outputs 140, the batch hint generation system 110 can determine a set of batch outputs 150 (e.g., the final result or batch hint output) for the task 126 applied to the batch dataset 130. The batch hint generation system 110 can utilize various techniques described below (such as combining...) Figure 2C The described technique) determines the batch output set 150.

[0043] In some embodiments, LLM 108 may determine a confidence score 142 for each output 144. For example, LLM 108 may determine the confidence level of the accuracy of the associated output 144 when evaluating each data input 128 relative to task 126. For example, LLM 108 may evaluate whether it determines each output 144 with high confidence (e.g., at or above a threshold confidence value) or low confidence (e.g., below a threshold confidence value or other thresholds). Figure 2B As shown, LLM 108 can indicate the association confidence score 142 associated with the output 144 of the permutation output set 140.

[0044] like Figure 2B As shown, LLM 108 may optionally provide one or more of the permutation output sets 140 to SEAS Manager 116. In this example, permutation output sets 140 are provided to SEAS Manager 116 when generating permutation output sets 140 and during iterative analysis of each batch permutation 134. For example, LLM 108 may generate and provide a first permutation output set 140a for a first batch permutation 134a, and then may generate and provide a second permutation output set 140b for a second batch permutation 134b. SEAS Manager 116 may then perform a narrowing or filtering process on the data inputs 128 of the batch dataset 130 by enabling the batch manager 112, LLM 108, or other components of the batch prompt generation system 110 to generate and / or evaluate new and / or subsequent batch permutations 134 with a smaller number of data inputs 128 to be processed by LLM 108.

[0045] In one or more embodiments, the SEAS manager 116 narrows down the batch dataset 130 between iterations of evaluating the batch permutation 134 by taking into account the confidence scores 142 determined by the LLM 108 relative to the permutation output set 140 (e.g., output sets 140a-b). For example, in one or more embodiments, the SEAS manager 116 removes specific data inputs 128 for subsequent batch permutations 134 based on one or more outputs 144 corresponding to data inputs 128 that have accurate high confidence scores 142 for the outputs 144. For example, as Figure 2B As shown, output 144 indicates high confidence scores for data input 1 and data input 4, and the SEAS manager 116 can accordingly remove data input 1 and data input 4, allowing the LLM 108 to evaluate those data inputs without considering future batch permutations. In one or more embodiments, the SEAS manager 116 removes data input 128 for future batch permutations based on multiple iterations of output 144 for the corresponding data input 128 with a high confidence score 142. For example, the SEAS manager 116 can remove data input 128 from the third batch dataset 130c based on the corresponding output 144 being the same or similar for multiple iterations, and / or based on output 144 having a high confidence score for multiple iterations. Multiple iterations for the high confidence score of output 144 can be associated with the evaluation of multiple consecutive batch permutations or simply any two or more non-consecutive batch permutations. In this way, the resulting third output set 140c can include fewer data inputs than the previous iterations of the first output set 140a and the second output set 140b.

[0046] In this way, the SEAS manager 116 can maintain a record of the result output 144 and associated confidence score 142 for each data input 128 of the batch dataset 130, and can iteratively reduce the batch dataset 130 by eliminating data inputs 128 that have been determined multiple times with high confidence. The SEAS manager 116 can reduce the batch dataset 130 in such a way that it includes fewer data inputs 128 associated with high confidence scores for evaluation of future batch permutations 134. Therefore, for future batch permutations 134, the batch dataset 130 can be continuously and selectively reduced to improve the efficiency of LLM 108 in evaluating the batch dataset 130 (e.g., the remaining data inputs 128), and to facilitate the allocation of computational resources to processing those data inputs 128 that have not yet been evaluated with high confidence.

[0047] This iterative reduction performed by SEAS manager 116 in Figure 2BThe diagram shows that data flow (1) illustrates the LLM determining one or more permutation output sets 140 and providing them to the SEAS manager 116, data flow (2) illustrates the SEAS manager 116 correspondingly narrowing the batch dataset 130 by eliminating data inputs 128 associated with high confidence, and data flow (3) illustrates the LLM 108 determining and providing one or more additional permutation output sets 140 based on the reduced version of the batch dataset 130. The batch prompt generation system 110 can run any number of iterations while further reducing the data inputs 128 any number of times to include only those data inputs that the LLM 108 has less than a threshold metric of confidence in the corresponding output 144.

[0048] SEAS manager 116 may stop iteratively narrowing the batch dataset 130, and / or may stop iteratively generating and / or evaluating batch permutations based on various factors. In one or more embodiments, SEAS manager 116 causes a predetermined number of batch permutations to be generated and analyzed (e.g., 3 to 10 batch permutations). In one or more embodiments, SEAS manager 116 causes any number of batch permutations to be generated and analyzed based on how many data inputs have been removed due to meeting a sufficient confidence threshold relative to a particular data input. For example, SEAS manager 116 may cause up to a threshold number of batch permutations 134 to be generated until a threshold percentage or a threshold number of data inputs 128 have been removed or until the token budget or computation budget has been consumed.

[0049] Figure 2C It shows Figure 2A and Figure 2B The workflow 200 shown is a continuation. In this example, the SEAS manager 116 provides multiple permutation output sets 140a-c to the output generator 118. The output generator 118 may consider the output from each data input across the multiple permutation output sets 140a-c to determine a batch output set 150 (or batch prompt output) including the final output set 146. As described above, the batch output set 150 may be an output set determined in response to batch prompt 124 and / or in response to multiple input prompts 122a-c used in generating batch prompt 124. In one or more embodiments, the output generator 118 generates the batch output set 150 based on the output 144 of the permutation output sets 140a-c received from the SEAS manager 116 (or from the LLM 108, such as in the case where the SEAS manager 116 is not used and the batch data sets are not narrowed).

[0050] Output generator 118 can consider combinations of outputs 144 from multiple permutation output sets 140a-c in various ways to determine the final output 146 for the batch output set 150. For example, output generator 118 can determine the final output 146 for a given data input based on a majority vote (e.g., simple majority or majority) of the outputs 144 from the permutation output sets 140a-c associated with the data input 128. In this way, permutation output sets 140a-c can “vote” for the final output 146 of each data input of the batch dataset 130 based on the associated outputs 144 of the respective output sets 140a-c, and the final output 146 can be determined by majority voting.

[0051] In some embodiments, the output generator 118 applies weights 148 to the output 144 based on different criteria (e.g., weighting the votes for the permutation output sets 140a-c). In some embodiments, the weights 148 may be determined by and / or associated with a confidence score 142 for each data input in the permutation output sets 140a-c. For example, the weights 148 may be assigned accordingly based on the confidence level assigned by the LLM 108 during the evaluation of a given batch of data inputs. For example, the determination of a high confidence score 142 may be assigned a weight of 1.0. The determination of a low confidence score 142 may be assigned a weight of 0.2. Any other weights 148 may be determined and assigned for any associated confidence level. The output generator 118 may apply the weights 148 accordingly to the corresponding output 144 (or votes) and may determine the final output 146 accordingly based on the weighted values. For example, the final output 146 may be determined as a self-weighted majority vote or as a majority based on the weighted vote. In this way, output generator 118 can generate a batch output set 150 that corresponds to the answer, result, response, or other output for each instance of input data represented in batch prompt 124, as described above, and may include the final output 146 for each input prompt 122a-c used when generating batch prompt 124.

[0052] Figure 3 Another example workflow 300 is shown, illustrating an example implementation of a batch prompt generation system 110 according to one or more embodiments described herein. Figure 3As shown, the batch suggestion generation system 110 can receive multiple individual suggestions 152. In this example, each individual suggestion 152 may include a task specification 154, which instructs a task to generate tags for syntax checking of the corresponding statement. In this example, the statement refers to a data instance (e.g., data 156a). The batch suggestion generation system 110 can receive any number of individual suggestions 152. In this example, the batch suggestion generation system 110 receives sixty-four individual suggestions 152 with similar task specifications 154 and different data 156.

[0053] The batch prompt generation system 110 can generate batch prompts 158. For example... Figure 3 As shown, batch suggestion 158 includes a single task statement or task specification and a list of sixty-four data instances on which the task (corresponding to task 154) is executed. As further shown, batch suggestion generation system 110 can perform batch permutation and integration processes to generate multiple batch permutations 155a-c. In this example, batch suggestion generation system 110 generates three batch permutations 155a-c, or three rounds of batches including reordered data instances 157a-c. In this example, data instances 157a-c are randomly reordered. Other reordering mechanisms can be used.

[0054] As shown in this example, the batch suggestion generation system 110 can determine the answer 162 for each data instance 157a-c in each of the batch permutations 155a-c. For example, the batch suggestion generation system 110 can determine a "correct" or "incorrect" answer for each data instance. The batch suggestion generation system 110 can apply LLM 164 to the first batch permutation to determine a first set of "correct" and "incorrect" answers for the first batch permutation 155a. The batch suggestion generation system 110 can then apply LLM 164 to the second batch permutation 155b to determine a second set of "correct" and "incorrect" answers for the sorting of data instances associated with the second batch permutation 155b.

[0055] Consistent with one or more embodiments described herein, the batch suggestion generation system 110 may apply LLM 164 to each of the batch permutations 155a-c. Alternatively, in one or more embodiments, the batch suggestion generation system 110 may iteratively apply LLM 164 to each batch permutation 160 and apply a SEAS process to narrow the data instance 156 of subsequent batch permutations 160 by utilizing each iteration of applying LLM 164 to the batch permutation (or after a first predetermined number of iterations), as described herein.

[0056] In one or more embodiments, the batch suggestion generation system 110 applies weights 166 to answers 162 based on confidence scores 168. As an example, the batch suggestion generation system 110 may consider three sets of answers 162 associated with three batch permutations 155a-c to determine a batch output set based on combinations of each set of answers 162. In this example, the batch suggestion generation system 110 may apply a first weight (e.g., 0.2) to each answer whose confidence score 168 is less than a threshold. Alternatively, the batch suggestion generation system 110 may apply a higher second weight (e.g., 1.0) to each answer 162 having a confidence score 168 at or above a threshold (or other threshold). Other examples may include different ranges of confidence scores 168 associated with different weights 166. The batch output set based on combinations of individual answers 162 may be based on combinations of answers 162 weighted by corresponding weights 166 (e.g., majority).

[0057] In addition to weight 166, the batch suggestion generation system 110 can also consider more or fewer rounds of batch permutation 160 based on certain elimination or filtering of data instances between the evaluations of batch permutations 155a-c. For example, as Figure 3 As shown, the first evaluation of the first batch permutation 155a may include sixty-four data instances 157a processed and considered using LLM 164. Similarly, the second evaluation of the second batch permutation 155b may include sixty-four data instances 157b. In this example, where two data instances are associated with two high-confidence scores 168 respectively, the batch suggestion generation system 110 may perform a third evaluation of a simplified version of the third batch permutation 155c, in which only twenty-six data instances are considered associated with the third batch permutation 155c. The batch suggestion generation system 110 may remove one or more additional data instances in the fourth evaluation of the fourth batch permutation, in which only twelve data instances are considered. The batch suggestion generation system 110 may stop after a certain number of rounds (e.g., after a batch permutation of a certain number has been evaluated) or after a specific number or percentage of data instances has been removed. In cases where LLM 164 lacks confidence in the associated answers 162 for one or more data instances 156, the batch suggestion generation system 110 can simply implement the voting or weighting method discussed above to generate answers based on combinations of associated answers 162 for each batch of data instances.

[0058] The above example is intended to illustrate exemplary features and functionality of the batch prompt generation system 110 in response to multiple input prompts (or a batch of input prompts) provided as input to the LLM to generate a set of outputs.

[0059] Turn now Figure 4The figure illustrates an example flowchart of a series of actions for efficiently processing multiple input prompts, according to one or more embodiments described herein. While Figure 4 Actions according to one or more embodiments are shown, but alternative embodiments may omit, add, reorder, and / or modify them. Figure 4 Any action shown. Figure 4 The action can be performed as part of a method. Alternatively, a non-transitory computer-readable medium may include instructions that, when executed by one or more processors, cause a computing device to perform... Figure 4 The action. In another embodiment, the system can perform... Figure 4 The action.

[0060] For example, Figure 4 A series of actions 400 for efficiently handling multiple input prompts is shown. For example... Figure 4 As shown, a series of actions 400 includes action 410: generating a batch prompt that includes a task input and multiple data inputs. In one or more implementations, action 410 includes generating a batch prompt that includes a task input and multiple data inputs associated with the task input, the multiple data inputs having a first order. In one or more embodiments, the multiple data inputs may refer to a subset of data inputs from a set of data inputs associated with the task input. In this case, the subset may refer to an appropriate subset (e.g., a set of some but not all received data inputs), or it may refer to the entire set of data inputs.

[0061] The series of actions 400 further includes action 420: generating one or more batch permutations based on task input and multiple data inputs. In one or more embodiments, action 420 includes: generating one or more batch permutations based on batch prompts, each of the one or more batch permutations including data inputs from multiple data inputs in an order different from a first order. Batch permutations may include data inputs from multiple data inputs in different orders. Batch permutations may include a first batch permutation in which multiple data inputs are reordered relative to a first order. Batch permutations may include a second batch permutation in which multiple data inputs are reordered relative to a first order.

[0062] The series of actions 400 includes action 430: applying the LLM to batch prompts and / or (multiple) batch permutations to generate a first set of outputs corresponding to the task input based on a first ordered set of data inputs, and generating a second set of outputs corresponding to the task input based on a second ordered set of data inputs. In one or more embodiments, action 430 includes: applying the large language model to batch prompts and one or more batch permutations to generate (1) a first set of outputs based on multiple data inputs in response to the task input, and (2) a second set of outputs based on multiple data inputs from one or more batch permutations in response to the task input. In some embodiments, the method includes: determining a confidence value for each output from the first and second output sets. In some embodiments, one or more weights may be determined for each output from the first and second output sets based on the confidence score. For example, one or more weights may include one or more high weight values ​​based on the large language model that determine that the corresponding output is likely to be accurate. One or more weights may include one or more low weight values ​​based on the large language model that determine that the corresponding output is likely to be inaccurate.

[0063] One or more batch permutations may include multiple additional permutations, wherein the second output set includes the output set for each permutation. One or more batch permutations may include a reduced set of data inputs. For example, based on one or more data inputs with associated outputs having high correlation confidence values, the reduced set of data inputs may remove one or more data inputs.

[0064] The series of actions 400 further includes action 440: generating batch prompt output in response to task input and associated multiple data inputs. In one or more embodiments, action 440 relates to generating batch prompt output in response to batch prompts, the batch prompt output including multiple outputs based on a first output set and a second output set. In one or more implementations, the batch output is a collected set of outputs included in a single final output, wherein each output is determined by a majority or weighted vote among the corresponding outputs in an individual batch arrangement.

[0065] In one or more embodiments, one or more batch arrangements include a first arrangement of multiple data inputs, the first arrangement including a first reordered set of data inputs in which the multiple data inputs are reordered relative to a first order. In one or more embodiments, one or more batch arrangements include a second arrangement of multiple data inputs, the second arrangement including a second reordered set of data inputs in which the multiple data inputs are reordered relative to a first order. In one or more embodiments, one or more batch arrangements include multiple additional arrangements of multiple data inputs, wherein a second output set includes an output set for each arrangement from the first arrangement, the second arrangement, and the multiple additional arrangements. In one or more embodiments, a series of actions 400 includes: determining a confidence value for each output from the first output set and the second output set.

[0066] In one or more embodiments, one or more batch permutations in the batch permutation include a reduced set of data inputs. In one or more embodiments, the reduced set of data inputs excludes one or more data inputs based on one or more data inputs with associated outputs from a first set of inputs or a second set of inputs having highly correlated confidence values.

[0067] In one or more embodiments, generating batch prompt outputs includes determining one or more weights for each output from the first output set and the second output set based on a confidence score determined by a large language model for each output from the first output set and the second output set. In one or more embodiments, the one or more weights include one or more high weight values ​​that determine the corresponding output is likely to be accurate based on the large language model, and the one or more weights include one or more low weight values ​​that determine the corresponding output is likely to be inaccurate based on the large language model.

[0068] Figure 5 Certain components that may be included within computer system 500 are shown. One or more computer systems 500 may be used to implement the various devices, components, and systems described herein.

[0069] Computer system 500 includes processor 501. Processor 501 can be a general-purpose single-chip or multi-chip microprocessor (e.g., an advanced RISC (Reduced Instruction Set Computer) machine (ARM)), a special-purpose microprocessor (e.g., a digital signal processor (DSP)), a microcontroller, a programmable gate array, etc. Processor 501 can be referred to as a central processing unit (CPU). Although in Figure 5The computer system 500 shows only a single processor 501, but in alternative configurations, a combination of processors (e.g., ARM and DSP) may be used. In one or more embodiments, the computer system 500 also includes one or more graphics processing units (GPUs) that can provide processing services related to both entity classification and graph generation.

[0070] Computer system 500 also includes memory 503 that is in electronic communication with processor 501. Memory 503 can be any electronic component capable of storing electronic information. For example, memory 503 can be embodied as random access memory (RAM), read-only memory (ROM), disk storage media, optical storage media, flash memory devices in RAM, onboard memory included with the processor, erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, etc., including combinations thereof.

[0071] Instruction 505 and data 507 may be stored in memory 503. Instruction 505 may be executed by processor 501 to implement some or all of the functions disclosed herein. Executing instruction 505 may involve using data 507 stored in memory 503. Any of the various examples of modules and components described herein may be implemented in part or in whole as instruction 505 stored in memory 503 and executed by processor 501. Any of the various instances of data described herein may be in data 507 stored in memory 503 and used during the execution of instruction 505 by processor 501.

[0072] The computer system 500 may also include one or more communication interfaces 509 for communicating with other electronic devices. The communication interface 509 may be based on wired communication technology, wireless communication technology, or both. Some examples of the communication interface 509 include Universal Serial Bus (USB), Ethernet adapter, wireless adapter operating according to the Institute of Electrical and Electronics Engineers (IEEE) 802.11 wireless communication protocol, and Bluetooth. ® Wireless communication adapter and infrared (IR) communication port.

[0073] Computer system 500 may also include one or more input devices 511 and one or more output devices 513. Some examples of input devices 511 include keyboards, mice, microphones, remote control devices, buttons, joysticks, trackballs, touchpads, and light pens. Some examples of output devices 513 include speakers and printers. A particular type of output device typically included in computer system 500 is a display device 515. Display devices 515 used with the embodiments disclosed herein can utilize any suitable image projection technology, such as liquid crystal displays (LCDs), light-emitting diodes (LEDs), gas plasma, electroluminescence, etc. A display controller 517 may also be provided for converting data 507 stored in memory 503 into text, graphics, and / or moving images (as applicable) displayed on display device 515.

[0074] The various components of the computer system 500 can be coupled together through one or more buses, which may include power buses, control signal buses, status signal buses, data buses, etc. For clarity, the various buses are... Figure 5 It is shown as bus system 519.

[0075] The techniques described herein can be implemented in hardware, software, firmware, or any combination thereof, unless specifically described as being implemented in a particular manner. Any features described as modules, components, etc., can also be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, these techniques can be implemented at least in part by a non-transitory processor-readable storage medium comprising instructions that, when executed by at least one processor, perform one or more of the methods described herein. Instructions can be organized into routines, programs, objects, components, data structures, etc., which can perform specific tasks and / or implement specific data types, and can be combined or distributed as needed in various embodiments.

[0076] Without departing from the scope of the claims, the steps and / or actions of the methods described herein may be interchanged with each other. In other words, unless the correct operation of the described methods requires a specific order of steps or actions, the order and / or use of specific steps and / or actions may be modified without departing from the scope of the claims.

[0077] The term "determine" encompasses a wide variety of actions; therefore, "determine" can include calculation, operation, processing, deduction, investigation, searching (e.g., looking in a table, database, or other data structure), ascertaining, etc. Furthermore, "determine" can include receiving (e.g., receiving information), accessing (e.g., accessing data in memory), etc. Additionally, "determine" can include resolving, selecting, picking, establishing, etc.

[0078] The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that additional elements may exist in addition to those listed. Furthermore, it should be understood that references to “one embodiment” or “embodiment” in this disclosure are not intended to exclude the existence of additional embodiments that also include the described features. For example, where compatible, any element or feature described with respect to embodiments herein may be combined with any element or feature of any other embodiment described herein.

[0079] This disclosure may be embodied in other specific forms without departing from the spirit or characteristics thereof. The described embodiments are to be considered illustrative rather than restrictive. Therefore, the scope of this disclosure is indicated by the appended claims rather than by the foregoing description. Variations within the meaning and scope of equivalents of the claims will be included within their scope.

Claims

1. A method for processing batch input using one or more large language models, the method comprising: Generate batch prompts, the batch prompts including task input and multiple data inputs associated with the task input, the multiple data inputs having a first order; One or more batch arrangements are generated based on the batch prompts, each of the one or more batch arrangements including data inputs from the plurality of data inputs in a different order than the first order; The large language model is applied to the batch prompts and the one or more batch permutations to generate: A first set of outputs responding to the task input based on the plurality of data inputs; as well as A second set of outputs responding to the task input based on the data input from the plurality of data inputs from the one or more batch arrangements; as well as In response to the batch prompts, batch prompt outputs are generated, the batch prompt outputs including multiple outputs based on the first output set and the second output set.

2. The method of claim 1, wherein the one or more batch arrangements include a first arrangement of the plurality of data inputs, the first arrangement including a first reordered set of data inputs in which the plurality of data inputs are reordered relative to the first order.

3. The method of claim 2, wherein the one or more batch arrangements include a second arrangement of the plurality of data inputs, the second arrangement including a second reordered set of data inputs in which the plurality of data inputs are reordered relative to the first order.

4. The method of claim 3, wherein the one or more batch permutations include a plurality of additional permutations of the plurality of data inputs, wherein the second output set includes an output set for each permutation from the first permutation, the second permutation, and the plurality of additional permutations.

5. The method according to claim 4, further comprising: Determine the confidence value for each output from the first output set and the second output set.

6. The method of claim 1, wherein one or more batch arrangements in the batch arrangement include a reduced set of data inputs.

7. The method of claim 6, wherein the reduced set of data inputs excludes the one or more data inputs based on one or more associated outputs from a first set of inputs or a second set of inputs having highly correlated confidence values.

8. The method according to claim 1, wherein generating the batch prompt output comprises: One or more weights are determined for each output from the first output set and the second output set based on the confidence scores determined by the large language model for each output from the first output set and the second output set.

9. The method of claim 8, wherein the one or more weights include determining one or more high weight values ​​that may correspond to an accurate output based on the large language model, and the one or more weights include determining one or more low weight values ​​that may correspond to an inaccurate output based on the large language model.

10. A system comprising: At least one processor; A memory that communicates electronically with the at least one processor; Instructions stored in the memory, the instructions being executed by the at least one processor to: Generate batch prompts, the batch prompts including task input and multiple data inputs associated with the task input, the multiple data inputs having a first order; One or more batch arrangements are generated based on the batch prompts, each of the one or more batch arrangements including data inputs from the plurality of data inputs in a different order than the first order; The large language model is applied to the batch prompts and the one or more batch permutations to generate: A first set of outputs responding to the task input based on the plurality of data inputs; as well as A second set of outputs responding to the task input based on the data input from the plurality of data inputs from the one or more batch arrangements; as well as In response to the batch prompts, batch prompt outputs are generated, the batch prompt outputs including multiple outputs based on the first output set and the second output set.

11. The system of claim 10, wherein the one or more batch arrangements include a first arrangement of the plurality of data inputs, the first arrangement including a first reordered set of data inputs in which the plurality of data inputs are reordered relative to the first order.

12. The system of claim 11, wherein the one or more batch arrangements include a second arrangement of the plurality of data inputs, the second arrangement including a second reordered set of data inputs in which the plurality of data inputs are reordered relative to the first order.

13. The system of claim 12, further comprising instructions executed by the at least one processor for: determining a confidence value for each output from the first output set and the second output set.

14. The system of claim 10, wherein one or more batch arrangements in the batch arrangement include a reduced set of data inputs.

15. The system of claim 14, wherein the reduced set of data inputs excludes the one or more data inputs based on one or more associated outputs from a first set of inputs or a second set of inputs having highly correlated confidence values.

16. The system of claim 10, wherein generating the batch prompt output comprises: One or more weights are determined for each output from the first output set and the second output set based on the confidence scores determined by the large language model for each output from the first output set and the second output set.

17. The system of claim 16, wherein the one or more weights include determining one or more high weight values ​​that may correspond to an accurate output based on the large language model, and the one or more weights include determining one or more low weight values ​​that may correspond to an inaccurate output based on the large language model.

18. A non-transitory computer-readable medium having instructions stored thereon, the instructions causing a computing device, when executed by at least one processor, to: Generate batch prompts, the batch prompts including task input and multiple data inputs associated with the task input, the multiple data inputs having a first order; One or more batch arrangements are generated based on the batch prompts, each of the one or more batch arrangements including data inputs from the plurality of data inputs in a different order than the first order; The large language model is applied to the batch prompts and the one or more batch permutations to generate: A first set of outputs responding to the task input based on the plurality of data inputs; as well as A second set of outputs responding to the task input based on the data input from the plurality of data inputs from the one or more batch arrangements; as well as In response to the batch prompts, batch prompt outputs are generated, the batch prompt outputs including multiple outputs based on the first output set and the second output set.

19. The computer-readable medium of claim 18, wherein the one or more batch arrangements include a first arrangement of the plurality of data inputs, the first arrangement including a first reordered set of data inputs in which the plurality of data inputs are reordered relative to the first order.

20. The computer-readable medium of claim 19, wherein one or more of the batch arrangements comprise a reduced set of data inputs, and wherein the reduced set of data inputs excludes the one or more data inputs based on one or more associated outputs from a first set of inputs or a second set of inputs having highly correlated confidence values.

Images