Adaptive query routing to natural language generator based on query difficulty

Abstract

Natural language generators (NLGs), including large language models, are powerful technologies that are widely used. However, generally, as NLGs become more powerful and complex, the increased complexity of the NLGs requires a large amount of processing resources. The present disclosure provides automated techniques for dynamically routing queries between at least two NLGs based on an assessment of the difficulty of the query. Less difficult queries can be routed to less resource-intensive NLGs, while more difficult queries are routed to more complex but more resource-intensive NLGs. Thus, routing less difficult queries to less resource-intensive models can conserve computing resources while providing little or no degradation in response quality, and in some cases, providing improved response quality.

Description

Technical Field

[0001] This disclosure generally relates to the processing of queries using natural language generators. Specific implementations provide selective and dynamic routing of queries among different natural language generators based on the evaluated difficulty of the query, wherein routing less difficult queries to natural language generators with lower computational resource usage provides improved efficiency compared to processing such queries by more resource-intensive natural language generators. Background Technology

[0002] Generative AI (artificial intelligence), such as that achieved using large language models like CHATGPT from OPEN AI, is experiencing one of the fastest adoption rates in technology history. Beyond its “standalone” applications (such as consumer-facing implementations of CHATGPT), companies are showing growing interest in integrating generative AI into a wide range of software applications, including spreadsheet programs and word processing software.

[0003] Specific techniques used in generative AI, such as natural language generators (including Large Language Models (LLMs) and other transformer-based techniques), differ from some other AI processes, such as neural network classifiers. Generative AI techniques consume significant computational resources during both the training and inference phases, whereas for neural network classifiers, training is typically resource-intensive, but inference is not. This resource requirement of generative AI techniques is partly attributed to the exceptionally large number of parameters employed by these models. For example, LLMs can include billions of parameters.

[0004] Therefore, while the increased adoption of generative AI offers substantial practical benefits, it also requires significant computational resources. Consequently, there is ample room for improvement in terms of efficiency and resource optimization in the use of generative AI. Summary of the Invention

[0005] This summary is provided to introduce the selection of concepts in a simplified form, which is further described below in the detailed description. This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter.

[0006] Natural Language Generators (NLGs), including large language models, are a widely used and powerful technique. However, typically, as NLGs become more powerful and complex, their corresponding increase in complexity requires significant processing resources. This disclosure provides an automated technique for dynamically routing queries between at least two NLGs based on an assessment of query difficulty. Less difficult queries can be routed to less resource-intensive NLGs, while more difficult queries are routed to more complex but more resource-intensive NLGs. Therefore, routing less difficult queries to less resource-intensive models can save computational resources while providing little or no degradation in response quality, and in some cases, improved response quality.

[0007] In one aspect, this disclosure provides a process for performing adaptive query routing using the disclosed techniques. A first query is received. For the first query, a first set of one or more query difficulty metric values ​​for a set of one or more query difficulty metrics is generated. Based on the first set of the one or more query difficulty metric values, it is determined that the first query has a first query difficulty level. Based on determining that the first query has the first query difficulty level, the first query is routed to a first natural language generator model specified for the first query difficulty level. A second query is received. For the second query, a second set of one or more query difficulty metric values ​​for the set of one or more query difficulty metrics is generated. Based on the second set of the one or more query difficulty metric values, it is determined that the second query has a second query difficulty level, which is different from the first query difficulty level. Based on determining that the second query has the second query difficulty level, the second query is routed to a second natural language generator model specified for the second query difficulty level.

[0008] In another aspect, this disclosure provides a process for training a router capable of performing adaptive query routing using techniques according to this disclosure. The router is trained to perform routing to a determined natural language generator model among a plurality of available natural language generator models for received queries. The training includes performing various operations on a given query from a set of training queries.

[0009] The operation includes generating a vector space embedding of a given query from a set of training queries. A first natural language generator model from among the plurality of available natural language generator models is used to generate a first response to the given query. A second natural language generator model from among the plurality of available natural language generator models is used to generate a second response to the given query.

[0010] Calculate a quality metric for the first response. Calculate a quality metric for the second response. Assign labels to the given query using the quality metrics for the first and second responses. The labels indicate the difficulty of the given query. Generate query difficulty metrics by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model.

[0011] The machine learning model is trained to provide a trained machine learning model by minimizing a loss function that uses a label for the given query and a query difficulty metric for the given query. The router includes the trained machine learning model and selectively routes less difficult queries to a first natural language generator model, saving computational resources compared to using a second natural language generator model to process less difficult queries.

[0012] This disclosure also includes computing systems and tangible, non-transitory computer-readable storage media configured to perform or include instructions for performing the methods described above. Various other features and advantages, as described herein, can be incorporated into the described techniques as needed. Attached Figure Description

[0013] Figure 1 This diagram illustrates a computational environment that supports adaptive routing of queries to different natural language generators based on query difficulty.

[0014] Figure 2 The graph illustrates the probability distribution function results of query response quality for two LLM models and the associated BART score evaluations, demonstrating the quality gap between responses provided by such models.

[0015] Figure 3 The graphs provided respectively illustrate the distribution of values ​​using unadjusted labels, the grid search used to identify adjusted values, and the distribution of values ​​using labels adjusted with said adjusted values.

[0016] Figure 4 This is a diagram illustrating a scenario where an NLG model is used to score query responses.

[0017] Figure 5 This is a flowchart of the query routing method according to this disclosure.

[0018] Figure 6 This is a flowchart of the process of training an adaptive query router that routes queries based on query difficulty.

[0019] Figure 7 This is a diagram of an example computing system in which some of the described embodiments can be implemented.

[0020] Figure 8 This is an example cloud computing environment that can be used in conjunction with the technologies described in this article. Detailed Implementation

[0021] Example 1 - Overview Generative AI (artificial intelligence), such as that achieved using large language models like CHATGPT from OPEN AI, is experiencing one of the fastest adoption rates in technology history. Beyond its “standalone” applications (such as consumer-facing implementations of CHATGPT), companies are showing growing interest in integrating generative AI into a wide range of software applications, including spreadsheet programs and word processing software.

[0022] Specific techniques used in generative AI, such as Large Language Models (LLM) and other transformer-based techniques, differ from some other AI processes, such as neural network classifiers. Generative AI techniques consume significant computational resources during both the training and inference phases, whereas for neural network classifiers, training is typically resource-intensive, but inference is not. This resource requirement is partly attributed to the exceptionally large number of parameters employed by these models. For example, LLMs can include billions of parameters.

[0023] Therefore, while the increased adoption of generative AI offers substantial practical benefits, it also requires significant computational resources. Consequently, there is ample room for improvement in terms of efficiency and resource optimization in the use of generative AI.

[0024] This disclosure provides techniques for dynamically routing queries to different Natural Language Generation (NLG) generative AI models, particularly LLM models. Routing decisions are made by analyzing the query to determine at least one metric describing the difficulty of the query. Query difficulty refers to a metric related to the performance gap between two or more NLG models when generating a query response. Typically, this disclosure involves routing queries between two models where the models vary in terms of "cost." Cost can include factors such as response time / latency, energy usage, or the use of computing resources such as processors, memory, or network resources.

[0025] Therefore, query difficulty can be reflected as a measure of how the response provided by a less resource-intensive NLG model compares to that provided by a more resource-intensive NLG model. A query can be characterized as less difficult if the performance gap in response quality is relatively small between the two NLG models, and as more difficult if the performance gap is relatively large. Typically, less difficult queries are routed to less resource-intensive NLG models, while more difficult queries are routed to more resource-intensive NLG models, which is expected to produce higher-quality responses for the queries.

[0026] Various techniques can be used to determine whether a specific query to be routed is low-difficulty or high-difficulty. In a specific example, a classifier, such as a neural network-based classifier, is trained using a set of queries and their associated difficulty metrics. When a query is received, the classifier can be used to classify the query, and based on that classification, the query can be sent to the appropriate NLG model.

[0027] The disclosed techniques therefore offer numerous advantages. For queries that provide acceptable responses to less resource-intensive NLG models, routing the query to such a model instead of a more resource-intensive one saves significant computational resources, reduces the burden on more complex, resource-intensive systems, and makes them more suitable for query processing, where the higher quality responses from the more resource-intensive NLG models justify their use. The disclosed techniques are also beneficial compared to other types of routing because they save computational resources without a significant loss of response quality. That is, users do not receive low-quality responses due to the use of less resource-intensive NLG models, because the query is "filtered," ensuring that the less resource-intensive NLG model will provide an acceptable quality response.

[0028] Indeed, in some cases, manually examining a query may lead to the conclusion that a particular query is "difficult," but in reality, less resource-intensive NLG models provide higher quality results compared to more resource-intensive NLG models. The disclosed techniques, such as using a query classifier, are able to provide improved query responses for specific queries, even compared to using a more resource-intensive NLG model for all queries.

[0029] Note that the disclosed techniques differ from other techniques used for query routing, including query routing based on "task type." For example, some techniques can route queries to specific types of AI models based on task type, such as routing an image generation task to one model, or routing various types of NLG tasks (such as summarizing content, creating content, evaluating content, performing mathematical calculations or performing logical tasks, answering factual questions, translating languages, generating programming code or scripts) to other models more specifically configured for particular task types. Even for common "types" of tasks, some queries may still be more "difficult" than others, and such routing techniques will not take this into account. The disclosed techniques allow selective routing between NLG models of the same general type, including for queries with content suitable for processing by any NLG model to which the query can be routed.

[0030] Relatedly, the disclosed technique differs from task-type-based routing queries because the difficulty of a query is not evaluated during task-type-based routing. Instead, task-type routing decisions are hard-coded so that queries requesting a certain type of task are routed to a specific AI model for that type. Therefore, as described above, task difficulty can correspond more to the suitability of a given AI model for a given task than to whether any particular query is more or less difficult.

[0031] Example 2 - Example computing environment for adaptive query routing Figure 1 The figure illustrates an example computing environment 100 in which the disclosed techniques can be implemented. The computing environment 100 typically includes an application 110 capable of submitting queries to an NLG model. The NLG model can be either a less resource-intensive (LNLG model) or a more resource-intensive (HNLG model). As described in Example 1, the disclosed techniques involve using a router to analyze queries from application 110 to route queries to either an LNLG model or an HNLG model based on the complexity / difficulty of the query.

[0032] Application 110, query, router, at least one LNLG, and at least one HNLG are the basic components of computing environment 100. In addition, computing environment 100 may have different components that communicate in different ways. These will be described relative to specific example use cases. Figure 1 However, alternative scenarios will also be described.

[0033] Client computing system 108 is shown to include application 110. Application 110 can be any application configured to receive queries or prompts, submit them for processing by the NLG model, and receive responses. As described, the application can be a "standalone" application, such as a consumer-facing implementation of CHATGPT, but can also be an application integrating NLG functionality, such as a spreadsheet program or word processing program. Queries can be received from a user or from another computing process. Similarly, responses can be provided to a user or to another computing process. Queries can be sent and responses can be received using APIs provided by the NLG model.

[0034] As shown, query 114 is sent from client computing system 108 to computing system 118. In this case, computing system 118 primarily serves to host router 122. Router 122 is responsible for analyzing query 114 to determine which of the multiple NLG models the query should be sent to.

[0035] In the example scenario, router 122 is able to send query 114 to be processed by computing system 126 hosting HNLG model 130 or computing system 134 hosting LNLG model 138.

[0036] The computing system 100 can be modified in various ways. For example, the router 122 can be located at the client computing system 108 instead of the computing system 118. Positioning the router 122 at the client computing system 108 is particularly advantageous when one or more NLG models in the NLG model are located on the client computing system. Therefore, the client computing system 108 is shown as potentially including the router 122 (shown in dashed lines), and including one or both of the HNLG model 144 or the LNLG model 146.

[0037] Typically, client computing system 110 may be relatively resource-constrained compared to cloud-based systems. In some examples, client computing system 110 includes router 122 and LNLG model 146 because the NLG model can run satisfactorily on the client computing system. If router 122 determines that query 114 is sufficiently difficult, the router is able to forward the query to an HNLG model on another computing system, such as HNLG model 130 of computing system 126.

[0038] In another example, router 122 resides on computing system 118, which also includes one or both of HNLG model 150 or LNLG model 152. In one implementation of this scenario, router 122 routes query 114 between models 150 and 152, and the system does not need to include computing systems 126 and 134. In another example, computing system 118 uses one of NLG models 150 and 152 and routes query 114 to an NLG model of complementary complexity (e.g., an HNLG model, if LNLG model 152 is used) of computing system 126 or 134.

[0039] In another implementation, computing system 126 or computing system 134 may include both LNLG and HNLG models. Computing system 126 is shown as including LNLG model 158, while computing system 134 is shown as including HNLG model 160. In these scenarios, a single computing system, either computing system 126 or 134, can be used. For example, router 122 of computing system 118 can route queries between HNLG model 130 and LNLG model 158.

[0040] Computing environment 100 can include more than two computing systems hosting NLG models. Computing environment 100 is shown to include computing systems 164 up to the nth computing system 166, wherein a given computing system 164 can include one or more NLG models 168, which can be HNLG models, LNLG models, or combinations thereof.

[0041] In response to query 114, response 130 is provided. In some cases, response 130 is first provided by computing systems 126, 134, and 164 to system 118 of the managed router 122. System 118 then sends response 130 to client 108 for use by application 110. In other cases, computing systems 118, 126, 134, and 164 are able to provide the response directly to client 108 for use by application 110.

[0042] Figure 1Additional details about router 122 are also provided. As shown, router 122 includes a query analyzer 170, a machine learning model 172, and rules 174. Query analyzer 170 receives query 114 and submits information about the query to machine learning model 172. In a particular example, query analyzer 170 submits query 114 to embedding generator 171. Embedding generator 171 can be a context embedding generator, such as BERT (bidirectional encoder representation from a transformer), De-BERTa (where "De" stands for "not fluent"), or GloVe (global vector for text representation). In other scenarios, static embedding techniques such as Word2Vec can be used. Embedding generator 171 provides a vector-space representation of the query. This disclosure provides techniques that can assume queries with similar vector-space representations have comparable difficulty.

[0043] Using information about query 114 (such as contextual embeddings), machine learning model 172 is able to classify the query into a difficulty category or assign a difficulty score to the query. Query analyzer 170 is able to use rule 174 to determine where query 114 should be routed based on the output provided by machine learning model 172.

[0044] Machine learning model 172 can provide various metrics that can be used for query routing. In a specific example, machine learning model 172 provides the probability that a given query 114 corresponds to a query of a given difficulty level (such as a simple query). In other cases, machine learning model 172 can provide a binary label of whether the query is simple or difficult. However, it would be beneficial to provide machine learning results that can take more than two values, as this allows for greater flexibility in routing queries. For example, having a wider range of values ​​allows router tuning, which can be performed based on operating conditions (such as the load on a particular NLG model) or on the expected level of resource savings. That is, in some cases, it might be desirable to send more queries 114 to the less resource-intensive NLG model, even if the response may be of lower quality (with a larger quality gap compared to HNLG).

[0045] The machine learning model training component 180 can be used to provide a trained machine learning model 172. The machine learning model training component 180 uses a set of training queries 182. In some cases, the training queries 182 and the labels for said queries (as will be described further) are provided by humans. In other cases, one or both of the training queries 182 or the labels can be generated automatically.

[0046] In a particular implementation, the machine learning model training component 180 includes a query generator 184. The query generator 184 can include an NLG model 186 that generates queries. Queries can be generated by submitting appropriate requests to the NLG model 186, such as providing instructions to the LLM to generate queries, optionally utilizing instructions regarding the form or content of the queries, including the type of task to be performed when answering the queries.

[0047] In other implementations, query generator 184 can utilize other technologies. Template-based query generation is possible, employing predefined templates where specific keywords or phrases are dynamically inserted, providing a structured yet flexible approach to query creation. As another example, a data-driven approach can analyze existing datasets or query logs to generate queries that reflect real user behavior or common query patterns. Rule-based systems can be used to generate queries based on a predefined set of rules or heuristics, which can be customized for a specific domain or application.

[0048] In another example, a Markov model can be used. Markov models use probabilistic techniques, such as chains of statistically probable word sequences, to generate queries. These queries are able to capture the inherent randomness and unpredictability in language use. Syntactic and semantic manipulation techniques can be used to generate queries, involving altering existing queries or sentences by changing their structure (syntactic) or meaning (semantic) to create new, modified query forms.

[0049] When an unlabeled training query 182 is provided, a label generator 190 can be used to determine labels and apply them to the query. Labels can take many forms, but typically indicate the level of difficulty for the query. For example, the label can be a binary value indicating whether the query is easy or hard, or it can represent the probability that the query is easy or hard. In another example, labels can be applied, but adjustment factors can be considered when determining the "hard" or probability label. These techniques are further described in Example 3.

[0050] When generating labels, the label generator 190 can utilize the functionality of the NLG model 192 or the response quality evaluator 194. As will be further described in Examples 3 and 4, the NLG model 192 (such as LLM) can be provided with query results generated by the HNLG model and an LNLG model provided in response to the training query 182, and the response is required to be evaluated or scored. Additional information, such as the training query 182 or instructions on how the response should be evaluated or scored, can be provided to the NLG model 192.

[0051] The response quality evaluator 194 can use various techniques to measure the quality of the response provided by the NLG model. In one example, the BERT score is used. However, scores such as ROUGE (Gisting Evaluation Research for Memory Orientation) or BLEU (Bilingual Evaluation Research) can also be used, although the quality of the evaluation can vary depending on the technique used to measure the response quality. Another score that can be calculated for a query response is the METEOR (Measure for Translation Evaluation for Explicit Ranking) score. The BART score is another evaluation metric that can be used (such as described in Yuan et al., “BARTore: Evaulating Generated Text as Text Generation,” arXiv: 2106.11520 (June 2021)). NLG models can also be used to evaluate responses, such as using GPT models (such as GPT-4 from OPENAI). Human evaluation of the response can also be used using the disclosed techniques.

[0052] When training ML model 196 to provide ML model 172 used in router 122, embedding generator 171 can be used to generate embeddings for training query 182. The embeddings for training query 182 and its labels are provided to model 196 to provide results. A metric of loss is calculated and used to train ML model 196. The training process can correspond to minimizing loss function 198, where an example loss function is described in Example 3. Typically, loss function 198 attempts to correct the results of ML model 196 to be more consistent with the labels provided for training query 182.

[0053] Example 3 - Example Router Training and Implementation Example 3 provides a specific implementation of a machine learning-based router that routes between two NLG models (specifically LLM models) with different inference costs and accuracies. This allows NLG platforms (such as FACE, OPENAI, or C) to route queries across backend LLMs to reduce costs while providing satisfactory responses to users. The routing also allows the platform to dynamically tune the ratio of queries allocated to each model. In one example, this dynamic tuning can be based on quality requirements or specific uses or applications where some uses may be more tolerant of lower quality responses than others. The described routing technique also allows users with small models on local (edge) devices to invoke remote, more expensive platforms only for difficult queries, thus significantly reducing their costs (which can be costs in a financial sense, or costs in terms of latency or network usage, or the overall computational cost associated with query processing for local devices).

[0054] X and Z These are used to represent the input query space and the set of all possible output responses, respectively. Representing a large model, and This represents a small model. The goal is to learn the router. , so that if Then each user query Routed to small model Otherwise, it will be routed to a larger model. Here, 0 indicates a simple query, and 1 indicates a difficult query. The routing technique causes queries to be routed to a single NLG model during inference, rather than using an ensemble or cascade of NLG models, which could invoke multiple NLG models to solve a single query and incur significant computational overhead.

[0055] Various techniques can be used to evaluate the text of a query response by an NLG model. Metrics originally designed for machine translation and generalization, such as BLEU and ROUGE, can be used. However, other techniques can be used and may provide better results. In one example, LLM (such as using a general pre-trained transformer as used in products available from OpenAI) can be used to determine the ranking. Specifically, GPT-4 (OpenAI) can be used to provide the relative ranking between the generated output pairs. However, these techniques have computational costs and may not be sufficient to distinguish examples of similar quality.

[0056] In a specific example, the disclosed technique uses BART scores (Binary Total Regression Transformer) to evaluate the response quality of different models. BART scores are beneficial because they are relatively inexpensive to compute and the results are comparable to ground truth rankings. Ground truth rankings can correspond to human classifications of response quality. As will be described in the specification, ground truth rankings can be provided automatically; however, typically, the results of automated rankings correlate reasonably well with human-generated rankings. BART scores (response quality) can be used to evaluate the model response. of To express.

[0057] The cost of a model (such as an NLG model) may be precisely known or imprecisely known. Typically, cost can be expressed using various metrics, including latency, FLOPs, or energy consumption. Furthermore, in NLG inference, each of these metrics is influenced by several latent variables, such as different cue word templates, hardware capabilities, or network connectivity. That is, for example, different cue word templates affect the computing resources used, while network connectivity affects latency, beyond the NLG model's own contribution to the inference cost for a particular query. Additionally, some cost metrics may be more or less relevant in different use cases.

[0058] In some cases, model size can be used as a measure of model cost. That is, it can be assumed that inference using a smaller model is more efficient than using a larger model. Therefore, an improvement in one or more cost metrics can be expected by routing more queries to the smaller model. The cost advantage can be defined as the percentage of queries routed to the smaller model. However, the disclosed technique can be used to route queries between two different models based on query difficulty using other metrics of model cost.

[0059] Queries whose response quality for a small model (or otherwise "lower-cost" model) is close to that for a large model (or otherwise "more-cost" model) can be called "simple" queries. An example of a "simple" query is "Rewrite the following sentence in the present tense." I went to the store Both the smaller and larger models were able to provide the same "I'm going to the store" response.

[0060] As an example of a "hard query," consider the open-ended question "What are the benefits of owning a dog?" A smaller model might generate responses such as "Dogs are beneficial for physical activity." On the other hand, a larger model might generate more complete and detailed responses, such as: Owning a dog can bring many benefits to your life, including: 1. Companionship: Dogs are known for their loyalty and companionship. They provide emotional support and can help alleviate loneliness.

[0061] 2. Physical Activity: Owning a dog encourages physical activity. Daily walks, playtime, and exercise with your dog will help you stay active and maintain a healthy lifestyle.

[0062] 3. Mental Health: Interacting with dogs can reduce stress, anxiety, and depression. They can improve your mood and provide a sense of purpose and responsibility.

[0063] 4. Social Connections: Dogs can be excellent conversation starters, thereby increasing social interaction with other dog owners and people in the community.

[0064] 5. Unconditional love: Dogs offer unconditional love and affection. They are non-judgmental and provide constant love and comfort.

[0065] 6. Sense of security: Many dogs can provide a sense of security and protection, alerting you to potential threats or intruders.

[0066] 7. Routine: Dogs rely on routines, which can help you establish and maintain a structured daily schedule.

[0067] 8. Learn responsibility: Caring for a dog teaches you a sense of responsibility because you need to provide food, exercise, grooming, and healthcare.

[0068] 9. Emotional support: Studies have shown that dogs can sense and respond to human emotions, providing comfort during difficult times.

[0069] 10. Therapeutic benefits: Some dogs are trained as therapy or service animals to provide help and support to individuals with specific needs, such as people with disabilities or medical conditions.

[0070] 11. Happiness: The presence of a dog can bring joy and happiness to your life. Their playful nature and comical antics are both amusing and heartwarming.

[0071] 12. Reduce allergies: Studies have shown that early exposure to dogs may reduce the risk of children developing allergies and asthma.

[0072] 13. Sense of purpose: Owning a dog can give you a sense of purpose because you are taking care of their needs and well-being.

[0073] It's important to note that owning a dog comes with responsibilities, including the time, energy, and financial investment required for their care. Different breeds have different needs and characteristics, so choosing a dog that matches your lifestyle and preferences is crucial. Ultimately, the decision to own a dog should be based on your willingness to provide love and care for your furry companion.

[0074] As can be seen, the larger model provides significantly more detailed examples, which both humans and automated processes can rate as having a higher quality response than that provided by the smaller model.

[0075] As discussed, the disclosed techniques provide a router that dynamically identifies simple queries and routes them to smaller models, thereby providing a significant reduction in inference costs without compromising response quality. Note that simple queries, as defined herein, are not necessarily simple / cheap queries to be responded to; they are simply queries that smaller models can match larger models. Furthermore, more generally, and as described above, the disclosed techniques involve routing queries between two or more NLG models based on expected differences in response quality, but the routing criterion is not necessarily "size." For example, it may be beneficial to route a query to a specific NLG model, such as a local model, regardless of model size, even if the "preferred" model for a simple query is larger than another model.

[0076] For a pair of NLG models, query x The quality gap can be defined as In other words, the quality gap is the response of the small model. Response to large models The quality difference. The quality gap can be a random variable because NLG responses are typically nondeterministic. That is, different requests for the same query may receive different responses from the same model. This is in Figure 2 The figure in the middle shows that the llama-2 model is observed to provide a higher quality response than the FLAN-t5 model for a given query.

[0077] In this particular technique of Example 3, a BERT-type encoder model is used to route queries, which is trained on a dataset of representative queries. That is, as explained in Example 2, BERT or similar vector space embeddings are used as input during inference and for model training. The encoded model learns to predict scores for the input query. Since the router is an encoder model, a single pass through the query is sufficient to generate a score. In some cases, the cost of route determination can be considered negligible compared to the cost of generating a query response using a larger model. Therefore, it can be expected that using the router to route queries to a smaller model will not significantly diminish its cost advantage.

[0078] The ability to calculate router scores for simple queries as defined above is significant. The quality gap is greater than or equal to zero. The probability estimate is the appropriate candidate to be routed to the smaller model, because a large value of this probability is equal to the probability that the smaller model has a quality greater than or equal to that of the larger model. ).

[0079] However, routing queries to the smaller model can be beneficial, even if the results aren't at least as good as those for the larger model. However, queries will be routed to the larger model if a significant drop in quality is expected. In other words, in scenarios where the larger model is significantly more powerful than the smaller model (…). For appropriate t >0, the definition of a simple query can be broadened to... .in this case, t This is a positive threshold used as the cutoff point for routing queries to the larger model. In other words, it's the threshold where the expected quality of the smaller model is no lower than the quality provided by the larger model. t The query will continue to be routed to the smaller model.

[0080] Therefore, during use, by tuning the threshold for the score and routing queries with scores above the threshold to smaller models, a trade-off between the desired level of performance and accuracy can be achieved relative to query cost. For models with parameters... w For routers, we represent router scores as... , In other words, for a given query x In query space The router score is calculated, and the value is either 0 or 1, corresponding to two different NLG models that can be used. Assume... N One query x 1, ... xN The training set was used to discuss different router score designs.

[0081] To simplify the analysis, we can assume that the NLG model is a deterministic function that maps a query (as represented by a vector space embedding) to a single response in the output space. In practice, this can be reflected by obtaining a single response for each query from each model. Boolean labels can be assigned for each training query according to the following formula (…). ): in, It is an indicator function (provided with a value of 0 or 1). The BART score corresponds to the quality function. .

[0082] The example router is trained by minimizing the binary cross-entropy loss: in, It is the loss function (which depends on the model's parameters). w ),and This is for queries The probability predicted by the router. In this equation, the loss function... Subject to each query Predicted probabilities from routers The impact. Tags This is a binary indicator, where a value of 1 suggests that the query is "simple" and can be satisfactorily handled by a smaller model. Conversely, a value of 0 indicates a "difficult" query, which is better suited to a larger model.

[0083] The loss function is constructed such that if the label If it is 1, then when When maximized (because the second term in the summation decreases), the loss is minimized, encouraging the router to correctly classify simple queries. If If it is 0, then when When the loss is minimized (because the first term in the summation decreases), it correctly identifies a difficult query. This technique ensures that the router's predictions align with the predetermined difficulty level of the query.

[0084] Given a single response from each query of each model, it is possible to assign labels. Considered as targeting The estimation of router scores thus minimizes the above losses and encourages router score optimization. Close to the test query This deterministic router is called... .

[0085] Note that the assumption that AI models generate a single output for each query is more reasonable for tasks where truth labels are typically explicit and unique (such as image classification and video segmentation). In many cases, for NLP tasks, such as using NLG models, a single output for each query may not exist, likely due to the inherent ambiguity and complexity of natural language. LLMs can be used as nondeterministic generators to capture the inherent uncertainty of NLP tasks, such as in... Figure 2 As shown in the diagram, the nondeterminism primarily stems from the randomness during the decoding phase.

[0086] In this context, decoding refers to the process of generating human-readable text or a response based on a given input or prompt word (referred to as a query in this paper). A decoding strategy is a technique and scheme used to determine how this text generation process occurs. In some cases, users can control the level of uncertainty for LLM by selecting different decoding strategies, such as kernel sampling. Kernel sampling involves selecting words or lemmas for the generated text from a fixed portion of the probability distribution of possible words.

[0087] Users can also control the level of uncertainty by selecting a value for the hyperparameter temperature. The hyperparameter temperature controls the degree of randomness in the generated text. Higher temperature values ​​result in a higher level of randomness and diversity among the generated responses, while lower values ​​make the text generation more focused and deterministic. It can be expected that higher temperature values ​​will lead to a higher level of randomness and diversity among the generated responses. For black-box LLM APIs such as GPT-4 (OPENAI), it has been observed that even when the temperature is set to a minimum value of 0, it can still provide different responses to the same input query.

[0088] In one implementation, hard labels are used. Relaxed to soft label It can incorporate the uncertainty caused by the nondeterministic properties of NLG (such as LLM) into router training, whereby... The expected value (a measure of the central tendency or mean of a random variable, such as the sum of all possible values ​​multiplied by the probability of each value occurring) is obtained by sampling multiple responses from each model and averaging the results of the indicator function. For example, It can be estimated by sampling 10 responses from each model and calculating the sample average of the corresponding indicator function values. Note the hard labels. yes The higher variance estimate (because it is obtained from a single sample). That is, while hard labels are limited to values ​​between 0 and 1, soft labels can have continuous values ​​ranging from 0 to 1, allowing the values ​​to more accurately represent query difficulty. Therefore, improved performance can be expected with the following training loss. This probabilistic router can be called... .

[0089] In some cases, when the large model is significantly more powerful than the small model, Empirical estimates are often extremely small (in) Figure 2 Nearly 90% of the queries returned 0, with Flan-t5 (800m) being a small model and Llama-2 (13b) being a large model. This is because in some cases... Therefore, probability estimation can provide relatively weak training.

[0090] Since class labels are primarily used for routing decisions, it is possible to define new labels that can provide stronger training and correspondingly improved query routes. Specifically, for some The changed label Can be defined as .because- t <0, according to the definition of the tail distribution, Therefore, relaxing the labeling can provide a stronger signal for router training while still allowing simple queries to be those with such identifiers—queries targeting those identifiers. Having close to or The high probability. Visually, this corresponds to comparing the distribution of responses from a small model to a large model with a shifted distribution of responses from a large model (e.g., in...). Figure 2 In the middle, shift left for the curve of llama-2).

[0091] Regarding t The choice, assuming tail probability Located in [0, 1], t This can be achieved by maximizing the average pairwise difference between the transformed labels, thus pushing them apart as much as possible and providing a strong signal for training. Therefore, maximizing... t , t* It can be identified as: This optimization problem can be solved using grid search or other methods. Figure 3 Figure (c) shows the training data and the transformed labels. The distribution t The optimization objective has different values, where Figure (a) provides the values ​​before the transformation, and Figure (b) represents the grid search. The distribution is significantly more balanced, and therefore the router is expected to be more efficient. The router can be trained again by minimizing the loss: This probabilistic router is called .

[0092] Additional details regarding the disclosed techniques, including evaluation results, can be found at openreview.net / pdf?id=02f3mUtqnM, which is incorporated herein by reference in its entirety to the extent that it is inconsistent with this disclosure.

[0093] Example 4 - Automatic annotation of training query examples As already described, in at least some embodiments, AI models are used to classify queries as easy or hard, or to generate scores that can be used to classify queries as easy or hard. Specific models among these are trained using supervised learning techniques, where example queries are provided along with training labels in the form of easy / hard classifications or scores. While manual labeling of queries is possible, generating sufficient training data to train the AI ​​model to the desired level can be challenging.

[0094] Therefore, this disclosure provides various techniques that can be used for automatically labeling training queries. In one implementation, it is assumed that a larger model, or otherwise a more computationally expensive model, or another model designated for handling difficult queries, provides the "correct" response. In this case, the complexity of the query can be determined by comparing how close the response of the smaller model is to the response of the larger model. Again, a simple query can be defined as those queries for which the smaller model provides a response of similar quality to the response provided by the larger model.

[0095] However, this technique has various drawbacks. Despite having a more complex model and potentially additional processing power, in some cases, a large model may produce results inferior to a smaller model. In some cases, this difference may involve the nature of the query, such as whether the query is an "open-ended question" that may not have a single correct answer. That is, there is a problem if it is always assumed that the large model has a "better" response, because fewer queries will be routed to the smaller model, even if a person might determine that the response of the smaller model is equal to or even better than the response provided by the larger model.

[0096] As an example of how the additional processing available to more complex / larger models can lead to inferior query results compared to those from smaller models, some more complex models are capable of directly accessing information beyond what is used in the training dataset. For example, some models may have the ability to analyze the files or images provided in the query or independently access internet resources. Considering the ability to access internet resources, while in some cases this can allow the model to provide superior answers, including answering questions requiring information developed after the latest information date in the training dataset, in other cases, content retrieved from the internet may provide responses inferior to those generated by a larger model that might only be based on parameters learned from the training dataset. Therefore, even less complex models that do not use internet resources to answer queries can produce results superior to larger models.

[0097] In one example, this disclosure provides a technique in which an NLG model (which can be one or another model to which a query will be routed) is used to analyze the generated response based on a set of criteria, such as... Figure 4 As illustrated, the NLG model can be provided with inputs to Task Introduction 410 and Evaluation Criteria 414. Task Introduction 410 provides an overview of the tasks that the NLG model will perform when evaluating the query response, assessing whether the NLG response indicates that the NLG model has correctly evaluated the query, and evaluating the accuracy, precision, and depth of the NLG response. Evaluation Criteria 414 defines more specifically how the tasks are performed.

[0098] Task description 410 and evaluation criteria 414 can be provided to the NLG model to generate a set of evaluation steps 418 to be performed by the NLG model for query evaluation, wherein the evaluation steps are consistent for queries of similar types in at least one set of training data. That is, in at least some embodiments, different types of training data can be used to train the router. Some types of training data may already be labeled and therefore may not require... Figure 4 The technology. Similarly, query responses for other types of queries can benefit from using different technologies (or using...) Figure 4 The technology (but with different task definitions, evaluation instructions / standards, or evaluation steps) is used for scoring or classification. As described in the evaluation criteria, the score (IRGQ score – Intelligent Response Generation Quality) can take into account additional evaluation criteria, such as relevance.

[0099] exist Figure 4 In this context, the "input context" 422 (training query) is provided to both the small LLM and the large LLM to generate the corresponding input target 426 (in Figure 4 (Only one of them is shown in the image). Then, the NLG model 430 that performs the scoring uses the evaluation step 418 to analyze the input context 422 and the corresponding input target 426 (in the form of multiple responses provided by the NLG, which can be aspects of the response to a single query or an aggregation of multiple responses to repeated inputs of the same query) to produce an evaluation 434 that includes one or more scores 438, or more generally, a classification or category of the responses provided by the large or small model, or the relative performance of the model.

[0100] A score of 438 can be used to determine whether a smaller model performs better, comparable, or worse than a larger model. In some cases, the response with the highest score of 438 can be selected as the "correct" response. The techniques described in Example 3 can be adapted to use these scores in place of BART scores. However, in at least some cases, it is useful to use BART scores or similarity scores without employing an NLG model to determine correctness. Generally, BART scores are consistent with quality ratings from other techniques and human assessments, but generate less resource intensity compared to scores using an NLG model.

[0101] Because at least some NLG models are nondeterministic, it is possible to execute the query response generation and scoring process multiple times, and then compare these scores to determine which model generates the "correct" response to the query, such as for generating score aggregations. For example, a probability or confidence value for a score can be multiplied by a given score, and the resulting value is summed over all generated responses. The average of the scores can then be used as the score for a specific query for a specific NLG model.

[0102] In many cases, NLG models generate multiple candidate responses for a given query. These candidate responses can be evaluated using the NLG model and the top response provided as a response to the query. These evaluations can include scoring of candidate responses for quality, relevance, or consistency. In this case, instead of generating a single “top” query response for multiple inputs to the query, multiple candidate responses can be selected for the scoring process. Because candidate responses are generated as part of generating a single selected query response, this technique is computationally more efficient than inputting the same query multiple times into the model.

[0103] Various aspects of the NLG model can be adjusted as needed for the scoring process, such as when interacting with the model using an API that provides more control over its operation. For example, the "temperature" of the NLG model can be adjusted, such as to increase or decrease the amount of randomness or creativity in the response. In a specific example, the temperature is set to a relatively low value, meaning the response is relatively deterministic (the model will generate the same or more similar responses to a given query). Similarly, the "p" value of the NLG model can be adjusted, where the p value is associated with kernel sampling. That is, when selecting words in the response, words or terms are selected for the response up to a cumulative probability p. A p value of 1 indicates that the most likely word is being selected for the response, while a lower p value provides a wider range of choices.

[0104] Just as manually tagging or rating queries can be time-consuming and impractical for large datasets, it is essential to generate queries first. Therefore, NLG models can also be used to generate queries. These queries can be generated randomly or according to a specific set of instructions, such as query type (e.g., providing a factual response rather than generating or summarizing content). In another implementation, a crowdsourcing platform such as AMAZON MCHANICAL TURN (AMAZON.COM, Inc.) can be used to generate queries.

[0105] This approach can be extended to provide particularly realistic queries and query patterns. For example, given the conversational nature of NLG models (especially LLMs), users typically engage in a dialogue with the model that continues over time and can loop back to previous queries / responses, rather than simply having a series of discrete queries. These dialogues can be replicated by having a single NLG model generate and respond to its own queries, or by having multiple NLG interactions, such as having one NLG model generate a query and another NLG model generate a response, where the query-generating model can generate additional queries based on the response. Alternatively, two NLG models can alternately generate queries and responses.

[0106] Example 5 - Example operations for implementing the disclosed technology Figure 5 This is a flowchart of a process 500 for performing adaptive query routing using the disclosed techniques. At 505, a first query is received. At 510, a first set of one or more query difficulty metric values ​​is generated for the first query. At 515, based on the first set of the one or more query difficulty metric values, it is determined that the first query has a first query difficulty level. At 520, based on the determination that the first query has the first query difficulty level, the first query is routed to a first natural language generator model specified for the first query difficulty level.

[0107] At point 525, a second query is received. At point 530, a second set of one or more query difficulty metric values ​​is generated for the second query. At point 535, based on the second set of the one or more query difficulty metric values, it is determined that the second query has a second query difficulty level, which is different from the first query difficulty level. At point 540, based on the determination that the second query has the second query difficulty level, the second query is routed to a second natural language generator model specified for the second query difficulty level.

[0108] Example 6 - Example operations for implementing the disclosed technology Figure 6 This is a flowchart of a process 600 for training a router capable of performing adaptive query routing using techniques according to this disclosure. At 605, the router is trained to perform routing from a received query to a determined natural language generator model among a plurality of available natural language generator models. The training includes performing various operations for a given query in a set of training queries.

[0109] The operation includes generating a vector space embedding of a given query from a set of training queries at 610. At 615, a first natural language generator model from among multiple available natural language generator models is used to generate a first response for the given query. At 620, a second natural language generator model from among multiple available natural language generator models is used to generate a second response for the given query.

[0110] At point 625, a quality metric is calculated for the first response. At point 630, a quality metric is calculated for the second response. At point 635, a label is assigned to a given query using the quality metrics for the first and second responses. The label indicates the difficulty of the given query. At point 640, a query difficulty metric is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model.

[0111] At point 645, a machine learning model is trained to provide a trained machine learning model by minimizing a loss function that uses a label for the given query and a query difficulty metric for the given query. The router includes the trained machine learning model and selectively routes less difficult queries to the first natural language generator model, saving computational resources compared to using the second natural language generator model to process less difficult queries.

[0112] Example 7 - Example Technical Advantages The disclosed techniques offer numerous technical advantages, addressing the problem of high computational resource usage for queries processed by NLG, particularly when such queries can be satisfactorily answered using less resource-intensive NLG models. For queries where less resource-intensive NLG models can provide acceptable responses, routing the query to such a model rather than a more resource-intensive one saves significant computational resources, alleviates the burden on more complex, resource-intensive systems, and makes them easier to process, where the higher quality response from the more resource-intensive NLG model justifies its use. The disclosed techniques are also beneficial compared to other types of routing because they save computational resources without a significant loss of response quality. That is, users do not receive low-quality responses due to the use of less resource-intensive NLG models because the query is "filtered," ensuring that the less resource-intensive NLG model provides a response of acceptable quality.

[0113] Indeed, in some cases, manually examining a query may lead to the conclusion that a particular query is "difficult," but in reality, less resource-intensive NLG models provide higher quality results compared to more resource-intensive NLG models. The disclosed techniques, such as using a query classifier, are able to provide improved query responses for specific queries, even compared to using a more resource-intensive NLG model for all queries.

[0114] Example 8 - Additional Examples Example 1 is a computing system including at least one memory and at least one hardware processor coupled to said at least one memory. The computing system also includes one or more computer-readable storage media storing computer-executable instructions that, when executed, cause the computing system to perform a routing operation for a natural language generator model. This routing operation reduces computational resource usage for a specific query without a significant reduction in query response quality. The operation includes receiving a first query; generating a first set of one or more query difficulty metric values ​​for the first query; determining that the first query has a first query difficulty level based on the first set of one or more query difficulty metric values; and routing the first query to a first natural language generator model specified for the first query difficulty level based on the determination that the first query has the first query difficulty level.

[0115] Receive a second query. Generate a second set of one or more query difficulty metric values ​​for the second query. Based on the second set of one or more query difficulty metric values, determine that the second query has a second query difficulty level, which is different from the first query difficulty level. Based on the determination that the second query has the second query difficulty level, route the second query to a second natural language generator model specified for the second query difficulty level.

[0116] Example 2 includes the subject of Example 1 and further specifies that the operation includes generating a vector space embedding of the first query. The vector space embedding is submitted to a machine learning model to provide at least one query difficulty metric value for the one or more query difficulty metrics for the first query.

[0117] Example 3 includes the subject of Example 1 or Example 2, and further specifies that: the query difficulty measure in the one or more query difficulty measures corresponds to the probability that the evaluated query has the first query difficulty level.

[0118] Example 4 includes the subject of Example 3 and further specifies that: determining that the first query has a first query difficulty level based on a first set of one or more query difficulty metrics includes comparing a probability with a threshold probability and routing the first query to the first natural language generator model based on determining that the probability satisfies the threshold probability.

[0119] Example 5 includes the subject of Example 1 and further specifies that the operation includes training a router to perform routing for received queries, including the first query and the second query. The training includes generating a vector space embedding of a given query in a set of training queries. A first response is generated for the given query using the first natural language generator model. A second response is generated for the given query using the second natural language generator model. A quality metric is computed for the first response. A quality metric is computed for the second response. Labels are assigned to a given query in the set of training queries using the quality metric for the first response and the quality metric for the second response, the labels indicating whether the given query has a first difficulty level or a second difficulty level. A query difficulty metric value is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. A loss function using the label for the given query and the query difficulty metric value for the given query is minimized.

[0120] Example 6 includes the subject of Example 1 and further specifies that the operation includes training a router to perform routing for received queries, including the first query and the second query. The training includes generating a vector space embedding of a given query in a set of training queries. A first response is generated for the given query using the first natural language generator model. A second response is generated for the given query using the second natural language generator model. A quality metric is computed for the first response. A quality metric is computed for the second response. Labels are assigned to a given query in the set of training queries using the quality metric for the first response and the quality metric for the second response, the labels indicating a probability that the quality metric for the first response is at least as high as the quality metric for the second response. A query difficulty metric is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. A loss function using the label for the given query and the query difficulty metric for the given query is minimized.

[0121] Example 7 includes the subject of Example 1 and further specifies that the operation includes training a router to perform routing for received queries, including the first query and the second query. The training includes generating a vector space embedding of a given query in a set of training queries. A first response is generated for the given query using the first natural language generator model. A second response is generated for the given query using the second natural language generator model. A quality metric is computed for the first response. A quality metric is computed for the second response. Labels are assigned to a given query in the set of training queries using the quality metric for the first response and the quality metric for the second response, the labels indicating a probability that the quality metric for the first response is at least as high as the quality metric for the second response minus an adjustment value. A query difficulty metric is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. A loss function using the label for the given query and the query difficulty metric for the given query is minimized.

[0122] Example 8 includes the subject of Example 1 and further specifies that the operation includes training a router to perform routing for received queries, including the first query and the second query. The training includes generating a vector space embedding of a given query in a set of training queries. A first response is generated for the given query using the first natural language generator model. A second response is generated for the given query using the second natural language generator model. A quality metric is computed for the first response. A quality metric is computed for the second response. Labels are assigned to a given query in the set of training queries using the quality metric for the first response and the quality metric for the second response, the labels indicating the difficulty of the given query. A query difficulty metric value is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. A loss function using the label for the given query and the query difficulty metric for the given query is minimized.

[0123] Example 9 includes the subject of Example 8 and further specifies that: the computation of the quality metric for the first response is performed by a third natural language generator model, wherein the third natural language generator model is the first natural language generator model, the second natural language generator model, or a natural language generator model other than the first natural language generator model and the second natural language generator model.

[0124] Example 10 includes the subject of Example 9 and further specifies that the operation also includes submitting the first response to the third natural language generator model and instructions on how to calculate the quality metric for the query response.

[0125] Example 11 includes the subject of Example 1 and further specifies that the operation includes training a router to perform routing for received queries, including the first query and the second query. The training includes assigning labels to given queries in a set of training queries, the labels indicating whether the given query has a first difficulty level or a second difficulty level. A vector space embedding of the given queries in the set of training queries is generated. A first response is generated for the given query using the first natural language generator model. A second response is generated for the given query using the second natural language generator model. A quality metric is computed for the first response. A quality metric is computed for the second response. A query difficulty metric is generated by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model. A loss function using the labels for the given query and the query difficulty metric for the given query is minimized.

[0126] Example 12 includes the subject of Example 11 and further specifies that the operation includes receiving user input that assigns the label to the given query.

[0127] Example 13 is a method implemented in a computer system including at least one memory and at least one hardware processor coupled to the at least one memory. The method includes training a router that performs routing of a received query to a determined natural language generator model among a plurality of available natural language generator models. The training includes generating a vector space embedding of a given query in a set of training queries. A first natural language generator model among the plurality of available natural language generator models is used to generate a first response to the given query. A second natural language generator model among the plurality of available natural language generator models is used to generate a second response to the given query. A quality metric is computed for the first response. A quality metric is computed for the second response. A label indicating the difficulty of the given query is assigned to the given query using the quality metric for the first response and the quality metric for the second response. A query difficulty metric value is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. The machine learning model is trained to provide a trained machine learning model by minimizing a loss function that uses the label for the given query and the query difficulty metric for the given query. The router includes the trained machine learning model and selectively routes less difficult queries to the first natural language generator model, saving computational resources compared to using the second natural language generator model to process less difficult queries.

[0128] Example 14 includes the subject of Example 13, and also includes at least a portion of the set of training queries generated using a third natural language generator model.

[0129] Example 15 includes the subject of Example 13 or Example 14, and further specifies that: the calculation of the quality metric for the first response is performed by a third natural language generator model, wherein the third natural language generator model is the first natural language generator model, the second natural language generator model, or a natural language generator model other than the first natural language generator model and the second natural language generator model.

[0130] Example 16 includes the subject of Example 15, and also includes submitting the first response to the third natural language generator model and instructions on how to calculate a quality metric for the query response.

[0131] Example 17 incorporates the subject of any one of Examples 13-16 and further specifies that the label assigned to the given query indicates that the given query has a first difficulty level or a second difficulty level.

[0132] Example 18 incorporates the subject of any one of Examples 13-16 and further specifies that the label indicates a probability that the quality metric for the first response is at least as high as the quality metric for the second response.

[0133] Example 19 incorporates the subject matter of any of Examples 13-16 and further specifies that the label indicates the probability that the quality metric for the first response is higher than the quality metric for the second response minus the adjustment value.

[0134] Example 9 - Computing System Figure 7 A generalized example of a suitable computing system 700 in which the described innovations can be implemented is depicted. The computing system 700 is not intended to impose any limitation on the scope or functionality of this disclosure, as these innovations can be implemented in various general-purpose or special-purpose computing systems.

[0135] refer to Figure 7 The computing system 700 includes one or more processing units 710, 715 and memories 720, 725. Figure 7 In this diagram, the basic configuration 730 is included within the dashed lines. Processing units 710 and 715 execute computer-executable instructions, such as those for implementing the features described in Examples 1-8. The processing units can be general-purpose central processing units (CPUs), processors in application-specific integrated circuits (ASICs), or any other type of processor. In a multiprocessor system, multiple processing units execute computer-executable instructions to increase processing power. For example, Figure 7 A central processing unit 710 and a graphics processing unit or coprocessor 715 are shown. Physical memories 720, 725 may be volatile memories (e.g., registers, caches, RAM), non-volatile memories (e.g., ROM, EEPROM, flash memory, etc.), or some combination of both accessible to processing unit(s) 710, 715. Memories 720, 725 store one or more innovative software 780 described herein, implemented in the form of computer-executable instructions suitable for execution by processing unit(s) 710, 715.

[0136] The computing system 700 may have additional features. For example, the computing system 700 includes a storage device 740, one or more input devices 750, one or more output devices 760, and one or more communication connections 770, including input devices, output devices, and communication connections for user interaction. Interconnection mechanisms (not shown), such as buses, controllers, or networks, interconnect the components of the computing system 700. Typically, operating system software (not shown) provides an operating environment for other software executing in the computing system 700 and coordinates the activities of the components of the computing system 700.

[0137] The physical storage device 740 may be removable or non-removable and includes a magnetic disk, magnetic tape or tape cartridge, CD-ROM, DVD, or any other medium capable of storing information in a non-transitory manner and accessible within the computing system 700. The storage device 740 stores instructions for implementing one or more innovative software 780 described herein.

[0138] One or more input devices 750 may be touch input devices, such as a keyboard, mouse, pen or trackball, voice input device, scanning device, or another device that provides input to the computing system 700. One or more output devices 760 may be a monitor, printer, speaker, CD burner, or another device that provides output from the computing system 700.

[0139] One or more communication connections 770 enable communication with another computing entity via a communication medium. The communication medium transmits information such as computer-executable instructions, audio or video input or output, or other data in a modulated data signal. A modulated data signal is a signal having one or more characteristics that are set or altered in a manner that encodes information in the signal. By way of example and not limitation, the communication medium can use electrical, optical, RF, or other carrier waves.

[0140] Innovations can be described in the general context of computer-executable instructions that execute on a computing system, such as those included in a program module or on a target real or virtual processor. Typically, a program module or component includes routines, programs, libraries, objects, classes, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The functionality of a program module can be combined or split among program modules as needed in various embodiments. The computer-executable instructions for a program module can execute within a local or distributed computing system.

[0141] The terms “system” and “device” are used interchangeably herein. Unless the context clearly indicates otherwise, the terms do not imply any limitation on the type of computing system or computing device. Generally, a computing system or computing device can be local or distributed and can include any combination of dedicated hardware and / or general-purpose hardware with software that implements the functions described herein.

[0142] In the various instances described herein, modules (e.g., components or engines) can be "coded" to perform specific operations or provide specific functionality, thereby instructing a computer to execute the module to perform, cause to perform, or otherwise provide such functionality. While the functionality described with respect to a software component, module, or engine can be executed as discrete software units (e.g., programs, functions, class methods), it does not need to be implemented as discrete units. That is, functionality can be incorporated into a larger or more general program, such as as one or more lines of code within a larger or more general program.

[0143] For presentation purposes, detailed descriptions are provided using terms such as "determine" and "use" to describe computer operations in a computing system. These terms are high-level abstractions of operations performed by a computer and should not be confused with actions performed by humans. The actual computer operations corresponding to these terms vary depending on the implementation.

[0144] Example 10 - Cloud Computing Environment Figure 8 An example cloud computing environment 800 is depicted in which the described technologies can be implemented. The cloud computing environment 800 includes a cloud computing service 810. The cloud computing service 810 can include various types of cloud computing resources, such as computer servers, data repositories, networking resources, etc. The cloud computing service 810 can be centrally located (e.g., provided by a data center of an enterprise or organization) or distributed (e.g., provided by various computing resources located in different locations such as different data centers and / or in different cities or countries).

[0145] The cloud computing service 810 is utilized by various types of computing devices (e.g., client computing devices) such as computing devices 820, 822, and 824. For example, computing devices (e.g., 820, 822, and 824) can be computers (e.g., desktop or laptop computers), mobile devices (e.g., tablets or smartphones), or other types of computing devices. For example, computing devices (e.g., 820, 822, and 824) can utilize the cloud computing service 810 to perform computing operations (e.g., data processing, data storage, etc.).

[0146] Example 11 - Implementation Although the operations of some of the disclosed methods are described in a specific, sequential order for ease of presentation, it should be understood that this descriptive approach encompasses rearrangement unless a specific ordering for a particular language is described herein. For example, in some cases, the sequentially described operations may be rearranged or performed concurrently. Furthermore, for simplicity, the accompanying figures may not show the various ways in which the disclosed methods can be combined with other methods.

[0147] Any of the disclosed methods can be implemented as computer-executable instructions or a computer program product stored on one or more computer-readable storage media and executed on a computing device (e.g., any available computing device, including smartphones or other mobile devices that include computing hardware). A tangible computer-readable storage medium is any available tangible medium that can be accessed within a computing environment (e.g., one or more optical media discs such as DVDs or CDs, volatile memory components such as DRAM or SRAM, or non-volatile memory components such as flash memory or hard disk drives)). As an example and reference. Figure 7 Computer-readable storage media include memories 720 and 725 and storage device 740. The term "computer-readable storage medium" excludes signals and carrier waves. Additionally, the term "computer-readable storage medium" excludes communication connections (e.g., 770).

[0148] Any computer-executable instructions used to implement the disclosed technology and any data created and used during the implementation of the disclosed embodiments may be stored on one or more computer-readable storage media. The computer-executable instructions may be a dedicated software application or part of a software application, accessible or downloaded, for example, via a web browser or other software application, such as a remote computing application. Such software may be executed, for example, on a single local computer (e.g., any suitable commercially available computer) or in a network environment (e.g., via the Internet, a wide area network, a local area network, a client-server network, such as a cloud computing network or other such networks) using one or more network computers.

[0149] For clarity, only specific selected aspects of the software-based implementation are described. It should be understood that the disclosed techniques are not limited to any particular computer language or program. For example, the disclosed techniques can be implemented using software written in C++, Java, Perl, JavaScript, Python, Ruby, ABAP, SQL, Adobe Flash, or any other suitable programming language, or in some examples, using markup languages ​​such as HTML or XML, or a combination of suitable programming languages ​​and markup languages. Similarly, the disclosed techniques are not limited to any particular computer or type of hardware.

[0150] Furthermore, any software-based implementation (including, for example, computer-executable instructions for causing a computer to perform any of the disclosed methods) can be uploaded, downloaded, or remotely accessed via suitable communication means. Such suitable communication means include, for example, the Internet, the World Wide Web, intranets, software applications, cables (including fiber optic cables), magnetic communication, electromagnetic communication (including RF, microwave, and infrared communication), electronic communication, or other such communication means.

[0151] The disclosed methods, apparatus, and systems should not be construed as limiting in any way. Rather, this disclosure is intended to highlight all novel and non-obvious features and aspects of the various disclosed embodiments, individually and in various combinations and sub-combinations. The disclosed methods, apparatus, and systems are not limited to any particular aspect or feature or combination thereof, and the disclosed embodiments are not required to have any one or more particular advantages or to solve any particular problem.

[0152] The techniques from any example can be combined with the techniques described in any one or more of the other examples. Given the many possible embodiments to which the principles of the disclosed techniques can be applied, it should be understood that the illustrated embodiments are examples of the disclosed techniques and should not be considered as limiting the scope of the disclosed techniques. Rather, the scope of the disclosed techniques includes what is covered by the scope and spirit of the appended claims.

Claims

1. A computing system, comprising: At least one memory; At least one hardware processor is coupled to the at least one memory; as well as One or more computer-readable storage media storing computer-executable instructions, which, when executed, cause the computing system to perform routing operations for a natural language generator model, the natural language generator model reducing computational resource usage for a specific query without a significant degrade in query response quality, the operations including: Receive the first query; Generate a first set of one or more query difficulty metric values ​​for the first query, which are sets of one or more query difficulty metrics. The first query is determined to have a first query difficulty level based on a first set of one or more query difficulty metrics; Based on the determination that the first query has the first query difficulty level, the first query is routed to the first natural language generator model specified for the first query difficulty level; Receive the second query; For the second query, generate a second set of one or more query difficulty metric values ​​for the set of one or more query difficulty metrics; The second query is determined to have a second query difficulty level based on a second set of one or more query difficulty metrics, wherein the second query difficulty level is different from the first query difficulty level; and Based on the determination that the second query has the second query difficulty level, the second query is routed to the second natural language generator model specified for the second query difficulty level.

2. The computing system according to claim 1, further comprising: Generate the vector space embedding of the first query; The vector space is embedded and submitted to a machine learning model to provide at least one query difficulty metric value for the one or more query difficulty metrics for the first query.

3. The computing system according to claim 1 or claim 2, wherein, The query difficulty measure in the one or more query difficulty measures corresponds to the probability that the evaluated query has the first query difficulty level.

4. The computing system according to claim 3, wherein, Determining that the first query has a first query difficulty level based on a first set of one or more query difficulty metrics includes: comparing the probability with a threshold probability, and routing the first query to the first natural language generator model based on determining that the probability satisfies the threshold probability.

5. The computing system according to any one of claims 1-4, wherein the operation further comprises: Training a router to execute routes for received queries, including the first query and the second query, the training includes: For a given query in the set of training queries: Generate the vector space embedding of a given query in the set of training queries; Use the first natural language generator model to generate a first response to the given query; Use the second natural language generator model to generate a second response to the given query; Calculate the quality metric for the first response; Calculate the quality metric for the second response; Labels are assigned to a given query in the set of training queries using a quality metric for the first response and a quality metric for the second response, the labels indicating whether the given query has the first difficulty level or the second difficulty level; and A query difficulty metric is generated by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model; and Minimize the loss function using the label for the given query and the query difficulty metric for the given query.

6. The computing system according to any one of claims 1-4, wherein the operation further comprises: Training a router to execute routes for received queries, including the first query and the second query, the training includes: For a given query in the set of training queries: Generate the vector space embedding of a given query in the set of training queries; Use the first natural language generator model to generate a first response to the given query; Use the second natural language generator model to generate a second response to the given query; Calculate the quality metric for the first response; Calculate the quality metric for the second response; A label is assigned to the given query in the set of training queries using a quality metric for the first response and a quality metric for the second response, the label indicating a probability that the quality metric for the first response is at least as high as the quality metric for the second response; and A query difficulty metric is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. Minimize the loss function using the label for the given query and the query difficulty metric for the given query.

7. The computing system according to any one of claims 1-4, wherein the operation further comprises: Training a router to execute routes for received queries, including the first query and the second query, the training includes: For a given query in the set of training queries: Generate the vector space embedding of a given query in the set of training queries; Use the first natural language generator model to generate a first response to the given query; Use the second natural language generator model to generate a second response to the given query; Calculate the quality metric for the first response; Calculate the quality metric for the second response; A label is assigned to a given query in the set of training queries using a quality metric for the first response and a quality metric for the second response, the label indicating the probability that the quality metric for the first response is higher than the quality metric for the second response minus an adjustment value; and A query difficulty metric is generated by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model; and The machine learning model is trained by minimizing a loss function that uses the label for the given query and the query difficulty metric for the given query.

8. The computing system according to any one of claims 1-4, wherein the operation further comprises: Training a router to execute routes for received queries, including the first query and the second query, the training includes: For a given query in the set of training queries: Generate the vector space embedding of a given query in the set of training queries; Use the first natural language generator model to generate a first response to the given query; Use the second natural language generator model to generate a second response to the given query; Calculate the quality metric for the first response; Calculate the quality metric for the second response; Labels are assigned to the given query in the set of training queries using a quality metric for the first response and a quality metric for the second response, the labels indicating the difficulty of the given query; A query difficulty metric is generated by submitting the vector space embedding of the given query and the label for the given query to a machine learning model. The machine learning model is trained by minimizing a loss function that uses the label for the given query and the query difficulty metric for the given query.

9. The computing system according to claim 8, wherein, The calculation of the quality metric for the first response is performed by a third natural language generator model, wherein the third natural language generator model is the first natural language generator model, the second natural language generator model, or a natural language generator model other than the first natural language generator model and the second natural language generator model.

10. The computing system according to claim 9, further comprising: Submit the first response and instructions on how to calculate the quality metric for the query response to the third natural language generator model.

11. The computing system according to any one of claims 1-4, wherein the operation further comprises: Training a router to execute routes for received queries, including the first query and the second query, the training includes: For a given query in the set of training queries: A label is assigned to a given query in the set of training queries, the label indicating whether the given query has a first difficulty level or a second difficulty level; Generate the vector space embedding of a given query in the set of training queries; Use the first natural language generator model to generate a first response to the given query; Use the second natural language generator model to generate a second response to the given query; Calculate the quality metric for the first response; Calculate the quality metric for the second response; A query difficulty metric is generated by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model; and Minimize the loss function using the label for the given query and the query difficulty metric for the given query.

12. The computing system according to claim 11, further comprising: Receive user input that assigns the label to the given query.

13. A method implemented in a computing system, the computing system including at least one hardware processor and at least one memory coupled to the at least one hardware processor, the method comprising: Training a router that performs routing for a selected natural language generator model among multiple available natural language generator models received in a query, the training comprising: For a given query in the set of training queries: Generate the vector space embedding of a given query in the set of training queries; A first response to the given query is generated using the first natural language generator model among the plurality of available natural language generator models; A second response to the given query is generated using a second natural language generator model from the plurality of available natural language generator models; Calculate the quality metric for the first response; Calculate the quality metric for the second response; A label is assigned to the given query using a quality metric for the first response and a quality metric for the second response, the label indicating the difficulty of the given query; A query difficulty metric is generated by submitting the vector space embedding of the given query and the labels for the given query to a machine learning model; and The machine learning model is trained to provide a trained machine learning model by minimizing a loss function that uses the label for the given query and the query difficulty metric for the given query. The router includes the trained machine learning model and selectively routes less difficult queries to the first natural language generator model to save computational resources compared to using the second natural language generator model to process the less difficult queries.

14. The method of claim 13, further comprising: At least a portion of the set of training queries is generated using a third natural language generator model.

15. The method according to claim 13, wherein, The calculation of the quality metric for the first response is performed by a third natural language generator model, wherein the third natural language generator model is the first natural language generator model, the second natural language generator model, or a natural language generator model other than the first natural language generator model and the second natural language generator model.

Images