Utilizing generative models in responding to multi-aspect queries

Abstract

Implementations utilize a generative model (e.g., a large language model (LLM)) to generate a plurality of candidate sub-queries for a multi-aspect natural language (NL)-based input, where each of the candidate sub-queries potentially aims at an aspect or problem of the multi-aspect NL-based input. Those implementations further use one or more evaluation metrics to select a subset of candidate queries from the plurality of candidate sub-queries. Those implementations further obtain at least one corresponding search result for each of the candidate sub-queries of the selected subset in response to selecting the subset of candidate queries. Those implementations further generate a response to the NL-based input based on the corresponding search results for the candidate sub-queries of the subset, and cause the response to be rendered in response to the NL-based input.

Description

Background Technology

[0001] Search systems can identify and present useful search results for a wide variety of queries. For example, for many queries, the top-ranked search results or the top... N One or more search results within a search result can efficiently resolve a query. For example, viewing the search results and / or the underlying search result documents in response to a query can efficiently resolve a query.

[0002] However, for some queries, the search system is unable to determine and present useful search results. For example, the search system may fail to determine useful results for multifaceted and / or noisy queries. For instance, none of the determined results may be useful, or the determined results may only address a subset of the aspects of a multifaceted query. A multifaceted query is multifaceted because it involves two or more aspects (e.g., a topic or question). A noisy query is noisy because it includes elements irrelevant to the aspects (e.g., not necessary for resolving the aspects) and fails to characterize one or more parts of the aspects.

[0003] Because search systems fail to determine useful results for multifaceted and / or noisy queries, users seeking resolution of multifaceted queries through search systems are forced to manually break down the multifaceted query into multiple individual queries, formulate and submit each of those queries separately, and view the search results for those individual queries separately. This prolongs the duration of user interaction with the search engine via client devices, resulting in significant usage of battery, processor, and / or other typically limited resources of client devices. As an alternative to utilizing search systems in seeking resolution of multifaceted queries, some users post multifaceted queries to forums and wait for answers formulated by other users via corresponding client devices. However, formulating posts and waiting for answers similarly prolongs the duration of user interaction with client devices in resolving multifaceted queries, resulting in significant usage of typically limited resources of client devices.

[0004] As a non-restrictive example, suppose the following query is given: “I'm moving to a new city and into a 2,500 square foot house and need to find a wireless router setup that will cover the whole house. A smart thermostat would also be ideal to replace the existing analog one, especially since the weather will be different where I'm moving. I also need a vacuum cleaner.” This example query is multifaceted because it includes a first aspect or problem related to finding a suitable wireless router setup for a 2,500 square foot house, a completely different second aspect or problem related to finding a smart thermostat, and a completely different third aspect or problem related to finding a vacuum cleaner. Furthermore, the example query is noisy because it contains parts irrelevant to the aspects and fails to characterize them, such as "I'm moving to a new city" and "weather will be different where I'm moving." A search system that processes the entire search query may fail to identify and present useful search results. For example, a search engine might search based on the entire query. This could result in not returning, or only returning limited search results that address wireless routers, smart thermostats, and vacuum cleaners—and any limited search results might be of lower quality and / or only superficially address each aspect of the aspect.

[0005] Separately, various generative models have been proposed that can be used to process natural language (NL) content and / or other inputs to generate outputs that reflect generative content in response to the inputs. For example, Large Language Models (LLMs) have been developed that can be used to process NL content and / or other inputs to generate LLM outputs that reflect NL content and / or other content in response to the inputs. For example, LLM can be used to process NL content such as "how to change DNS settings on Acme router" to generate LLM output that reflects several responsive NL sentences, such as: "First, type the router's IP address in a browser; the default IP address is 192.168.1.1. Then enter the username and password; the defaults are admin and admin. Finally, select the advanced settings tab and find the DNS settings section." However, current utilization of generative models has one or more drawbacks.

[0006] As an example, an LLM can be used as part of a text-based conversational application to generate responses to text input / queries provided by the application's user. However, multifaceted and / or noisy input can be difficult for an LLM to handle effectively. For instance, using an LLM to process multifaceted and noisy NL-based input and using the resulting LLM output to generate a response to noisy NL-based input may result in a response with limited usefulness. For example, the response may fail to address all (or even any) aspects of the multifaceted NL-based input, leading the user to further interact with the LLM and / or a separate search engine while attempting to resolve the multifaceted NL-based input. This significantly prolongs the duration of user interaction with the LLM and / or search engine via the client device, resulting in significant use of the client device's typically limited resources. Summary of the Invention

[0007] The implementations described herein can be used to reduce the duration of interactions via client devices in resolving multifaceted and / or noisy NL-based inputs, and to reduce the duration of interactions by interacting with and operating within the technical constraints of a generative model (e.g., LLM) and a search system. Some of these implementations generate and present a response to a single multifaceted and / or noisy NL-based input (e.g., a text query submitted via a search system interface), wherein the response resolves each of the multiple aspects of the NL-based input and / or does not resolve the noisy aspects of the NL-based input. This makes it possible to resolve multifaceted and / or noisy NL-based inputs without manually formulating the NL-based input into multiple separate queries, manually submitting each of those queries separately, and manually viewing the respective search results for those separate queries separately after the corresponding manual submission. In these and other ways, the implementations can be used to reduce the duration of user interactions with a search system via client devices. Although any given user may decide to provide subsequent NL-based inputs to the search system after providing a response to a multifaceted query generated according to the implementation disclosed herein, a “average” reduction in the number of subsequent NL-based inputs to the search system can provide significant benefits in terms of computational resource usage.

[0008] More specifically, the implementations disclosed herein utilize LLM or other generative models to generate multiple candidate subqueries for multifaceted NL-based input (e.g., text input submitted via a search system interface), wherein each candidate subquery potentially addresses an aspect or question of the multifaceted NL-based input. Those implementations further employ one or more evaluation metrics to select a subset of candidate subqueries from the multiple candidate subqueries. Those implementations further, in response to the selection of the subset of candidate subqueries, obtain at least one corresponding search result for each candidate subquery in the selected subset. Those implementations further generate a response to the NL-based input based on the corresponding search results for the subset of candidate subqueries, and cause this response to be rendered in response to the NL-based input.

[0009] In generating candidate subqueries using LLM, many implementations seek to generate different groups of subqueries to ensure that the generated subqueries include at least one subquery for each aspect of the NL-based input. For example, some implementations may generate two, three, or more subquery generation hints, each including all or part of the NL-based input but each including different additional content (e.g., different few-sample examples and / or other different content) relative to other subquery generation hints in the subquery generation hints. For instance, three subquery generation hints may be generated, each including the NL-based input, but the first hint may include a first few-sample example, the second hint may alternatively include a completely different second few-sample example, and the third hint may alternatively include a completely different third few-sample example. Each few-sample example may include a corresponding example of the multifaceted NL-based input and a corresponding ground truth subquery for the multifaceted NL-based input.

[0010] Continuing the example, an LLM can be used to process a first subquery generation hint to generate a first LLM output, which is then used (e.g., decoded) to determine multiple first candidate subqueries. Similarly, an LLM can be used to process a second subquery generation hint to generate a second LLM output, which is then used to determine multiple second candidate subqueries. Due to the uniqueness of the hint, second candidate subqueries can include one or more subqueries that are different from those in the first candidate subqueries. Likewise, an LLM can be used to process a third subquery generation hint to generate a third LLM output, which is then used to determine multiple third candidate subqueries, one or more of which may be different from those in the first and second candidate subqueries. Therefore, by making multiple calls to the LLM based on different hints, large and diverse groups of candidate subqueries can be generated. Besides, or instead of, different few-sample hint examples, other aspects of the hint can be changed in one or more calls, and / or LLM parameters (e.g., temperature, seed, etc.) can be changed in one or more calls.

[0011] While multiple calls to LLM can generate large and diverse sets of candidate subqueries, the implementation disclosed herein also recognizes that the resulting set of candidate subqueries may include one or more subqueries unrelated to the aspects of resolving the NL-based input, and / or may include more candidate subqueries than are needed for resolving the NL-based input. Furthermore, the implementation recognizes that blindly obtaining search results for all candidate subqueries leads to overutilization of search system resources, and that combining all such search results into a response may require significant time to view the response and / or substantial screen and / or audio resources to view the response.

[0012] Therefore, the implementation disclosed herein utilizes one or more evaluation metrics to select a subset of candidate subqueries. These evaluation metrics seek to ensure that the selected subqueries are relevant to the NL-based input and / or that the selected subqueries do not duplicate another selected subquery. In these and other ways, it can be ensured that search system resources are used only for selected candidate subqueries that have been objectively determined to be relevant to and non-duplicative of the NL-based input. This also ensures that the responses generated based on the corresponding search results can be rendered and / or viewed efficiently.

[0013] The evaluation metrics used in determining whether to select a given candidate subquery for inclusion in a subset may include a diversity metric and / or a relevance metric for the given candidate subquery. A relevance metric may indicate how relevant the given candidate subquery is to the NL-based input. Each diversity metric may indicate the degree of similarity between the given candidate subquery and any already selected candidate subqueries (if any). For example, in determining whether to select a given candidate subquery, it may be selected only if (a) its relevance metric meets a threshold (indicating relevance to the NL-based input) and (b) (i) the selected set is empty or (ii) each diversity metric between the given candidate subquery and any already selected candidate subqueries meets a threshold (indicating diversity relative to any already selected candidate subquery).

[0014] In some implementations, an encoding neural network model (e.g., Word2Vec or other statement encoders) can be used to process NL-based input to generate NL-based input codes, and can be used to process each candidate subquery separately to generate a corresponding subquery code. In some of those implementations, the relevance metric for the candidate subqueries can be based on the cosine distance (and / or other distance metrics) between the NL-based input codes and the corresponding subquery codes. Similarly, the diversity metric between two candidate subqueries can be based on the distance metric between two corresponding subquery codes. Therefore, in those implementations, the generated codes can be used to efficiently (e.g., using a simple cross product) generate relevance and / or diversity metrics. Note that encoding neural network models can be computationally more efficient than LLMs. For example, the encoding neural network model can include 10% or more fewer parameters, allowing it to process given data with less memory and / or in fewer processor loops. Further note that the encoding neural network generates... N One code can be used to generate more than N One metric—having a computationally efficient cross product or other simple distance metric. For example, NL-based input encoding, first subquery encoding, second subquery encoding, and third subquery encoding can be used to generate two metrics that produce three relevance metrics and three metrics that produce three diversity metrics.

[0015] Some implementations may additionally or alternatively utilize LLM to determine relevance metrics (e.g., using "on a scale of 1-10 how related is [subquery] to [NL-based input]" to prompt the LLM) and / or diversity metrics (e.g., using "is [candidate prompt] sufficiently diverse relative to [already selected prompt(s)]" to prompt the LLM). However, various implementations may at least selectively (e.g., when server load is high) utilize the encoded neural network model techniques described herein to reduce computational resource utilization and / or latency.

[0016] The implementation retrieves search results only for subqueries of a selected subset and generates a response to the NL-based input based on those search results. For example, top-ranked search results for each subquery can be retrieved, and a response can be generated based on these top-ranked search results for each subquery. As an example, the response could include each subquery, and under each subquery, top-ranked search results for that subquery could be included. As another example, the response could include top-ranked search results for each subquery, optionally without any indication of the subqueries or which subqueries respond to which results. As yet another example, the response could be a shortened summary of the top-ranked search results, such as a shortened summary generated based on each search result processed using LLM and a summary hint (e.g., "generate a summary of [search results]").

[0017] In some implementations, before generating and executing multiple subqueries for NL-based input, it is first determined, based on one or more criteria for the NL-based input, whether to do so (e.g., rather than simply submitting the entire NL-based input to the search system). These criteria may seek to ensure that the generation and / or execution of multiple search queries is not performed if the search system can generate a helpful response to the entire NL-based input. In other words, these criteria seek to ensure that the generation and / or execution of multiple search queries is performed when it is beneficial (and achieves the corresponding technical benefit), but not unnecessarily when it is not beneficial. In some implementations, these criteria may include, for example, criteria based on: the length of the NL-based input, the frequency of submission of all or part of the NL-based input, the quality score and / or other metrics of the search results for the entire NL-based input, the current server load, and / or other criteria. For example, for a given NL-based input, the generation and execution of multiple subqueries may occur based on the given NL-based input having a length greater than a threshold, being submitted at a frequency less than a threshold, and / or having low-quality results. As another example, given an alternative or NL-based input, multiple subqueries may be generated and executed based on the current server load being less than a threshold.

[0018] In some implementations, LLMs or other generative models may include at least hundreds of millions of parameters. In some of those implementations, the generative model includes at least billions of parameters, such as a hundred billion or more. In some additional or alternative implementations, the generative model is a sequence-to-sequence model, a Transformer-based model, and / or may include an encoder and / or decoder (e.g., a decoder-only generative model). A non-limiting example of an LLM is Google's Pathways language model (PaLM). Another non-limiting example of an LLM is Google's Language Model for Conversational Applications (LaMDA). Yet another non-limiting example of a generative model is Google's Gemini model suite. However, and as noted, it should be observed that the LLM described herein is an example of a generative machine learning model and is not intended to be limiting.

[0019] The above is presented as an overview of only some of the implementations disclosed in this article. This article discloses these and other implementations in more detail. Attached Figure Description

[0020] Figure 1 A block diagram depicts an example environment that illustrates various aspects of this disclosure and in which some of the implementations disclosed herein can be implemented.

[0021] Figure 2 It describes the various implementation methods Figure 1 Examples of how various engines can interact in generating responses to natural language (NL) based input.

[0022] Figure 3 A flowchart is depicted illustrating example methods for generating responses to NL-based inputs according to various implementations.

[0023] Figure 4 The diagram illustrates various implementation methods. Figure 3 The flowchart for example of box 366.

[0024] Figure 5A , Figure 5B and Figure 5C An example client device for rendering a graphical interface is described, which includes different examples of responses that can be generated according to various implementations.

[0025] Figure 6 Example architectures of computing devices based on various implementations are depicted. Detailed Implementation

[0026] Turn now Figure 1The diagram depicts an example environment illustrating various aspects of this disclosure and in which implementations disclosed herein can be carried out. The example environment includes a client device 110 and a response system 120.

[0027] In some implementations, all or some aspects of the response system 120 can be implemented locally on the client device 110. In additional or alternative implementations, all or some aspects of the response system 120 can be implemented from, for example... Figure 1 The depicted client device 110 is implemented remotely (e.g., at a remote server). In those implementations, the client device 110 and the response system 120 may be communicatively coupled to each other via one or more networks 199, such as one or more wired or wireless local area networks (“LANs”, including Wi-Fi, mesh networks, Bluetooth, near field communication, etc.) or wide area networks (“WANs”, including the Internet) .

[0028] Client device 110 may be one or more of the following: desktop computer, laptop computer, tablet computer, mobile phone, vehicle computing device (e.g., in-vehicle communication system, in-vehicle entertainment system, in-vehicle navigation system), independent interactive speaker (optionally with a display), smart home appliance (such as a smart TV), and / or user's wearable device including a computing device (e.g., user's watch with computing device, user's glasses with computing device, virtual or augmented reality computing device). Additional and / or alternative client devices may be provided.

[0029] Client device 110 may execute one or more software applications via application engine 115, through which NL-based input can be submitted, and / or NL-based output and / or other outputs in response to the NL-based input can be rendered (e.g., audibly and / or visually). Application engine 115 may execute one or more software applications separate from the operating system of client device 110 (e.g., software applications installed "on top" of the operating system), or alternatively, may be implemented directly by the operating system of client device 110. For example, application engine 115 may execute a web browser or automation assistant installed on top of the operating system of client device 110. As another example, application engine 115 may execute a web browser software application or automation assistant software application integrated as part of the operating system of client device 110. Application engine 115 (and one or more software applications executed by application engine 115) may interact with response system 120.

[0030] In various implementations, client device 110 may include a user input engine 111 configured to detect user input provided by a user of client device 110 using one or more user interface input devices. For example, client device 110 may be equipped with one or more microphones that capture audio data, such as audio data corresponding to the user's spoken words or other sounds in the environment of client device 110. Alternatively, client device 110 may be equipped with one or more visual components configured to capture visual data corresponding to images and / or movements (e.g., gestures) detected in the field of view of one or more visual components. Alternatively, client device 110 may be equipped with one or more touch-sensitive components (e.g., keyboard and mouse, stylus, touchscreen, touch panel, one or more hardware buttons, etc.) configured to capture signals corresponding to touch input directed at client device 110.

[0031] Some examples of NL-based input described herein can be queries that formulate an NL response based on user input provided by the user of client device 110 and detected by user input engine 111. For example, the query can be a typing query typed via a physical or virtual keyboard, a suggested query selected via the touchscreen or mouse of client device 110, a spoken voice query detected via the microphone of client device 110 (and optionally directed to an automated assistant that is at least partially executed at client device 110), or an image or video query based on visual data captured by the vision component of client device 110 (or based on NL input generated based on image processing using, for example, object detection models, caption generation models, etc.).

[0032] In various implementations, client device 110 may include rendering engine 112 configured to render content (e.g., responses) to be presented to a user of client device 110 in an audible and / or visual manner using one or more user interface output devices. For example, client device 110 may be equipped with one or more speakers that enable the content to be provided to the user audibly via client device 110. Alternatively, client device 110 may be equipped with a display or projector that enables the content to be provided to the user visually via client device 110.

[0033] In various implementations, client device 110 may include a context engine 113 configured to determine the context (e.g., current or recent context) of client device 110 and / or its users (e.g., active users of client device 110 when client device 110 is associated with multiple users). In some of those implementations, context engine 113 may determine the context based on, for example, user interaction data characterizing current or recent interactions of client device 110 and / or its users; location data characterizing current or recent location of client device 110 and / or its users; user attribute data characterizing one or more attributes of the users of client device 110; user preference data characterizing one or more preferences of the users of client device 110; user profile data characterizing profiles of the users of client device 110; third-party (3P) data indicating one or more response evaluation criteria defined by 3P; and / or any other data accessible via client device data database 110A or otherwise by context engine 113.

[0034] For example, the context engine 113 may determine the current context based on the current state of the conversation (e.g., considering one or more recent inputs provided by the user during the conversation), profile data, and / or the current location of the client device 110. For example, the context engine 113 may determine the current context of "best landmarks to visit in London" based on recently issued queries, profile data, and / or the current or expected future location of the client device 110 (e.g., based on calendar information associated with the user that the context engine 113 can access). As another example, the context engine 113 may determine the current context based on which software application is active in the foreground of the client device 110, the current or recent state of the active software application, and / or the content currently or recently rendered by the active software application. The context determined by the context engine 113 can be used, for example, to supplement or rewrite NL-based inputs specified based on user input, generate implicit NL-based inputs (e.g., implicit queries or prompts specified independently of any explicit NL-based inputs provided by the user of the client device 110), and / or determine the submission of implicit NL-based inputs and / or render results (e.g., NL-based outputs) for implicit NL-based inputs.

[0035] In various implementations, client device 110 may include an implicit input engine 114 configured to: generate implicit NL-based input independently of any explicit NL-based input provided by a user of client device 110; optionally submit implicit NL-based input independently of any explicit NL-based input requested by a user; and / or optionally cause rendering to respond to implicit NL-based input independently of any explicit NL-based input requested by rendering. For example, implicit input engine 114 may use one or more past or current contexts from context engine 113 to generate implicit NL-based input, determine the submission of implicit NL-based input, and / or determine the response that causes rendering to respond to implicit NL-based input. For example, implicit input engine 114 may automatically generate and submit implicit queries or implicit hints based on the one or more past or current contexts. Furthermore, the implicit input engine 114 may automatically push responses generated in response to implicit queries or implicit prompts, causing the responses to be rendered automatically, or may automatically push notifications of the responses (such as optional notifications), which, when selected, cause the responses to be rendered. Alternatively, the implicit input engine 114 may submit corresponding implicit NL-based input at regular or irregular intervals, causing corresponding responses to be provided automatically (or notifications of corresponding responses to be provided automatically). For example, implicit NL-based input may be “Automation Assistant News” based on one or more past or current contexts indicating the user’s general interest in the automation assistant; implicit NL-based input or variations thereof may be submitted periodically, and corresponding responses may be provided automatically (or notifications of corresponding responses to be provided automatically). Note that, given the existence of new / recent search result documents over time, for example, the corresponding responses may vary over time.

[0036] Furthermore, client device 110 and / or response system 120 may include one or more memories for storing data and / or software applications, one or more processors for accessing data and executing software applications, and / or other components facilitating communication via one or more networks in network 199. In some implementations, one or more software applications may be installed locally on client device 110, while in other implementations, one or more software applications may be remotely hosted (e.g., by one or more servers) and may be accessible by client device 110 via one or more networks in network 199.

[0037] although Figure 1The aspects described herein relate to a single client device with a single user, but it should be understood that this is for illustrative purposes and is not intended to be limiting. For example, one or more additional client devices of the user and / or additional users may also implement the techniques described herein. For example, client device 110, one or more additional client devices, and / or any other computing devices of the user may form a device ecosystem that can employ the techniques described herein. These additional client devices and / or computing devices may communicate with client device 110 (e.g., via network 199). As another example, a given client device may be utilized by multiple users in a shared setting (e.g., user group, home, workplace, hotel, etc.).

[0038] Response system 120 Figure 1 The engine is shown as including a selective subquery engine 122, a subquery suggestion engine 124, a subquery generation engine 126, a subset selection engine 128, a search result engine 130, and a response engine 132. Some of these engines may be combined and / or omitted in various implementations. Furthermore, these engines may include various sub-engines. Therefore, it should be understood that... Figure 1 The various engines of the response system 120 shown are depicted to describe certain functionalities and are not intended to impose limitations.

[0039] As described in more detail herein, response system 120 may be used to: generate multiple candidate subqueries for (e.g., submitted via client device 110) multifaceted NL-based input; select a subset of candidate subqueries from the multiple candidate subqueries using one or more evaluation metrics; obtain at least one corresponding search result for each candidate subquery in the selected subset of candidate subqueries in response to the selection of the subset of candidate subqueries; generate a response to the NL-based input based on the corresponding search results for the candidate subqueries of the subset; and cause the response to be rendered (e.g., at the client device) in response to the NL-based input.

[0040] Selective subquery engine 122 can determine, for NL-based input, whether one or more of the other engines 124 to 132 should be utilized. In other words, in determining whether some or all of the other engines 124 to 132 should be utilized in processing NL-based input, the selective subquery engine can consider one or more criteria for the NL-based input, or alternatively, the search system 140 should process the NL-based input without generating, for example, candidate subqueries, without selecting a subset of those candidate subqueries, and / or without obtaining any search results for the selected subset. The criteria considered by selective subquery engine 122 may include historical data 156, such as historical data reflecting all or part of the counts and / or frequencies of previous submissions of the NL-based input. The criteria considered by selective subquery engine 122 may additionally or alternatively include criteria provided by one or more search systems based on the NL-based input, such as quality criteria reflecting the quality of search results for the NL-based input.

[0041] Subquery suggestion engine 124 can generate one or more subquery suggestions for NL-based input. In doing so, subquery suggestion engine 124 may optionally utilize sample database 156, which may include multiple samples, each comprising a corresponding multifaceted and / or noisy NL-based input and a corresponding ground truth subquery. When subquery suggestion engine 124 generates multiple suggestions for NL-based input, each of the generated suggestions may be distinct from every other suggestion in the generated suggestions. For example, each of the generated suggestions may comprise a unique set of few-sample samples relative to all other suggestions in the generated suggestions.

[0042] The subquery generation engine 126 can generate multiple candidate subqueries for NL-based input. In generating candidate subqueries, the subquery generation engine 126 can use LLM 152 to process corresponding hints generated by the subquery suggestion engine 124 to generate LLM output, and can determine candidate subqueries based on the LLM output. When multiple hints are provided for NL-based input, the subquery generation engine 126 can perform multiple iterations of processing, each iteration using different hints from the LLM and the hints, generating multiple candidate subqueries based on the LLM output at each iteration.

[0043] The subset selection engine 128 selects a subset of candidate subqueries from a plurality of candidate subqueries generated by the subquery generation engine 126. In doing so, the subset selection engine may use one or more evaluation metrics, such as relevance metrics and / or diversity metrics described herein. Optionally, in generating metrics, the subset selection engine 128 may utilize encoded neural network model 154 and / or LLM 152.

[0044] For each subquery within the subset selected by subset selection engine 128, search result engine 130 interacts with search system 140 to obtain results for the subquery. For example, search result engine 130 may obtain top-ranked results, top-ranked results, etc. N A result or any result with a quality score (and / or other score) above the threshold.

[0045] The response engine 132 uses the results obtained by the search results engine 130 to generate a response to the NL-based input.

[0046] Now go to Figure 2 It describes the implementation methods. Figure 1 Examples of how various engines can interact in generating responses 206 to natural language (NL) based input 201.

[0047] Subquery suggestion engine 124 uses input 201 and sample database 158 to generate three subquery generation suggestions 202A, 202B, and 202C. Each of subquery generation suggestions 202A, 202B, and 202C includes input 201. However, suggestion 202A includes a first few-sample instance, while suggestion 202B instead includes a completely different second few-sample instance, and suggestion 202C instead includes a completely different third few-sample instance.

[0048] The subquery generation engine 126 uses an LLM to perform three processing iterations (optionally in parallel), where each iteration processes different subquery generation hints 202A, 202B, and 202C. In the first iteration of processing hint 202A, the subquery generation engine 126 generates a first subquery 203A based on the LLM output from the first iteration. In the second iteration of processing hint 202B, the subquery generation engine 126 generates a second subquery 203B based on the LLM output from the second iteration. In the third iteration of processing hint 202C, the subquery generation engine 126 generates a third subquery 203C based on the LLM output from the third iteration.

[0049] Subset selection engine 128 selects a subset 204 of subqueries from the first subquery, the second subquery, and the third subqueries 203A-C. In selecting subset 204, subset selection engine 128 may consider relevance and / or diversity measures as described herein, and may optionally utilize embeddings generated using the encoded neural network model 154 in generating the measures. For example, subset selection engine 128 may process subqueries 203A-C one by one, and add a subquery to subset 204 if the relevance measure of the subquery meets a threshold indicating relevance and if the diversity measure for the subquery and relative to any subquery already added to subset 204 meets a threshold indicating diversity.

[0050] The search result engine 130 interacts with the search system 140 to obtain one or more corresponding results for each subquery in the subset of subqueries, and provides the common result 205 to the response engine 132.

[0051] Response engine 132 generates response 206 based on result 205.

[0052] Figure 3 A flowchart illustrating example methods for generating responses to NL-based inputs according to various implementations is provided. For convenience, the operation of method 300 is described with reference to a system performing the operation. This system of method 300 includes one or more processors, memories, and / or other components of a computing device (e.g., client device 110, response system 120, computing device 610, one or more servers, and / or other computing devices). Furthermore, although the operations of method 300 are shown in a specific order, this is not intended to be limiting. One or more operations may be reordered, omitted, and / or added.

[0053] At box 352, the system recognizes NL-based inputs, such as NL-based inputs specified based on user interface inputs at the client device.

[0054] At box 354, the system determines whether one or more criteria are met for generating and executing multiple subqueries. Such criteria may include, for example, whether the NL-based input has a threshold length, threshold complexity, is rare, has never been observed, and / or other criteria.

[0055] If the system determines at box 354 that the criteria are not met, the system proceeds to box 356 and performs a search based on the NL-based input without executing any other boxes of method 300. However, if the system determines at box 354 that the criteria are met, the system proceeds to box 358.

[0056] At box 358, the system generates a subquery generation suggestion that includes the NL-based input (e.g., all or at least part of it) and additional NL content that facilitates subquery generation. The additional NL content that facilitates subquery generation may include few-sample examples and / or wording such as “break the following into simplepertinent questions”. In some implementations, box 358 includes subbox 358A, where the system selects one or more few-sample examples from a superset of selected few-sample examples as all or part of the additional NL content. In some versions of those implementations, the selection may be random (e.g., truly random or pseudo-random). In some other versions of those versions, the selection may be guided based on, for example, a similarity measure (e.g., by embedding) between the NL-based input and the selected few-sample examples.

[0057] At box 360, the system uses LLM to process subquery generation hints to generate LLM output.

[0058] At box 362, the system generates one or more candidate subqueries based on the LLM output. For example, the system can perform one or more decodings of the LLM output to generate candidate subqueries. For example, some candidate subqueries in the candidate subqueries can be generated based on greedy decoding of the LLM output, and other candidate subqueries in the candidate subqueries can be generated based on alternative decoding of the LLM output.

[0059] At box 364, the system determines whether to generate more subquery generation hints for the NL-based input. In some implementations, this can be based on whether a threshold number of subquery generation hints have already been generated. This threshold can be a fixed value such as 3, or it can vary. The variation can be based on, for example, the length and / or complexity of the NL-based input (a higher threshold in the case of longer length and / or higher complexity) or server load (a higher threshold in the case of reduced server load). For example, the threshold could be a first fixed value when the server load is in a first range, a second fixed value when the server load is in a second range, and / or a third fixed value when the server load is in a third range. For example, the first fixed value can be less than the second fixed value, which can be less than the third fixed value, and the first range can indicate a higher load than the second range, which can indicate a higher load than the third range.

[0060] If the decision at box 364 is yes, then an additional subquery generation hint is generated at box 358, and additional candidate subqueries are generated based on that hint at boxes 360 and 362. Note that, although for convenience... Figure 3 The diagrams are drawn sequentially, but for a given NL-based input, multiple iterations of boxes 358, 360, and 362 can be performed in parallel.

[0061] If the decision at box 364 is negative, the system proceeds to box 366. At box 366, the system uses an evaluation metric to select a subset of the generated subqueries produced at the iteration at box 362. In some implementations, box 366 includes subbox 366A, where the system generates and uses a relevance metric when selecting the subset. In some implementations, box 366 includes subbox 366B, where the system additionally or alternatively generates and uses a diversity metric when selecting the subset. Figure 4 The following description and illustrations show various implementation methods. Figure 3 The flowchart for example of box 366.

[0062] At box 368, the system obtains one or more search results for a subquery of the subset selected at box 366.

[0063] At box 370, the system determines whether an unprocessed subquery exists in the subset. If yes, the system returns to box 368 and obtains the search results for that unprocessed query. If no, the system proceeds to box 372. Note that, although for convenience... Figure 3 The images are drawn sequentially, but multiple iterations of boxes 368 and 370 can be executed in parallel to retrieve search results for all subqueries targeting a subset simultaneously.

[0064] At box 372, the system generates a response based on the search results obtained through multiple iterations of box 368.

[0065] At box 374, the system causes a rendering response. For example, when NL-based input is received via a request from a client device, the system can respond to that request by causing a response to be rendered at the client device.

[0066] Go to Figure 4 This illustrates the depiction of various implementation methods. Figure 3 The flowchart for example of box 366.

[0067] At box 366A1, the system uses an encoding model to generate a representation for box 352 ( Figure 3 ) based on NL-based input embedding.

[0068] At box 366A2, the system selects box 362 ( Figure 3 Unprocessed subqueries generated during the iteration of )

[0069] At box 366A3, the system uses an encoding model to generate an embedding for the selected subquery.

[0070] At box 366A4, the system generates a relevance metric by comparing the embeddings used for subqueries with the embeddings used for NL-based inputs.

[0071] At box 366A5, the system determines whether the selected subquery is sufficiently relevant to the NL-based input based on a relevance measure. For example, the system can determine whether a diversity measure meets a threshold. If the decision at box 366A5 is that the selected subquery is irrelevant, the selected query is discarded and not added to the subset. If the decision at box 366A5 is that the selected subquery is relevant, the system proceeds to box 366B1.

[0072] At box 366B1, the system generates diversity measures for the selected subqueries. Each diversity measure is based on comparing the embedding for the selected subquery with the corresponding embedding for the corresponding subquery that has already been selected to be included in the subset (e.g., in a previous iteration at 366C1). For example, if three subqueries have already been included in the subset, three diversity measures can be generated. If the subset is empty, a default diversity measure indicating the presence of diversity can be used.

[0073] At box 366B2, the system determines whether the selected subquery is distinct relative to all other (if any) subqueries already included in the subset, based on a diversity measure. For example, the system may determine whether all diversity measures meet a threshold. If the decision at box 366A5 is that the selected subquery is not distinct, the selected subquery is discarded and not added to the subset. If the decision at box 366A5 is that the selected subquery is relevant, the system proceeds to box 366C1 and adds the selected subquery to the subset.

[0074] The system proceeds to box 366C2 and determines whether an unprocessed subquery exists. If so, the system returns to box 366A2 and selects the unprocessed subquery. If not, the system proceeds to box 366C3, and the subset selection ends.

[0075] Now go to Figure 5A , Figure 5B and Figure 5C Each figure depicts an example client device 510 rendering a graphical interface, which includes different examples of responses that can be generated according to various implementations. For example, in box 372 ( Figure 3 Generate in and based on box 374 ( Figure 3 The rendered response can optionally have Figure 5A , Figure 5B or Figure 5C The types shown in the image.

[0076] exist Figure 5A The NL-based input 501A has already been provided, and only a portion of this NL-based input is shown. It is generated according to the implementation disclosed in this document. Figure 5A The response includes a first region 502A1, which shows a first subquery of the selected subset determined according to the implementation disclosed herein, and includes two search results (A and B) for the first subquery. The first region 502A1 includes the first subquery and reflects the results (A and B) in response to the first subquery. Therefore, the viewer can easily identify both the subquery of the subset and the results associated with that subquery. The response is generated according to the implementation disclosed herein. Figure 5A The response includes a second region 502A2, which illustrates a second subquery of the selected subset determined according to the implementation disclosed herein, and includes a search result (N) for the second subquery. The second region 502A2 includes the second subquery and indicates that the result (N) responds to the second subquery. Figure 5A The vertical ellipsis in the text indicates that additional regions for appending subqueries and their corresponding results can be optionally provided.

[0077] exist Figure 5B The NL-based input 501B has already been provided, and only a portion of this NL-based input is shown. It is generated according to the implementation disclosed herein. Figure 5B The response in the document consists of only a single region, 502B, which shows multiple subqueries for a selected subset determined according to the implementation disclosed herein, and the search results for those selected subqueries. Therefore, the viewer can easily identify both the subqueries of the subset and the results associated with those subqueries. However, compared to... Figure 5A Unlike other methods, there is no indication of the relevance between the results and the corresponding queries they respond to. In other words, while users can identify subsets of subqueries and the results for those subqueries, they cannot determine which results are paired with which subqueries.

[0078] exist Figure 5C The NL-based input 501C has already been provided, and only a portion of this NL-based input is shown. It is generated according to the implementation disclosed in this document. Figure 5CThe response includes only a single region 502C, which shows multiple subqueries of a selected subset determined according to the implementation disclosed herein, and displays a generative summary generated based on the search results for those selected subqueries. For example, a generative summary can be generated by using LLM to process the content of those search results and combining it with hints for requesting summary generation. Therefore, the user can easily identify the subqueries of the subset, and establish a summary of the results associated with those subqueries. However, compared to... Figure 5B Unlike other methods, it does not describe the results separately. Instead, it provides only a combined summary of the results. Furthermore, compared to... Figure 5A Unlike other methods, there is no indication of the relevance between the results and the corresponding queries they respond to.

[0079] Turn now Figure 6 This diagram depicts a block diagram of an example computing device 610 that can be optionally utilized to perform one or more aspects of the techniques described herein. In some implementations, one or more of a client device, a cloud-based automation assistant component, or other cloud-based software application component and / or other components may include one or more components of the example computing device 610.

[0080] Computing device 610 typically includes at least one processor 614 that communicates with a plurality of peripheral devices via a bus subsystem 612. These peripheral devices may include a storage subsystem 624 (which includes, for example, a memory subsystem 625 and a file storage subsystem 626), a user interface output device 620, a user interface input device 622, and a network interface subsystem 616. The input and output devices allow user interaction with computing device 610. The network interface subsystem 616 provides an interface to an external network and is coupled to corresponding interface devices in other computing devices.

[0081] User interface input device 622 may include a keyboard, pointing device (such as a mouse, trackball, touchpad, or graphics tablet), scanner, touchscreen integrated into the display, audio input device (such as a voice recognition system, microphone), and / or other types of input device. Generally, the term "input device" is used to include all possible types of means and methods for inputting information into computing device 610 or into a communication network.

[0082] User interface output device 620 may include a display subsystem, a printer, a fax machine, or a non-visual display (such as an audio output device). The display subsystem may include a cathode ray tube (CRT), a flat panel device (such as a liquid crystal display (LCD)), a projection device, or some other mechanism for creating a visible image. The display subsystem may also provide non-visual displays, such as via an audio output device. Generally, the term "output device" is used to encompass all possible types of means and methods for outputting information from computing device 610 to a user or another machine or computing device.

[0083] Storage subsystem 624 stores the functional programming and data constructs of some or all of the modules described herein. For example, storage subsystem 624 may include selected aspects for performing the methods disclosed herein and for implementing... Figure 1 The logic of the various components described in the document.

[0084] These software modules are typically executed by processor 614 alone or in combination with other processors. The memory 625 used in storage subsystem 624 may include multiple memories, including main random access memory (RAM) 630 for storing instructions and data during program execution and read-only memory (ROM) 632 for storing fixed instructions therein. File storage subsystem 626 provides persistent storage for program and data files and may include hard disk drives, floppy disk drives with associated removable media, CD-ROM drives, optical disk drives, or removable media cartridges. Modules implementing certain functionalities of the implementation may be stored in file storage subsystem 626 within storage subsystem 624, or in other machines accessible by processor 614.

[0085] Bus subsystem 612 provides a mechanism for enabling various components and subsystems of computing device 610 to communicate with each other as intended. Although bus subsystem 612 is schematically shown as a single bus, alternative implementations of bus subsystem 612 may use multiple buses.

[0086] The computing device 610 can be of different types, including workstations, servers, computing clusters, blade servers, server farms, or any other data processing system or computing device. Due to the constantly evolving nature of computers and networks, [the following applies]. Figure 6 The description of the computing device 610 depicted herein is intended only as a specific example to illustrate some implementation methods. Many other configurations of the computing device 610 are possible, and these configurations are related to… Figure 6 The computing devices described in the document have more or fewer components compared to those in the document.

[0087] In situations where the systems described herein collect or otherwise monitor personal information about users, or may utilize personal information and / or monitored information, users may be provided with the opportunity to control whether programs or features collect user information (e.g., information about the user's social networks, social actions or activities, occupation, user preferences, or the user's current geographic location), or to control whether and / or how content more relevant to the user is received from content servers. Furthermore, certain data may be processed in one or more ways before it is stored or used, thereby removing personally identifiable information. For example, a user's identity may be processed to the point that the user's personally identifiable information cannot be determined, or, where geographic location information is available, the user's geographic location may be generalized (e.g., to the city, zip code, or state level) to the point that the user's specific geographic location cannot be determined. Therefore, users can control how information about themselves is collected and / or used.

[0088] In some implementations, a processor-implemented method is provided, comprising receiving natural language (NL)-based input associated with a client device. The method further comprises: in response to receiving the NL-based input; generating a subquery generation hint comprising the NL-based input and additional NL content facilitating subquery generation; generating a plurality of candidate subqueries for the NL-based input based on processing the subquery generation hint using a generative model; and selecting a subset of the candidate subqueries generated using the generative model from the plurality of candidate subqueries using one or more evaluation metrics. The method further comprises: in response to receiving the NL-based hint and in response to selecting a subset of candidate subqueries; obtaining at least one corresponding search result for each candidate subquery in the subset; and generating a response to the NL-based input based on the corresponding search results for the candidate subqueries in the subset. The method further comprises: causing the response to be rendered at the client device in response to the NL-based input.

[0089] These and other implementations of the techniques disclosed herein may include one or more of the following features.

[0090] In some implementations, the additional NL content of the subquery generation suggestion includes one or more minority samples, each of which includes a corresponding previously determined subquery paired with a corresponding previously determined subquery for the corresponding previously determined NL-based input. In some of those implementations, the subquery generation suggestion includes randomly selecting one or more minority samples from a superset of minority samples to include in the subquery generation suggestion.

[0091] In some implementations, the method further includes generating an additional subquery generation hint, which includes NL-based input, omitting additional NL content, and including alternative NL content that facilitates subquery generation. In some versions of those implementations, generating multiple candidate subqueries for the NL-based input includes: generating some candidate subqueries from the candidate subqueries based on processing the subquery generation hint using a generative model or an alternative generative model in a first iteration; and generating other candidate subqueries from the candidate subqueries based on processing the additional subquery generation hint using a generative model or an alternative generative model in a second iteration. In some versions of those versions, the alternative NL content of the additional subquery generation hint includes one or more alternative few-sample examples, each of which includes a corresponding alternative previous NL-based input paired with a corresponding alternative previously determined subquery for the corresponding alternative previous NL-based input. Optionally, generating the additional subquery generation hint includes randomly selecting one or more few-sample examples from a superset of few-sample examples to include in the subquery generation hint.

[0092] In some implementations, one or more evaluation metrics used in selecting a subset of candidate subqueries include a corresponding diversity metric and / or a corresponding relevance metric for each candidate subquery. In some versions of those implementations, one or more evaluation metrics used in selecting a subset of candidate subqueries include a corresponding diversity metric, and each of the corresponding diversity metrics characterizes the diversity of a candidate subquery relative to any candidate subqueries already selected for inclusion in the subset. In some implementations of those versions, the method further includes: using an encoding neural network model to generate a first encoding of a first subquery among the candidate subqueries, and using an encoding neural network model to generate a second encoding of a second subquery among the candidate subqueries. In those implementations of those versions, generating a corresponding diversity metric for the second subquery is at least partially based on a distance metric between the second encoding and the first encoding, and the encoding neural network model is computationally more efficient than a generative model. In some additional or alternative implementations of those versions, one or more evaluation metrics used in selecting a subset of candidate subqueries include a corresponding relevance metric, and each of the corresponding relevance metrics characterizes the relevance of a candidate subquery to an NL-based input. In some of those additional or alternative implementations, the method further includes: using an encoding neural network model to generate a first encoding of a first subquery among candidate subqueries; and using an encoding neural network model to generate an NL-based input encoding of the input—wherein the generation of a corresponding relevance metric for the first subquery is based at least in part on a distance metric between the first encoding and the NL-based input encoding, and wherein the encoding neural network model is computationally more efficient than a generative model. In some further additional or alternative implementations of those versions, selecting a subset of candidate subqueries includes selecting a given candidate subquery to include in the subset in response to determining that the corresponding diversity metric for the given candidate subquery satisfies a threshold, and the corresponding relevance metric for the given candidate subquery satisfies a threshold or an alternative threshold.

[0093] In some implementations, generating a response to an NL-based input based on the corresponding search results for a subset of candidate subqueries includes generating a response that includes each of the visually separate corresponding search results. In some versions of those implementations, generating a response to an NL-based input based on the corresponding search results for a subset of candidate subqueries includes generating a response that includes each of the candidate subqueries in the subset of candidate subqueries. In some versions of those versions, for each corresponding search result in the corresponding search results, the response visually indicates the relevance obtained to the corresponding candidate subquery in the subset of candidate subqueries on which the corresponding search result is based.

[0094] In some implementations, generating a response to an NL-based input based on the search results corresponding to candidate subqueries for a subset includes: using a generative model or an additional generative model to process the search results to generate a shortened summary of the search results, and including the shortened summary of the search results in the response.

[0095] In some implementations, the method further includes: in response to receiving NL-based input, determining, based on one or more criteria, whether to generate and execute multiple subqueries based on the NL-based input. In those implementations, only in response to determining that multiple subqueries should be generated and executed based on the NL-based input, the following operations are performed: generating subquery generation suggestions; generating multiple candidate subqueries; selecting a subset of candidate subqueries; obtaining corresponding search results; generating a response to the NL-based input based on the corresponding search results; and / or causing the response to be rendered at a client device in response to the NL-based input. In some implementations of those implementations, the one or more criteria include a length criterion based on the number of terms in the NL-based input, one or more search result quality criteria, and / or a current server load criterion. For example, the one or more criteria may include a length criterion that is a length threshold, and determine that the number of terms in which multiple subqueries are generated and executed is greater than the length threshold. As another example, the one or more criteria may additionally or alternatively include one or more search criteria based on the frequency of submissions to the NL-based input and / or based on one or more metrics for one or more search results in response to the overall NL-based input.

[0096] Furthermore, some implementations include one or more processors (e.g., a central processing unit (CPU), a graphics processing unit (GPU), and / or a tensor processing unit (TPU)) of one or more computing devices, wherein the one or more processors are operable to execute instructions stored in associated memory, and wherein the instructions are configured to cause any of the methods described above to be performed. Some implementations also include one or more computer-readable storage media (e.g., transient and / or non-transient) storing computer instructions executable by one or more processors to perform any of the methods described above. Some implementations also include a computer program product comprising instructions executable by one or more processors to perform any of the methods described above.

Claims

1. A method implemented by one or more processors, the method comprising: Receive natural language (NL) based input associated with the client device; In response to receiving the NL-based input: Generate subquery generation suggestions that include the NL-based input and additional NL content that facilitates subquery generation; Based on the use of a generative model to process the subquery, multiple candidate subqueries are generated for the NL-based input; Use one or more evaluation metrics to select a subset of the candidate subqueries generated using the generative model from the plurality of candidate subqueries; In response to selecting the subset of the candidate queries: For each candidate subquery in the subset, obtain at least one corresponding search result; as well as A response to the NL-based input is generated based on the corresponding search results for the candidate subqueries of the subset; as well as The response is rendered at the client device in response to the NL-based input.

2. The method as described in claim 1, wherein, The additional NL content of the subquery generation prompt includes one or more few sample instances, each of which includes a corresponding previously determined subquery paired with the corresponding previously determined subquery for the corresponding previously determined NL-based input.

3. The method as described in claim 2, wherein, Generating the subquery generation suggestion includes: randomly selecting one or more minority samples from a superset of minority samples to include in the subquery generation suggestion.

4. The method according to any of the preceding claims, further comprising: Generate additional subquery generation hints, which include the NL-based input, omit the additional NL content, and include alternative NL content that facilitates subquery generation; Generating the multiple candidate subqueries for the NL-based input includes: Based on the subquery generation hints generated in the first iteration using the generative model or an alternative generative model, some candidate subqueries are generated from the candidate subqueries. Based on the second iteration, using the generative model or the alternative generative model to process the additional subquery to generate hints, other candidate subqueries are generated from the candidate subqueries.

5. The method of claim 4, wherein, The alternative NL content of the additional subquery generation suggestion includes one or more alternative few sample instances, each of which includes a corresponding alternative previously determined NL-based input paired with the corresponding alternative previously determined subquery for the corresponding alternative previously determined NL-based input.

6. The method of claim 5, wherein, Generating the additional subquery generation suggestion includes: randomly selecting one or more of the few sample cases from the superset of few sample cases to include in the subquery generation suggestion.

7. The method as described in any of the preceding claims, wherein, The one or more evaluation metrics used in selecting the subset of candidate subqueries include a corresponding diversity metric for each candidate subquery and / or a corresponding relevance metric for each candidate subquery.

8. The method of claim 7, wherein, The one or more evaluation metrics used in selecting the subset of candidate subqueries include the corresponding diversity metrics, and wherein each of the corresponding diversity metrics characterizes the diversity of the candidate subquery relative to any candidate subquery that has been selected to be included in the subset.

9. The method of claim 8, further comprising: A coding neural network model is used to generate the first code of the first subquery in the candidate subqueries; as well as The encoding neural network model is used to generate the second encoding of the second subquery in the candidate subqueries; Specifically, the diversity metric generated for the second subquery is at least partially based on the distance metric between the second encoding and the first encoding; and The encoded neural network model is more computationally efficient than the generative model.

10. The method according to any one of claims 7 to 9, wherein, The one or more evaluation metrics used in selecting the subset of candidate subqueries include the corresponding relevance metrics, wherein each of the corresponding relevance metrics characterizes the relevance of the candidate subquery to the NL-based input.

11. The method of claim 10, further comprising: A coding neural network model is used to generate the first code of the first subquery in the candidate subqueries; as well as The NL-based input encoding is generated using the coded neural network model. Specifically, the relevance metric generated for the first subquery is at least partially based on the distance metric between the first encoding and the NL-based input encoding; and The encoded neural network model is more computationally efficient than the generative model.

12. The method according to any one of claims 7 to 11, wherein, Selecting the subset of the candidate subqueries includes: in response to determining the following, selecting a given candidate subquery to include in the subset: The diversity metric used for the given candidate subquery satisfies a threshold, and The corresponding relevance metric used for the given candidate subquery satisfies the threshold or an alternative threshold.

13. The method as described in any of the preceding claims, wherein, Generating the response to the NL-based input based on the corresponding search results for the candidate subqueries of the subset includes: The response is generated to include each of the corresponding search results, which are visually separated from each other.

14. The method of claim 13, wherein, Generating the response to the NL-based input based on the corresponding search results for the candidate subqueries of the subset includes: The response is generated to include each of the candidate subqueries in the subset.

15. The method of claim 14, wherein, For each corresponding search result in the corresponding search results, the response visually indicates the relevance to the corresponding candidate subquery in the subset on which the corresponding search result was based.

16. The method as claimed in any of the preceding claims, wherein, Generating the response to the NL-based input based on the corresponding search results for the candidate subqueries of the subset includes: The search results are processed using the generative model or an additional generative model to generate a shortened summary of the search results; and The response includes the shortened summary of the search results.

17. The method of any of the preceding claims, further comprising: In response to receiving the NL-based input: Determine whether to generate and execute multiple subqueries based on the NL-based input based on one or more criteria; Specifically, the following operations are performed only in response to determining that the plurality of subqueries are generated and executed based on the NL-based input: generating the subquery generation prompt, generating the plurality of candidate subqueries, selecting the subset of the candidate subqueries, obtaining the corresponding search result, generating the response to the NL-based input based on the corresponding search result, and / or causing the response to be rendered at the client device in response to the NL-based input.

18. The method of claim 17, wherein, The one or more criteria include a length criterion based on the number of lexical units in the NL-based input.

19. The method of claim 18, wherein, The length criterion is a length threshold, and the determination to generate and execute the plurality of subqueries is based on the fact that the number of terms is greater than the length threshold.

20. The method of any one of claims 16 to 18, wherein, The one or more standards include one or more search result quality standards.

21. The method of claim 20, wherein, The one or more search criteria are based on the frequency of submissions to the NL-based input and / or on one or more metrics used in response to the overall NL-based input.

22. The method according to any one of claims 16 to 21, wherein, The one or more criteria include the current server load metric.

23. A system comprising: One or more hardware processors; as well as A memory that stores instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform the method as described in any one of claims 1 to 22.

24. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more hardware processors, cause the one or more hardware processors to perform operations according to any one of claims 1 to 22.

Images