Dynamic determination for exchanging information via a user interface element or a dialogue operation for processing a query

Abstract

A computer-implemented method includes receiving a query from a user, dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user, providing, for presentation on a display device associated with the user, the UI element, receiving information from the user via an input to the UI element, generating a response to the query based on the information received from the user via the input to the UI element, and providing the response to the display device associated with the user.

Description

DYNAMIC DETERMINATION FOR EXCHANGING INFORMATION VIA A USER INTERFACE ELEMENT OR A DIALOGUE OPERATIONFOR PROCESSING A QUERYFIELD

[0001] The disclosure relates to computing systems and computer-implemented methods for processing a query. More particularly, the disclosure relates to computing systems and computer-implemented methods for processing the query by dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user.BACKGROUND

[0002] Recently, large language models (LLMs) have been implemented to emulate human conversations and even human reasoning through language generation. Chat hots may be used to implement human-computer interaction scenarios through dialogue. User interfaces also can be used to enable specific workflows to process a query or task.SUMMARY

[0003] Aspects and advantages of embodiments of the disclosure will be set forth in part in the following description, or can be learned from the description, or can be learned through practice of the embodiments.

[0004] An example aspect of the disclosure is directed to a computer-implemented method for processing (or responding to) a query (or task). The method includes receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

[0005] In some implementations, dynamically generating, via the machine-learned model, the UI element, comprises: selecting the UI element from a library including a plurality of UI elements based on a context of the query.

[0006] In some implementations, dynamically generating, via the machine-learned model, the UI element, comprises: dynamically generating, via the machine-learned model, code for creating the UI element.

[0007] In some implementations, the code includes code for creating the UI element using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

[0008] In some implementations, the method includes dynamically generating, via the machine-learned model, a UI screen which includes the UI element.

[0009] In some implementations, dynamically generating, via the machine-learned model, the UI screen, comprises: selecting the UI screen from a first library including a plurality of UI screens based on a context of the query, and selecting the UI element from a second library including a plurality of UI elements based on the context of the query.

[0010] In some implementations, dynamically generating, via the machine-learned model, the UI screen, comprises: dynamically generating, via the machine-learned model, code for creating the UI screen.

[0011] In some implementations, the code includes code for creating the UI screen using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

[0012] In some implementations, the method further includes, in response to receiving the query, generating a plan to process the query by dynamically determining whether to: receive information from the user for processing the query via a dialogue operation, or dynamically generate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query'.

[0013] In some implementations, generating the plan to process the query further comprises: determining a complexity value associated with processing the query; and when the complexity value exceeds a threshold complexity value, dynamically determining to dynamically generate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query'.

[0014] In some implementations, the complexity value is based on a number of expected dialogue exchanges between the user and a computing system for processing the query entirely via dialogue operations.

[0015] In some implementations, the complexity value is based on a number of expected dialogue exchanges between the user and a computing system for performing an intermediate operation via dialogue operations to respond to the query.

[0016] In some implementations, generating the plan to process the query further comprises: determining a context associated with the user; determining a suitability forreceiving at least some information for processing the query via the dialogue operation based on the context; and in response to determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than a threshold dialogue suitability level, dynamically determining to dynamically generate, via the machine- learned model, the UI element to receive the at least some of the information via the input to the UI element for processing the query.

[0017] In some implementations, the context associated with the user includes environmental information about a surroundings of the user, and when the environmental information about the surroundings of the user indicates dialogue exchanges between the user and a computing system for processing the query via dialogue operations would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than the threshold dialogue suitability level.

[0018] In some implementations, generating the plan to process the query further comprises: determining a context associated with the user; determining a suitability for receiving at least some information for processing the query via the input to the UI element based on the context; and in response to determining the suitability for receiving the at least some information for processing the query via the input to the UI element is less than a threshold UI suitability level, dynamically determining to receive the at least some of the information from the user for processing the query via the dialogue operation.

[0019] In some implementations, the context associated with the user includes environmental information about a surroundings of the user, and when the environmental information about the surroundings of the user indicates providing an input to the UI element would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the input to the UI element is less than the threshold UI suitability level.

[0020] In some implementations, the method includes implementing a further machine- learned model to generate the plan to process the query by dynamically determining whether to: receive information from the user for processing the query via the dialogue operation, or dynamically generate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query; and in response to the further machine-learned model dynamically determining to receive the information from the user for processing the query via the input to the UI element, dynamically generating, via the machine-learned model, the UI element configured to receive the information from the user.

[0021] Another example aspect of the disclosure is directed to a computing system for processing a query. The computing system includes at least one memory device configured to store instructions; and at least one processor configured to execute the instructions to perform operations, the operations comprising: receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

[0022] In some implementations, the operations further comprise: in response to receiving the query , implementing a further machine-learned model to generate a plan to process the query' by dynamically determining whether to: receive information from the user for processing the query via a dialogue operation, or dynamically generate, via the machine- learned model, the UI element to receive the information via the input to the UI element for processing the query; and in response to the further machine-learned model dynamically determining to receive the information from the user for processing the query via the input to the UI element, dynamically generating, via the machine-learned model, the UI element configured to receive the information from the user.

[0023] The at least one processor of the computing system can also be configured to execute the instructions to perform operations corresponding to any of the methods of operation, as described herein.

[0024] Other aspects of the disclosure are directed to various systems, apparatuses, non- transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer- readable medium) which stores instructions that are executable by one or more processors of a computing system or computing device is provided. In some implementations the computer-readable medium stores instructions which may include instructions to cause the one or more processors to perform one or more operations of any of the methods described herein (e g., operations of the computing system). The computer-readable medium may store additional instructions to execute other aspects of the computing system and corresponding methods of operation, as described herein.

[0025] For example, an aspect of the disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one ormore processors, cause the one or more processors to perform operations, the operations comprising: receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

[0026] These and other features, aspects, and advantages of various embodiments of the disclosure will become better understood with reference to the following description and appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate example embodiments of the disclosure and, together with the descnption, serve to explain the related principles.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Detailed discussion of embodiments directed to one of ordinary skill in the art is set forth in the specification, which makes reference to the appended drawings, in which:

[0028] FIG. 1 A depicts a block diagram of an example computing system that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0029] FIG. IB depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0030] FIG. 1C depicts a block diagram of an example computing device that performs various tasks using a machine-learned model, according to example embodiments of the disclosure.

[0031] FIG. 2A depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure.

[0032] FIG. 2B depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure.

[0033] FIG. 3A depicts an example query processor for processing a query, according to example embodiments of the disclosure.

[0034] FIG. 3B depicts an example system for dynamically generating a query plan, according to example embodiments of the disclosure.

[0035] FIG. 3C depicts an example system for dynamically generating a user interface, according to example embodiments of the disclosure.

[0036] FIGS. 4A through 4E depict example interactions for processing a query via a chat agent, according to example embodiments of the disclosure.

[0037] Reference numerals that are repeated across plural drawings are intended to identify the same features in various implementations.DETAILED DESCRIPTIONOverview

[0038] Reference now will be made to embodiments of the disclosure, one or more examples of which are illustrated in the drawings, wherein like reference characters across drawings are intended to denote like features in various implementations. Each example is provided by way of explanation of the disclosure and is not intended to limit the disclosure.

[0039] Terms used herein are used to describe the example embodiments and are not intended to limit and / or restrict the disclosure. The singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. In this disclosure, terms such as "including", "having", “comprising”, and the like are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not preclude the presence or addition of one or more of the features, elements, steps, operations, elements, components, or combinations thereof.

[0040] It will be understood that, although the terms first, second, third, etc., may be used herein to describe various elements, the elements are not limited by these terms. Instead, these terms are used to distinguish one element from another element. For example, without departing from the scope of the disclosure, a first element may be termed as a second element, and a second element may be termed as a first element.

[0041] It will be understood that when an element is referred to as being “connected” to another element, the expression encompasses an example of a direct connection or direct coupling, as well as a connection or coupling with one or more other elements interposed therebetween.

[0042] The term "and / or" includes a combination of a plurality of related listed items or any item of the plurality of related listed items. For example, the scope of the expression or phrase "A and / or B" includes the item "A", the item "B", and the combination of items "A and B”.

[0043] In addition, the scope of the expression or phrase "at least one of A or B" is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Likewise, the scope of the expression or phrase "at least one of A, B, or C" is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B, and at least one of C.

[0044] Although current conversational dialogue interfaces may be flexible or adaptable, there are many situations where a user interface screen may be more efficient than conversing back and forth with a chat bot or chat (dialogue) agent. For example, a chat agent can be used to book movie tickets for a user, however the chat agent may need to know how many people are coming, what the time constraints are, where the user would like to sit, what budget constraints the user may have, at what location the user wants to get tickets, etc. A well-designed user interface can enable the user to answer all of these questions with just a few clicks and a few button presses, while it might take several turns across minutes to elicit and provide the same information conversationally via the chat agent. Further a well- designed user interface screen can also restrict the type and range of information within allowed bounds. Additionally, scenarios may exist where it is not suitable (e.g., socially acceptable or physically convenient) to converse with a chat agent. For example, if an environment in which the user is located is noisy or if a user is walking through an office where others are busily working it may not be a good time to be typing or speaking to answer a chat agent’s questions.

[0045] According to examples of the disclosure, a computing system is configured to determine whether a query (e.g., a task) provided by a user can be processed (e.g., responded to or completed) more easily or more efficiently system by implementing a user interface (UI) screen including UI elements to receive at least some information for processing the query.

[0046] In some implementations, a machine-learned model can be trained and implemented to generate UI elements (e.g., dynamically generate UI elements, for example, in real-time) to facilitate the efficient completion of tasks where the information can more easily be provided through on-screen elements (e.g., via a combination of choice boxes, calendar widgets, number sliders, etc.).

[0047] According to examples of the disclosure, a computing system is configured to receive a request (query) from a user to perform an operation or task (e.g., book a flight,control an appliance, buy a concert ticket, etc.). The request may be received by a query agent (chat agent or chat bot) of the computing system. For example, the query agent can be configured to communicate with the user via chat through text or voice dialogue operations I exchanges.

[0048] The query agent may be configured to generate or formulate plan to accomplish the operation or task. The plan may include determining the information needed (or the information likely needed) by the query agent from the user to accomplish the operation or task. The plan may include determining potential methods for receiving the information from the user (e.g., by dialogue operations, by inputs received via a user interface, etc.). The plan may also include confirming with the user whether the user can provide the infomiation in a dialogue manner and / or via an input to a user interface which can be provided on a display device associated with the user.

[0049] As an example, in response to a query that the user would like to book a flight, the query agent may be configured to determine that it will need information including a departure airport, a destination airport, a date associated with the flight, a number of passengers, and other optional information including a preferred airline, time of day, number of stops, class of travel, whether the flight is one-way or round-trip, etc. The query agent may be configured to determine optimal methods for receiving the information from the user, for example, in the dialogue manner and / or via the input to the user interface which can be provided on a display device associated with the user.

[0050] Tn an example implementation, the query agent may be configured to determine whether some information is better collected (e.g., more efficiently collected, more safely collected, etc.) through dialogue or through a simple user interface.

[0051] In some implementations, the query agent may determine whether some information is better collected through dialogue or through a simple user interface based on an efficiency analysis. For example, the query agent may be configured to determine an expected number of dialogue exchanges between the user and the query agent for various sub-operations (e.g., for determining what airport the user would like to fly out of, what time the user wishes to depart the departure airport, etc.) as part of processing the user query and / or determine an expected number of dialogue exchanges between the user and the query agent for all of the operations needed to respond to the task (e.g., all of the operations needed to book the flight). As an example, the query agent may detemrine it is more efficient to receive various information (e.g., the preferred airline via a dialogue operation), while it is more efficient to receive various information (e.g., the departure and destination airports, thedate of the flight, etc.) through a user interface having one or more user interface elements that can be manipulated to provide the information.

[0052] In some implementations, the user interface, or a user interface element for a generic user interface that may be used for collecting the information, may not be preexisting. That is, the query agent may be configured to, in response to determining that at least some information should be collected via a user interface, dynamically generate the user interface including at least one user interface element included in the user interface or dynamically generate the at least one user interface element. In some implementations, the query agent is configured to implement a machine-learned model (e.g., a generative machine- learned model) to dynamically generate the user interface screen including the at least one user interface element included in the user interface screen or to dynamically generate the at least one user interface element.

[0053] For example, generation of the user interface (UI) screen and / or UI elements can be accomplished in a number of different ways. For example, the computing system (e.g., the machine-learned model) can be configured to select a UI element from among a library' of pre-specified UI elements, through the generation of some limited markup language, through the generation of code (e.g., CSS, HTML, and JavaScript), through arbitrary code generation of a module that can act as a plugin to the environment where the query' (chat) agent is provided, etc.

[0054] In some implementations, the computing system may be configured to provide a UI screen and / or UI element which can be used to represent or convey information that may be cumbersome or difficult to convey or describe via dialogue operations, for example, via a text-only format. For example, the UI screen may include images which represent various options (e.g., images of airline logos or images of different types of cuisines) instead of providing textual or audio dialogue descriptions of each of the options (e.g., text or audio descriptions of each of the airlines or each type of cuisine).

[0055] The disclosure provides numerous technical effects and benefits. According to examples of the disclosure, the methods and computing systems described herein enable at least some types of information to be collected and displayed more efficiently than a dialogue-only interface by which a user interacts with a query (chat) agent through various exchanges. Therefore, computing resources (e.g., processing resources, bandwidth and network resources, etc.), can be conserved or be more efficiently utilized. According to examples of the disclosure, the methods and computing systems described herein do not require a developer to foresee all possible scenarios where a UI would be useful by theimplementation of one or more machine-learned models to determine when certain information is better received or provided via a UI and the implementation of one or more machine-learned models to dynamically generate the UI and / or UI elements to collect or provide that information (e.g., without relying on pre-existing or pre-defined UIs that are previously defined and created by a developer). Therefore, the computing systems described herein are adaptable to each (real-time) situation where a user provides a query and can generate customized UIs to accurately obtain information needed for processing a query in a much timelier fashion. In addition, computing resources (e g., memory resources) can be conserved by the dynamic generation of user interface elements and user interface screens, as needed, and in real-time, rather than storing pre-generated user interface elements and user interface screens which have been designed in advance based on assumptions of the developer.

[0056] With reference now to the drawings, example embodiments of the disclosure will be discussed in further detail.

[0057] FIG. 1A depicts a block diagram of an example computing system 100 that performs various tasks using one or more machine-learned models (e.g., one or more machine-learned models to determine whether to implement a user-interface screen to provide and / or receive information associated with a query, one or more machine-learned models to dynamically generate the user interface screen and / or user interface elements for the user interface screen, etc.) according to example embodiments of the disclosure. The system 100 includes a user computing system 102, a server computing system 130, and a training computing system 150 that are communicatively coupled over a network 180.

[0058] The user computing system 102 can be any type of computing device, such as, for example, a personal computing device (e.g., laptop or desktop), a mobile computing device (e.g., smartphone or tablet), a gaming console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0059] The user computing system 102 includes one or more processors 112 and a memory 114. The one or more processors 112 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 114 can store data 116 and instructions 118 which are executed by the processor 112 to cause the user computing system 102 to perform operations.

[0060] In some implementations, the user computing system 102 can store or include one or more machine-learned models 120 (e.g., one or more machine-learned models to determine whether to implement a user-interface screen to provide and / or receive information associated with a query, one or more machine-learned models to dynamically generate the user interface screen and / or user interface elements for the user interface screen, etc.). For example, the machine-learned models 120 can be or can otherwise include various machine- learned models such as neural networks (e.g., deep neural networks) or other types of machine-learned models, including non-linear models and / or linear models. Neural networks can include feed-forward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks or other forms of neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-learned models can include multiheaded self-attention models (e.g., transformer models). For example, some example machine-learned models can include generative machine-learned models. Example machine- learned models 120 are discussed with reference to the drawings herein.

[0061] In some implementations, the one or more machine-learned models 120 can be received from the server computing system 130 over network 180, stored in the memory 114, and then used or otherwise implemented by the one or more processors 112. In some implementations, the user computing system 102 can implement multiple parallel instances of a single machine-learned model 120 (e.g., to perform parallel tasks across multiple instances of the one or more machine-learned models 120).

[0062] More particularly, the machine-learned models disclosed herein (e.g., one or more machine-learned models to determine whether to implement a user-interface screen to provide and / or receive information associated with a query, one or more machine-learned models to dynamically generate the user interface screen and / or user interface elements for the user interface screen, etc ), may be implemented to perform various tasks related to an input query that is initially provided to a chat or query agent for processing via dialogue operations between the user and the chat or query agent (e.g., via voice audio and / or via text). According to examples of the disclosure, a first machine-learned model may be configured and implemented by the computing system to determine whether some or all of the information needed to process the query would be better obtained via an input to a user interface element provided on a user interface screen. According to examples of the disclosure, a second machine-learned model may be configured and implemented by the computing system to generate the user interface screen and / or one or more user interfaceelements provided on the user interface screen, that can be used to obtain the information used for processing and responding to the query.

[0063] Machine-learned models trained according to the methods described herein may be utilized to determine or predict whether implementation of a user interface screen to obtain certain information for processing the query is more efficient, more suitable or appropriate, etc. than obtaining the information via dialogue operations. Machine-learned models trained according to the methods described herein may also be utilized to generate user interface screens and / or user interface elements for obtaining or providing information for processing the query based on various information (e.g., according to preferences of the user, according to a context of the query, according to the query itself, according to known or possible intermediate operations needed to process the query, based on characteristics of a computing device associated with the user, based on suitable methods for presenting and receiving information vi a user interface, etc.).

[0064] Additionally, or alternatively, one or more machine-learned models 140 can be included in or otherwise stored and implemented by the server computing system 130 that communicates with the user computing system 102 according to a client-server relationship. For example, the machine-learned models 140 can be implemented by the server computing system 130 as a portion of a web service (e.g., a recommendation service, a search service, an image analysis service, and the like). Thus, one or more machine-learned models 120 can be stored and implemented at the user computing system 102 and / or one or more machine- learned models 140 can be stored and implemented at the server computing system 130.

[0065] The user computing system 102 can also include one or more user input components 122 that receives user input. For example, the user input component 122 can be a touch-sensitive component (e.g., a touch-sensitive display screen or a touch pad) that is sensitive to the touch of a user input object (e.g., a finger or a stylus). The touch-sensitive component can serve to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, a mouse, or other devices by which a user can provide user input (e.g., a camera which captures an image).

[0066] The server computing system 130 includes one or more processors 132 and a memory 134. The one or more processors 132 can be any suitable processing device (e g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., andcombinations thereof. The memory 134 can store data 136 and instructions 138 which are executed by the processor 132 to cause the server computing system 130 to perform operations.

[0067] In some implementations, the server computing system 130 includes or is otherwise implemented by one or more server computing devices. In instances in which the server computing system 130 includes plural server computing devices, such server computing devices can operate according to sequential computing architectures, parallel computing architectures, or some combination thereof.

[0068] As described above, the server computing system 130 can store or otherwise include one or more machine-learned models 140. For example, the machine-learned models 140 can be or can otherwise include various machine-learned models. Example machine- learned models include neural networks or other multi-layer non-linear models. Example neural networks include feed forward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine-learned models can leverage an attention mechanism such as self-attention. For example, some example machine-1 earned models can include multi -headed self-attention models (e.g., transformer models). For example, some example machine-learned models can include generative machine-learned models. Example machine-learned models 140 are discussed herein with reference to the drawings.

[0069] The user computing system 102 and / or the server computing system 130 can train the machine-learned models 120 and / or 140 via interaction with the training computing system 150 that is communicatively coupled over the network 180. The training computing system 150 can be separate from the server computing system 130 or can be a portion of the server computing system 130.

[0070] The training computing system 150 includes one or more processors 152 and a memory 154. The one or more processors 152 can be any suitable processing device (e.g., a processor core, a microprocessor, an ASIC, an FPGA, a controller, a microcontroller, etc.) and can be one processor or a plurality of processors that are operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158 which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes or is otherwise implemented by one or more server computing devices.

[0071] The training computing system 150 can include a model trainer 160 that trains the machine-learned models 120 and / or 140 stored at the user computing system 102 and / or the server computing system 130 using various training or learning techniques, such as, for example, backwards propagation of errors. For example, a loss function can be back propagated through the model(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various loss functions can be used such as mean squared error, likelihood loss, cross entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations.

[0072] In some implementations, performing backwards propagation of errors can include performing truncated backpropagation through time. The model trainer 160 can perform a number of generalization techniques (e.g., weight decays, dropouts, etc.) to improve the generalization capability of the models being trained.

[0073] In particular, the model trainer 160 can train the machine-learned models 120 and / or 140 based on a set of training data 162. The training data 162 can include, for example, various datasets which may be stored remotely or at the training computing system 150. For example, the data may include data relating to expected dialogue interactions and possible information needed to respond to a given query. For example, the data may include data relating to user interface screens and user interface elements which are suitable for obtaining and providing information needed to respond to a given query. For example, the data may include data relating to previous user interactions via dialogue operations for processing a query. The data may include data associated with a number of dialogue exchanges for processing the query, data associated with a duration of time to perform the query, data associated with a number of dialogue exchanges for processing a portion of the query (e.g., an intermediate dialogue operation), data associated with a duration of time to perform the portion of query, etc. For example, the data may include data relating to previous user interactions via user interface operations for processing a query'. The data may include data associated with a number of inputs to the user interface screen for processing the query , data associated with a duration of time to perform the query via inputs to the user interface screen, data associated with a number of inputs to the user interface screen for processing a portion of the query' (e.g., an intermediate user interface operation), data associated with a duration of time to perform the portion of query via inputs to the user interface screen, etc. The data may include data associated with contextual information surrounding a query (e.g., a time of day, a time of year, weather information, noiseinformation, other environmental information including whether the query is associated with a home environment, an office environment, an entertainment environment (e.g., a movie theater, stadium, etc.), and the like). The training data 162 can include other data than the non-limiting example data described above.

[0074] In some implementations, if the user has provided consent, the training examples can be provided by the user computing system 102. Thus, in such implementations, the machine-learned model 120 provided to the user computing system 102 can be trained by the training computing system 150 on user-specific data received from the user computing system 102. In some instances, this process can be referred to as personalizing the model.

[0075] The model trainer 160 includes computer logic utilized to provide desired functionality. The model trainer 160 can be implemented in hardware, firmware, and / or software controlling a general purpose processor. For example, in some implementations, the model trainer 160 includes program files stored on a storage device, loaded into a memory and executed by one or more processors. In other implementations, the model trainer 160 includes one or more sets of computer-executable instructions that are stored in a tangible computer-readable storage medium such as RAM, hard disk, or optical or magnetic media.

[0076] The network 180 can be any type of communications network, including a local area network (LAN), wireless local area network (WLAN), wide area network (WAN), personal area network (PAN), virtual private network (VPN), or the like. For example, wireless communication between elements of the examples described herein may be performed via a wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi direct (WFD), ultra wideband (UWB), infrared data association (IrDA), Bluetooth low energy (BLE), near field communication (NFC), a radio frequency (RF) signal, and the like. For example, wired communication between elements of the examples described herein may be performed via a pair cable, a coaxial cable, an optical fiber cable, an Ethernet cable, and the like. Communication over the network can use a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encodings or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, secure HTTP, SSL).

[0077] The machine-learned models described in this specification may be used in a variety of tasks, applications, and / or use cases.

[0078] In some implementations, the input to the machine-learned model (s) of the disclosure can be text or natural language data (or a voice input that is then converted to text). The machine-learned model(s) can process the text or natural language data to generate an output. As an example, the machine-learned model(s) can process the text or naturallanguage data to generate a classification output. As another example, the machine-learned model(s) can process the text or natural language data to generate a semantic intent output. As another example, the machine-learned model(s) can process the text or natural language data to generate a prediction output. As another example, the machine-learned model(s) can process the text or natural language data to generate a search output. As an example, the machine-learned model(s) can be configured to classify a query as a certain type of request to perform a task (e.g., book a reservation at a restaurant, book a flight, control a home appliance remotely, control an entertainment system, etc ). The output of the machine- learned model(s) can be utilized for formulating a plan to gather and provide information for processing the query via dialogue operations and / or user interface operations.

[0079] In some implementations, the input to the machine-learned model (s) of the disclosure can be statistical data. Statistical data can be, represent, or otherwise include data computed and / or calculated from some other data source. The machine-learned model(s) can process the statistical data to generate an output. As an example, the machine-learned model(s) can process the statistical data to generate a recognition output. As another example, the machine-learned model(s) can process the statistical data to generate a prediction output. As another example, the machine-learned model(s) can process the statistical data to generate a classification output. As another example, the machine-learned model(s) can process the statistical data to generate a visualization output. As an example, statistical data relating to dialogue operations associated with various queries and statistical data relating to user interface operations associated with various queries, may be provided as input to a machine- learned model which may be configured to determine (e.g., in real-time) whether information for processing a query should be provided or received via a dialogue operation or via inputs to one or more user interface elements provided on a user interface screen, based on at least in part on the statistical data, where one or more machine-learned models can also be configured to dynamically (e.g., in real-time) generate the one or more user interface elements and / or user interface screen based on the determination.

[0080] In some implementations, the input to the machine-learned model(s) of the disclosure can be sensor data. The machine-learned model(s) can process the sensor data to generate an output. As an example, the machine-learned model(s) can process the sensor data to generate a recognition output. As another example, the machine-learned model(s) can process the sensor data to generate a prediction output. As another example, the machine- learned model(s) can process the sensor data to generate a classification output. As another example, the machine-learned model(s) can process the sensor data to generate a detectionoutput. As an example, sensor data obtained by one or more sensors (e.g., a position sensor, a microphone, image sensor, etc ), can indicate an environment of a user (e.g., whether the user is in an office environment, a noisy environment, etc.) and such data may be provided as input to a machine-learned model which may be configured to determine (e.g., in real-time) whether information for processing a query should be provided or received via a dialogue operation or via inputs to one or more user interface elements provided on a user interface screen based on at least in part on the sensor data, where one or more machine-learned models can also be configured to dynamically (e g., in real-time) generate the one or more user interface elements and / or user interface screen based on the determination.

[0081] FIG. 1A illustrates one example computing system that can be used to implement aspects of the disclosure. Other computing systems can be used as well. For example, in some implementations, the user computing system 102 can include the model trainer 160 and the training data 162. In such implementations, the machine-learned models 120 can be both trained and used locally at the user computing system 102. In some of such implementations, the user computing system 102 can implement the model trainer 160 to personalize the machine-1 earned models 120 based on user-specific data.

[0082] FIG. IB depicts a block diagram of an example computing device 10 that performs operations according to example embodiments of the disclosure. The computing device 10 can be a user computing device or a server computing device.

[0083] The computing device 10 includes a number of applications (e.g., applications 1 through N). Each application contains its own machine learning library' and machine-learned model(s). For example, each application can include a machine-learned model. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a social media application, an infotainment application, a browser application, etc.

[0084] As illustrated in FIG. IB, each application can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e g., a public API). In some implementations, the API used by each application is specific to that application.

[0085] FIG. 1C depicts a block diagram of an example computing device 50 that performs operations according to example embodiments of the disclosure. The computing device 50 can be a user computing device or a server computing device.

[0086] The computing device 50 includes a number of applications (e.g., applications 1 through N). Each application is in communication with a central intelligence layer. Example applications include a text messaging application, an email application, a dictation application, a virtual keyboard application, a social media application, an infotainment application, a browser application, etc. In some implementations, each application can communicate with the central intelligence layer (and model(s) stored therein) using an API (e.g., a common API across all applications).

[0087] The central intelligence layer includes a number of machine-learned models. For example, as illustrated in FIG. 1C, a respective machine-learned model can be provided for each application and managed by the central intelligence layer. In other implementations, two or more applications can share a single machine-learned model. For example, in some implementations, the central intelligence layer can provide a single model for all of the applications. In some implementations, the central intelligence layer is included within or otherwise implemented by an operating system of the computing device 50.

[0088] The central intelligence layer can communicate with a central device data layer. The central device data layer can be a centralized repository' of data for the computing device 50. As illustrated in FIG. 1C, the central device data layer can communicate with a number of other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e g., a private API).

[0089] FIG. 2A depicts a flow chart diagram of an example method to perform according to example embodiments of the disclosure. Although FIG. 2A depicts operations performed in a particular order for purposes of illustration and discussion, the methods of the disclosure are not limited to the particularly illustrated order or arrangement. The various operations of the method 2000 can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the disclosure.

[0090] According to some examples of the disclosure, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to process a query by dynamically determining (e g., via implementation of a first machine-learned model) whether to receive or provide some information for processing the query via an input to a user interface element of a user interface screen or via dialogue operations (e.g., through a query agent). In addition, the computing system may be configured to dynamically generate (e.g., via implementation of a second machine-learnedmodel) the user interface element and / or user interface screen when determining to receive at least some of the information via the input to a user interface element of a user interface screen.

[0091] For example, at 2100, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives a query from a user. For example, the computing system may be configured to receive the query' from the user through a query agent. For example, a query agent may refer to a chat agent or chatbot which can be implemented as a computer program or application which is configured to simulate human conversation through text or voice interactions. The query agent may be configured to utilize artificial intelligence (Al) and natural language processing (NLP) technologies to understand user inputs and generate relevant responses. For example, the query agent may be configured to automate and facilitate communication between the user and the computer system, providing information, answering questions, and performing tasks in a conversational manner.

[0092] For example, the query input by the user may be in the form of a question (e.g., “where can I get good Chinese food tonight near John’s house?”) or a command (e.g., “I want to book a flight” or “tell me when the tide is out”). For example, the query may be input by the user by voice or by text to the computing system (e.g., via user input component 122). The query agent may be configured to respond to the query by determining information needed to process the query. Processing the query may include various intermediate operations between the user and the query agent before the query is fully responded to and the task completed. For example, the query agent may ask various questions to the user (e.g., what type of Chinese food the user prefers, confirming the identity of John, confirming a time that the user wishes to eat, confirming whether the user wants to dine-in or order out, etc.). Once the query agent has sufficient information, the query agent may be configured to respond to the query (e.g., the query agent may offer a short list of Chinese restaurants that satisfy the user’s query and other conditions specified by the user explicitly, through known preferences of the user, the query agent may electronically transmit an order request to a specified restaurant, etc.).

[0093] For example, at 2200, the computing system (e g., user computing system 102, server computing system 130, training computing system 150) generates a plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of a dialogue operation or an input to a user interface (UI) element provided on a UI screen.

[0094] For example, the computing system may be configured to, in response to receiving the query , determine information that may be needed to process the query.

[0095] FIG. 3 A illustrates a query processor 310 which is configured to process a query, according to example embodiments of the disclosure. For example, the query processor 310 may include a query agent 312, query plan determiner 314, query plan generator 320, and UI generator 326. The query processor 310 may be configured to dynamically determine (e.g., via query plan determiner 314 and implementation of query plan machine-learned model 316 and / or query plan complexity determiner 318) whether to receive or provide some information for processing the query via an input to a UI element of a UI screen or via dialogue operations (e.g., through the query agent). In addition, the query processor 310 may be configured to dynamically generate (e.g., via UI generator 326 and implementation of UI generator machine-learned model 328) the UI element and / or UI screen when determining to receive at least some of the information via the input to the UI element of the UI screen. For example, the query processor 310 can be included in the computing system (e.g., user computing system 102, server computing system 130, training computing system 150).

[0096] For example, the query agent 312 may be configured to receive the query from the user through a query agent. As described above, the query agent 312 may refer to a chat agent or chatbot which can be implemented as a computer program or application which is configured to simulate human conversation through text or voice interactions.

[0097] The query plan determiner 314 may be configured to analyze the query to determine a dialogue plan for responding to the query, for example, through one or more of pre-defined rules, machine-learning techniques, and natural language processing (NLP) algorithms. The query plan determiner 314 can be configured to determine the context of the query and an intent of the user, extract key information, identify keywords, and recognize a goal of the user. For example, the query plan determiner 314 can be configured to classify the query into predefined categories or intents (e.g., through one or more machine-learned models including query plan machine-learned model 316, text classification models, etc.).

[0098] For example, the query plan determiner 314 can be configured to predict or estimate a number of dialogue exchanges between the user and the computing system to complete a task or respond to the query. The query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges based on historical information associated with the user, based on statistical information associated with a plurality of users who have made a similar query, based on known preferences of the user, based on a context of the query, etc. For example, the query plan determiner 314 can be configured to predict orestimate the number of dialogue exchanges between the user and the computing system to order food from a restaurant, to book a flight, to control a home appliance remotely, etc. In addition (or alternatively) to predicting or estimating the number of dialogue exchanges, the query plan determiner 314 can be configured to predict or estimate another metric associated with completing a task or responding to the query via dialogue operations (e.g., a duration of time, a processing burden, etc.).

[0099] For example, the query plan determiner 314 can be configured to predict or estimate a number of dialogue exchanges between the user and the computing system to obtain certain information needed to complete the task or respond to the query. The query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges based on historical information associated with the user, based on statistical information associated with a plurality of users who have provided similar information, based on known preferences of the user, based on a context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges between the user and the computing system to identify toppings for a pizza, to identify' a desired airline, to control a volume of a stereo, etc. In addition (or alternatively) to predicting or estimating the number of dialogue exchanges, the query plan determiner 314 can be configured to predict or estimate another metric associated with obtaining the certain information via dialogue operations (e.g., a duration of time, a processing burden, etc.).

[0100] Likewise, the query plan determiner 314 can be configured to predict or estimate a number of user inputs via one or more UI elements of a UI screen to complete a task or respond to the query. The query plan determiner 314 can be configured to predict or estimate the number of user inputs based on historical information associated with the user, based on statistical information associated with a plurality of users who have made a similar query, based on known preferences of the user, based on a context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of user inputs to order food from a restaurant, to book a flight, to control a home appliance remotely, etc. In addition (or alternatively) to predicting or estimating the number of user inputs, the query plan determiner 314 can be configured to predict or estimate another metric associated with completing a task or responding to the query via dialogue operations (e g., a duration of time, a processing burden, etc.).

[0101] Likewise, the query plan determiner 314 can be configured to predict or estimate a number of user inputs via one or more UI elements of a UI screen to obtain certain information needed to complete the task or respond to the query'. The query plan determiner314 can be configured to predict or estimate the number of user inputs based on historical information associated with the user, based on statistical information associated with a plurality of users who have provided similar information, based on known preferences of the user, based on a context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of user inputs to identify toppings for a pizza, to identify a desired airline, to control the volume of a stereo, etc. In addition (or alternatively) to predicting or estimating the number of user inputs, the query plan determiner 314 can be configured to predict or estimate another metric associated with obtaining the certain information via the user inputs (e.g., a duration of time, a processing burden, etc.).

[0102] Referring to FIG. 3B, an example system for dynamically generating a query plan is illustrated, according to example embodiments of the disclosure. In FIG. 3B, the query plan determiner 314 may receive as inputs one or more of user data 332, device data 334, query data 336, context data 338, query plan statistical data 342, external source data 344, feedback data 346, etc.

[0103] In some example embodiments, the user data 332 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the user data 332 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the user data 332 can include information regarding one or more user profiles, including a variety of user data such as user preference data, user demographic data, user calendar data, user social network data, user historical travel data, user historical query agent data, and the like. For example, the user data 332 can include, but is not limited to, email data including textual content, images, email-associated calendar information, or contact information; social media data including comments, reviews, check-ins, likes, invitations, contacts, or reservations; calendar application data including dates, times, events, description, or other content; virtual wallet data including purchases, electronic tickets, coupons, or deals; scheduling data; location data; SMS data; or other suitable data associated with a user account. According to one or more examples of the disclosure, the data can be analyzed to determine preferences of the user with respect to a query, for example, to automatically suggest or automatically determine a preferred restaurant, airline, navigation route, mode of navigation, etc.

[0104] The user data 332 is provided to illustrate potential data that could be analyzed, in some embodiments, by the computing system to identify user preferences, to determine optimal methods for exchanging information with the user (via dialogue operations and / or inputs to a UI screen), to generate an appropriate UI screen with appropriate UI elements, etc.However, such user data may not be collected, used, or analyzed unless the user has consented after being informed of what data is collected and how such data is used. Further, in some embodiments, the user can be provided with a tool (e.g., in a query agent application or via a user account) to revoke or modify the scope of permissions. In addition, certain information or data can be treated in one or more ways before it is stored or used, so that personally identifiable information is removed or stored in an encry pted fashion. Thus, particular user information stored as user data 332 may or may not be accessible to the computing system based on permissions given by the user, or such data may not be stored as user data 332 at all.

[0105] In some example embodiments, the device data 334 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the device data 334 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the device data 334 can include specification information about a computing device associated with the user (e.g., user computing system 102) which is used to provide inputs via dialogue operations and / or inputs to a UI screen that can be provided for presentation on a display. For example, the specification information can include display screen information associated with a display device of the computing device. The display screen information may include a resolution of the display screen, a size of the display screen, color information of the display screen, font information of the display screen, a type of the display screen, layout information of the display screen, tolerance information associated with the display screen, text placement information of the display screen, and the like. The device data 334 may also include information relating to a processing power of the computing device, information relating to storage capabilities of the computing device, compatibility information associated with the computing device, etc.

[0106] In some example embodiments, the query data 336 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the query data 336 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the query data 336 can include information about or corresponding to the query of the user For example, the query data 336 may include feature information or semantic information associated with the query. The query data 336 may include intent information, entity information (e.g., a date or time associated with a reservation query), context information (e.g., previous user inputs, the query agent’s previous responses, user preferences, time information, etc.), languageinformation, tone of the user’s query, information relating to a sentiment or emotion of the user, etc.

[0107] In some example embodiments, the context data 338 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the context data 338 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the context data 338 can include environmental information about a surroundings of the user (e.g., whether the query is associated with a home environment, an office environment, an entertainment environment (e.g., a movie theater, stadium, etc.), and the like), a time of day associated with the query, a time of year associated with the query, weather information, noise information, and the like. The context data 338 can include other contextual information including whether the user is travelling in a vehicle, indoors, outdoors, location information of the user, etc. The context data 338 can further include prior query history information, user preferences, user sentiment and emotion information, etc.

[0108] In some example embodiments, the query plan statistical data 342 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the query plan statistical data 342 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the query plan statistical data 342 can include statistical information concerning previous dialogue exchanges and / or previous inputs to user interfaces which are associated with a same or similar query as provided by the user. The statistical information concerning previous dialogue exchanges and / or previous inputs to user interfaces which are associated with a same or similar query as provided by the user can include metrics including time information associated with responding to a query or for completing a task associated with the query, information regarding the number of dialogue exchanges and / or user inputs to a UI screen for processing a query, information regarding computing resources utilized for processing a query, etc.

[0109] The statistical information concerning previous dialogue exchanges and / or previous inputs to user interfaces which are associated with a same or similar query as provided by the user can include information regarding expected inputs or responses received from a user for a particular query. For example, the statistical information may include data that reflects options that are most likely to be selected by a user for completing a task. For example, the statistical data may indicate 95% of users want to identify toppings they want for a pizza, or that a particular user is very unlikely to request a particular topping based onprior statistical data and as such that topping may not be offered as a choice on a main menu. The query plan statistical data 342 may further include data which tracks historical interactions between the query agent 312 and users that indicate preferences for dialogue operations versus UI screen inputs for exchanging certain information associated with a query . For example, the query plan statistical data 342 may indicate most users (or a particular user) prefer to provide certain information (e.g., toppings for a pizza) via a user input to a UI screen, rather than through back and forth dialogue exchanges. For example, the query plan statistical data 342 may indicate most users (or a particular user) prefer to receive certain information (e.g., a map showing locations of restaurants) via a graphic provided on a UI screen, rather than receiving an extensive amount of information in text or audio form (e.g., a list of addresses corresponding to the restaurant locations).

[0110] In some example embodiments, the external source data 344 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the external source data 344 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the external source data 344 can include any form of external content including news articles, webpages, video files, audio files, written descriptions, ratings, game content, social media content, photographs, commercial offers, transportation method, weather conditions, sensor data obtained by various sensors, or other suitable external content. External source data 344 can be searched by the computing system according to known searching methods and search results can be ranked according to relevance, popularity, or other suitable attributes, including location-specific filtering or promotion. For example, the external source data 344 can be used to predict or estimate information about expected dialogue exchanges for processing the query , about expected inputs to a UI screen for processing the query, etc.

[0111] In some example embodiments, the feedback data 346 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the feedback data 346 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the feedback data 346 can include information which is provided by one or more users relating to a query plan that is determined by the query plan determiner 314. For example, the feedback data 346 may include information indicating the user’s approval or agreement (or disapproval or disagreement) with a proposed plan for processing a query using a combination of dialogue exchanges and inputs provided to UI elements of a UI screen, a proposed plan for processing the query using only dialogue exchanges, or a proposed plan for processing the query usingonly inputs provided to UI elements of a UI screen. For example, the feedback data 346 may include information indicating the user’s approval or agreement (or disapproval or disagreement), subsequent to a query being completed or a task associated with the query being completed, with processing the query using a combination of dialogue exchanges and inputs provided to UI elements of a UI screen, with processing the query using only dialogue exchanges, or with processing the query using only inputs provided to UI elements of a UI screen. Such information may be used for further refining the operation of the query plan determiner 314, including the query plan machine-learned model 316 and query plan complexity determiner 318. The feedback data 346 may include other information including feedback concerning the quality or suitability of the responses by the computing system concerning the dialogue operations, the quality or suitability of the UI elements provided for selection by the user, the quality or suitability of the response or completion of the task associated with the query, etc.

[0112] As illustrated in FIG. 3B, the query plan determiner 314 may include the query plan machine-learned model 316 and query plan complexity determiner 318.

[0113] In some implementations, the query plan machine-learned model 316 may be configured to process input data (e.g., one or more of user data 332, device data 334, query data 336, context data 338, query plan statistical data 342, external source data 344, feedback data 346, etc.) to generate an output. For example, the output may include a classification of the query as a certain type of request to perform a task (e.g., book a reservation at a restaurant, book a flight, control a home appliance remotely, control an entertainment system, etc.). For example, the output may include a query plan which identifies expected or possible operations for processing the query. For example, the output may include a determination as to which operations would be better processed via a dialogue operation (exchange) with the user and which operations would be better processed via an input to a UI element of a UI screen.

[0114] For example, the query plan machine-learned model 316 may be configured to process input data based on statistical data to generate a prediction output. In addition, or alternatively, the query plan machine-learned model 316 may be configured to process input data based on sensor data indicating a context of the query to generate a prediction output. For example, sensor data indicating the user is in a noisy environment may cause the query plan machine-learned model 316 to output a query plan that includes at least some query operations being performed via user inputs to a UI screen. For example, sensor data indicating the user is driving may cause the query plan machine-learned model 316 to outputa query plan that includes at least some query operations being performed via dialogue exchanges unless the user is stopped and can safely provide user inputs to a UI screen. Therefore, the query plan machine-learned model 316 may be configured to dynamically change the query plan according to the state of the user.

[0115] In some implementations, the computing system (more particularly, the query plan machine-learned model 316) may be configured to generate the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen by: determining a context associated with the user and determining a suitability for receiving at least some information for processing the query via the dialogue operation based on the context. In response to the computing system (more particularly, the query plan machine-learned model 316) determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than a threshold dialogue suitability level, the computing system (more particularly, the query plan machine- learned model 316) may be configured to dynamically determine to receive the at least some of the information from the user for processing the query via the input to the UI element provided on the UI screen, and the computing system (more particularly, the UI generator 326) may be configured to dynamically generate at least one of the UI screen or the UI element to enable the user to provide at least some of the information via the input to the UI element provided on the UI screen. For example, the context associated with the user may include environmental information about a surroundings of the user (e g., environmental information indicating the user is in an office environment, an elevator, a movie theater, a concert, or some other environmental information indicating a state of the user in a particular environment). When the environmental information about the surroundings of the user indicates providing an input via dialogue operations would be inappropriate (e.g., a noise level is too high, a particular location is known to be unfit for speaking aloud, etc.), the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the dialogue operation is less than the threshold dialogue suitability level. Here, the threshold dialogue suitability level may correspond to a likelihood (probability level) that the user can successfully provide an input to a UI screen. In some implementations, particular environments may be excluded as environments for providing a dialogue operation or they may have a suitability level set to zero. As an example, when the context information indicates the user is in an elevator or at a concert with a noise levelgreater than a threshold level, the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the dialogue operation is less than the threshold dialogue suitability level. As another example, when the context information indicates the user is at home in a quiet environment, the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the dialogue operation is greater than the threshold dialogue suitability level. Therefore, the query plan machine- learned model 316 may be configured to dynamically change the query plan according to the state of the user.

[0116] In some implementations, the computing system (more particularly, the query plan machine-learned model 316) may be configured to generate the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen by: determining a context associated with the user and determining a suitability for receiving at least some information for processing the query via an input to a UI element provided on a UI screen based on the context. In response to the computing system (more particularly, the query plan machine-learned model 316) determining the suitability for receiving the at least some information for processing the query via the input to the UI element provided on the UI screen is less than a threshold UI suitability level, the computing system (more particularly, the query plan machine-learned model 31 ) may be configured to dynamically determine to receive the at least some of the information from the user for processing the query via the dialogue operation. For example, the context associated with the user may include environmental information about a surroundings of the user (e.g., environmental information indicating the user is driving a car, or environmental information indicating a state of the user in a particular environment). When the environmental information about the surroundings of the user indicates providing an input to the UI element provided on the UI screen would be inappropriate (e.g., unsafe or impractical), the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the input to the UI element provided on the UI screen is less than the threshold UI suitability level. Here, the threshold UI suitability level may correspond to a likelihood (probability level) that the user can safely or successfully provide an input to a UI screen. As an example, when the context information indicates the user is driving a vehicle at a high rateof speed or at a speed greater than some threshold level (e.g., ten mile per hour), the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the input to the UI element provided on the UI screen is less than the threshold UI suitability level. As another example, when the context information indicates the user is in a vehicle, but the vehicle is stationary, the computing system (more particularly, the query plan machine-learned model 316) may be configured to determine the suitability for receiving the at least some information for processing the query via the input to the UI element provided on the UI screen is greater than the threshold UI suitability level.Therefore, the query plan machine-learned model 316 may be configured to dynamically change the query plan according to the state of the user.

[0117] In some implementations, the query plan complexity determiner 318 may be configured to determine the plan for processing the query based on statistical information associated with the query, instead of the query plan machine-learned model 316. For example, the query plan complexity determiner 318 may be configured to determine or estimate a complexity value or metric which corresponds to an expected complexity for processing a query or a portion of a query and to determine whether to utilize a dialogue operation (exchange) or a UI screen to process the query or the portion of the query. In some implementations, the query plan complexity determiner 318 may be implemented as a classifier model which classifies an expected dialogue exchange pertaining to information that may be needed for processing the query as complex or not complex. When the expected dialogue exchange is classified as complex, the query plan determiner 314 may be configured to determine to implement a UI screen including one or more UI elements to exchange information with the user to process the query. When the expected dialogue exchange is classified as not complex, the query plan determiner 314 may be configured to determine to exchange information with the user via dialogue operations to process the query.

[0118] For example, the complexity value or metric may be based on a number of expected dialogue exchanges between the user and a computing system for processing the query entirely via dialogue operations or for performing an intermediate operation via dialogue operations to respond to the query. In some implementations, when the complexity value for processing the query exceeds a threshold complexity value (e.g., a threshold number of dialogue exchanges), the query plan complexity determiner 318 may be configured to dynamically determine to receive at least some information from the user for processing the query via an input to a UI element provided on a UI screen, and the computing system (e.g.,UI generator 326) may be configured to dynamically generate at least one of the UI screen or the UI element. In some implementations, when the complexity value for processing a portion of the query (e.g., an intermediate operation) exceeds a threshold complexity value, the query plan complexity determiner 318 may be configured to dynamically determine to receive the information associated with the portion of the query from the user via an input to a UI element provided on a UI screen, and the computing system (e.g., UI generator 326) may be configured to dynamically generate at least one of the UI screen or the UI element.

[0119] For example, the complexity value or metric may be based on a duration of expected dialogue exchanges between the user and a computing system for processing the query entirely via dialogue operations or for performing an intermediate operation via dialogue operations to respond to the query, in addition to, or alternatively to, the number of expected dialogue exchanges. In some implementations, when the complexity value for processing the query exceeds a threshold complexity value (e.g., a threshold duration of time), the query plan complexity determiner 318 may be configured to dynamically determine to receive at least some information from the user for processing the query via an input to a UI element provided on a UI screen. In some implementations, when the complexity value for processing a portion of the query exceeds a threshold complexity value (e.g., five dialogue exchanges, ten dialogue exchanges, etc.), the query plan complexity determiner 318 may be configured to dynamically determine to receive the information associated with the portion of the query from the user via an input to a UI element provided on a UI screen.

[0120] The output of the query plan machine-learned model 316 or the query plan complexity determiner 318 may be provided to the query plan generator 320 to generate one or more dialogue operations 322 for processing the query (or for processing a portion of the query ) and / or one or more UI inputs 324 for processing the query (or for processing a portion of the query).

[0121] As described herein, at operation 2200 the query plan can be generated by the query plan determiner 314 and query plan generator 320 dynamically, in real-time, responsive to a query' by the user.

[0122] For example, the query plan determiner 314 may determine that various information needs to be obtained to process the query, including a first portion of information and a second portion of infonnation.

[0123] At operation 2300 the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to determinewhether to receive the first portion of information via a UI element of a UI screen. More particularly, the query plan determiner 314 may determine whether to receive the first portion of information via a UI element of a UI screen. For example, the query plan determiner 314 may determine whether to receive the first portion of information via the UI element of the UI screen based on an output of the query plan machine-learned model 316 or the query plan complexity determiner 318, as described herein.

[0124] When the query plan determiner 314 determines to receive the first portion of information via the UI element of the UI screen, at operation 2310 the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to provide the UI screen including the UI element, and at operation 2320 the computing system may be configured to receive the first portion of information via the input to the UI element. For example, the computing system may be configured to, in response to dynamically determining to receive the first portion of the information from the user for processing the query via the input to the UI element provided on the UI screen, dynamically generate the UI element UI and / or the UI screen including the UI element. Generation of the UI element UI and / or the UI screen including the UI element will be described with respect to FIG. 3C.

[0125] Referring to FIG. 3C, an example system for dynamically generating a UI screen and UI elements is illustrated, according to example embodiments of the disclosure. In FIG. 3C, the UI generator 326 may receive as inputs one or more of user data 362, device data 364, query data 366, context data 368, UI library 372, UI code 374, external source data 376, feedback data 378, etc.

[0126] User data 362 may correspond to user data 362, device data 364 may correspond to, query data 366 may correspond to query data 336, context data 368 may correspond to context data 338, external source data 376 may correspond to external source data 344, feedback data 378 may correspond to feedback data 346, and a detailed description of each of these features will be omitted for the sake of brevity.

[0127] Briefly, user data 362 may include data that can be used to generate an appropriate UI element and / or UI screen. For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on the preferences of a user (e.g., a preference for a slider bar over selecting a button with a specific value) or traits of the user (e.g., setting a font size based on the eyesight of the user).

[0128] For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on information about a computing device associated with the user for displaying the UI screen which can be obtained from the device data 364.

[0129] For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on information included in the query data 366. For example, UI and / or UI elements may include information that ties directly to the query (e.g., including selectable UI elements that are associated with a date or time that is mentioned in the query) or that relate to the query (e g , using appropriate colors to calm a user when the query data 366 indicates the user is emotional).

[0130] For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on information included in the context data 368. For example, UI and / or UI elements may be provided based on environmental information (e.g., lighting information indicating the user views the display in a dark environment may cause the UI generator 326 to generate a UI screen that can be more easily viewed in the dark). As another example, UI and / or UI elements may be provided based on movement information associated with the user (e.g., movement information indicating the user is moving quickly may cause the UI generator 326 to generate a UI screen with fewer (and larger) UI elements can be more easily viewed and selected while the user is moving).

[0131] In some example embodiments, the UI library 372 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the UI library' 372 can be stored in a plurality of different databases accessible to the computing system (e.g., via network 180). In some examples, the UI library 372 can include a collection of pre-designed components, templates, and tools that the UI generator 326 can be configured to generate UI elements and / or UI screens for obtaining or providing information to a user for processing the query. For example, the UI library 372 can include buttons, forms, navigation menus, cards, modals, etc. For example, the UI library 372 can include a CSS framework, a HTML framework, etc. For example, the UI generator 326 can be configured to select various components from a UI library to dynamically generate a UI screen including one or more UI elements for processing the query. For example, the UI generator 326 can be configured to select one or more UI elements from the UI library 372 which includes a plurality of UI elements, based on the context of the query.

[0132] In some example embodiments, the UI code 374 can be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the UI code 374 can be stored in a plurality of different databases accessible to the computingsystem (e.g., via network 180). In some examples, the UI code 374 can include HyperText Markup Language (HTML) code which provides the structure and content of a UI screen by using markup elements that define various components of a UI, such as headings, paragraphs, lists, forms, images, links, etc. In some examples, the UI code 374 can include Cascading Style Sheets (CSS) code which can be implemented to control the presentation and styling of HTML elements, and to define how UI elements appear, including aspects like colors, fonts, spacing, positioning, and responsiveness, etc. In some examples, the UI code 374 can include JavaScript code which can be used to add animations, validate form data, fetch data from servers, and perform real-time updates to the UI. For example, the UI generator 326 can be configured to implement the UI code 374 to dynamically generate a UI screen including one or more UI elements for processing the query.

[0133] For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on information included in the external source data 376. For example, the external source data 344 can be used to populate UI elements with information obtained from websites or other electronic resources for presentation on a UI screen (e.g., pricing information, menu information, travel information, etc.), when the UI generator 326 dynamically generates a UI screen with UI elements.

[0134] For example, the UI generator 326 may be configured to generate a UI and / or UI element that is based on information included in the feedback data 378. For example, the feedback data 346 may include feedback concerning the quality or suitability of the UI elements provided for selection by the user, the quality or suitability of the UI screen, etc.

[0135] As illustrated in FIG. 3C, the UI generator 326 may include a UI generator machine-learned model 328. In some implementations, the UI generator 326 may be configured to dynamically generate (e.g., in real-time) as an output, a UI element for a UI screen and / or the UI screen, based on various inputs (e.g., one or more of user data 362, device data 364, query data 366, context data 368, UI library 372, UI code 374, external source data 376, feedback data 378, etc.).

[0136] For example, the UI generator machine-learned model 328 may be implemented by the UI generator 326 to generate one or more UI elements 384 for the UI screen 382 and / or the UI screen 382. For example, the UI generator machine-learned model 328 may be implemented as a generative machine-learned model to dynamically generate the UI screen 382 including the one or more UI elements 384 included in the UI screen 382 or to dynamically generate the one or more UI elements 384. For example, the UI generator machine-learned model 328 may be configured to dynamically generate code for creating onemore UI elements to be included in the UI screen and / or to dynamically generate code for creating the UI screen. For example, the code may be generated based on training data that includes code from UI code 374. The generated code for creating the UI screen and / or the one or more UI elements may include HyperText Markup Language, Cascading Style Sheets, JavaScript, and the like.

[0137] In some implementations, the UI generator 326 may be configured to select a UI screen from a first library which includes a plurality of UI screens, based on a context of the query , and the UI generator 326 may be configured to select one or more UI elements from a second library which includes a plurality7of UI elements, based on the context of the query.

[0138] For example, the UI generator machine-learned model 328 may be trained to generate UI elements and UI screens based on training data which may include existing UI designs including various UI elements such as buttons, text fields, icons, layouts, color schemes, and the like. For example, the UI generator machine-learned model 328 may be trained to generate UI elements and UI screens based on UI elements obtained from the UI library 372 and code provided from UI code 374. The UI generator machine-learned model 328 may include a neural network, such as a Generative Adversarial Network (GAN) or a Variational Autoencoder (VAE), which have layers of interconnected nodes that process information and learn patterns from the input data. The UI generator machine-learned model 328 may be configured to be refined over time based on an iterative process involving generating multiple designs, receiving feedback, and making adjustments.

[0139] The dynamically generated UI screen 382 and one or more UI elements 384 may be provided for presentation on a display device associated with the user for processing the query. In some implementations, the user may provide an input to the one or more UI elements 384 to obtain information for processing the query, for example, in a more efficient and / or a more convenient manner than would otherwise be obtained via dialogue operations. In some implementations, UI screen 382 including the one or more UI elements 384 may provide information to the user for processing the query, for example, in a more efficient and / or a more convenient manner than would otherwise be provided via dialogue operations. For example, the UI screen 382 may include a graphic image that can convey an extensive amount of information to the user quickly, instead of the query agent 312 providing such information through a plurality' of dialogue operations that may be difficult for a user to remember, may be cumbersome to read in text form, or may take a longer time.

[0140] Referring back to FIG. 2A, when the query plan determiner 314 determines to not receive the first portion of information via the UI element of the UI screen, at operation 2330the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to receive the first portion of information via the dialogue operation. For example, the computing system may be configured to, in response to dynamically determining to receive the first portion of the information from the user for processing the query via the dialogue operation, provide an output (e.g., via text or audio) requesting the first portion of information and receive the first portion of the information from the user via a text input or a voice input.

[0141] At operation 2400 the computing system (e g., user computing system 102, server computing system 130, training computing system 150) may be configured to determine whether to receive the second portion of infomiation via a dialogue operation (exchange). More particularly, the query plan determiner 314 may determine whether to receive the second portion of information via the dialogue operation. For example, the query plan determiner 314 may determine whether to receive the second portion of information via the dialogue operation based on an output of the query plan machine-learned model 316 or the query plan complexity determiner 318, as described herein.

[0142] When the query plan determiner 314 determines to receive the second portion of information via the dialogue operation, at operation 2410 the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to receive the second portion of information via the dialogue operation. For example, the computing system may be configured to, in response to dynamically determining to receive the second portion of the information from the user for processing the query via the dialogue operation, provide an output (e.g., via text or audio) requesting the second portion of information and receive the second portion of the information from the user via a text input or a voice input.

[0143] When the query plan determiner 314 determines to not receive the second portion of information via the dialogue operation, at operation 2420 the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to provide the UI screen including the UI element, and at operation 2430 the computing system may be configured to receive the second portion of information via the input to the UI element. For example, the computing sy stem may be configured to, in response to dynamically determining to receive the second portion of the information from the user for processing the query via the input to the UI element provided on the UI screen, dynamically generate the UI element UI and / or the UI screen including the UI element.Generation of a UI element for inclusion in a UI screen and / or the UI screen including the UI element is described with respect to FIG. 3C.

[0144] At operation 2500, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to process the query based on the first portion of the information and the second portion of the information (whether received via a dialogue operation or an input to a UI element of a UI screen).

[0145] FIG. 2B depicts a flow chart diagram of another example method to perform according to example embodiments of the disclosure. Although FIG. 2B depicts operations performed in a particular order for purposes of illustration and discussion, the methods of the disclosure are not limited to the particularly illustrated order or arrangement. The various operations of the method 2000’ can be omitted, rearranged, combined, and / or adapted in various ways without deviating from the scope of the disclosure.

[0146] According to some examples of the disclosure, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) may be configured to process a query by dynamically generating (e.g., via implementation of a machine-learned model) a user interface (UI) element configured to receive information from the user for processing the query.

[0147] For example, similar to operation 2100 of FIG. 2 A, at 2600 in FIG. 2B the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives a query from a user. For example, the computing system may be configured to receive the query from the user through a query agent. For example, a query agent may refer to a chat agent or chatbot which can be implemented as a computer program or application which is configured to simulate human conversation through text or voice interactions. The query agent may be configured to utilize artificial intelligence (Al) and natural language processing (NLP) technologies to understand user inputs and generate relevant responses. For example, the query agent may be configured to automate and facilitate communication between the user and the computer system, providing information, answering questions, and performing tasks in a conversational manner.

[0148] For example, the query input by the user may be in the form of a question (e.g., “where can I get good Chinese food tonight near John’s house?”) or a command (e.g., “I want to book a flight” or “tell me when the tide is out”). For example, the query may be input by the user by voice or by text to the computing system (e.g., via user input component 122). The query agent may be configured to respond to the query by determining informationneeded to process the query. Processing the query' may include various intermediate operations between the user and the query agent before the query is fully responded to and the task completed.

[0149] For example, at 2650, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) dynamically generates, via a machine-learned model, a user interface (UI) element configured to receive information from the user. FIG. 3C illustrates an example system for dynamically generating UI elements. For example, the example system of FIG. 3C may be implemented to dynamically generate, via UI generator machine-learned model 328, the UI element which is configured to receive information from the user. Detailed operations of the example system of FIG. 3C for generating UI elements and UI screens including the UI elements are described herein and will not be repeated for the sake of brevity .

[0150] For example, at 2700, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) provides, for presentation on a display device associated with the user, the UI element. For example, the one or more UI elements 384 may be provided for presentation on a display device associated with the user for processing the query.

[0151] For example, at 2750, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives information from the user via an input to the UI element. For example, the user may provide an input to the one or more UI elements 384 to obtain information for processing the query, for example, in a more efficient and / or a more convenient manner than would otherwise be obtained via dialogue operations. For example, the user may select (e.g., via a touch input) a checkbox if the UI element is configured as a selectable checkbox, the user may manipulate a scroll bar if the UI element is configured as a scroll bar, etc. The UI element may be configured as an interactive or selectable UI element for the user to easily provide an input that provides information to the computing system for processing the query.

[0152] For example, at 2800, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) generates a response to the query based on the information received from the user via the input to the UI element. For example, the computing system may generate any kind of output for responding to the query as determined, for example, via the query processor 310, query agent 312, query plan determiner 314, etc. For example, if a user selects a UI element that corresponds to a restaurant at a particular location, the computing system may be configured to generate aresponse which includes directions or an address to that location of the restaurant, or to dynamically generate further UI elements for continuing processing of the query to receive further information (e.g., selection of food items, selection of a vendor to deliver the food to the user, etc.). For example, if a user selects a UI element that corresponds to a restaurant at a particular location, the computing system may be configured to generate a response which includes a menu for the restaurant with selectable UI elements. In some implementations, the generated response may be a dialogue operation. For example, the dialogue operation may confirm a previous selection, confirm task completion, ask for further information needed to process the query, etc. The response may be any kind of output that is used to execute, complete, or further the processing of the query.

[0153] For example, at 2850, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) provides the response to the display device associated with the user. For example, the computing system may provide the response via another UI element (included in a UI screen) that is provided for presentation on the display device, via a dialogue operation, or via another method (e.g., haptic feedback, audio feedback, etc.).

[0154] FIGS. 4A through 4E depict example interactions for processing a query via a chat agent, according to example embodiments of the disclosure. In FIG. 4A, interactions between a chat agent (query agent) and a user are depicted on a display device associated with the user. In the first screenshot 4100, the user inputs a query' 4130 via a dialogue operation to order a pizza. The chat agent responds with another dialogue operation 4120 requesting certain information from the user (e.g., which restaurant the user wants to order the pizza from). As described herein, the dialogue operation corresponds to a text input or text output, or a voice input through a microphone (e.g., by selecting the UI element 4110 depicted as a microphone to enable the microphone) or an audio output which can be output through a speaker of the computing device associated with the user.

[0155] In FIG 4B, further interactions between a chat agent (query agent) and a user are depicted on the display device associated with the user. In the second screenshot 4200, the chat agent outputs a dialogue operation 4210 indicating three restaurants that match the user’s preferred restaurant choice. Here, in accordance with example embodiments of the disclosure, the query plan determiner 314 may be configured to determine to provide and receive information relating to the three restaurants via a plurality of UI elements of a UI screen, rather than through a dialogue operation. For example, the UI generator 326 may beconfigured to implement the UI generator machine-learned model 328 to dynamically generate (e.g., in real-time) an interactive map 4220 which depicts the locations of the three restaurants relative to the user, where the user can select the preferred restaurant location by a touch input 4230 to a UI element 4240 corresponding to the restaurant.

[0156] Tn FIG 4C, further interactions between a chat agent (query agent) and a user are depicted on the display device associated with the user. In the third screenshot 4300, the chat agent outputs a dialogue operation 4310 requesting that the user provide additional information to complete the task associated with the query. Here, in accordance with example embodiments of the disclosure, the query plan determiner 314 may be configured to determine to provide and receive information relating to the query (e.g., the pizza order) via a plurality of UI elements of a UI screen, rather than through a dialogue operation. For example, the UI generator 326 may be configured to implement the UI generator machine- learned model 328 to dynamically generate (e.g., in real-time) a UI screen 4320 including a plurality of UI elements including a crust option 4330 where the user can select the preferred crust by a touch input 4340 to a UI element 4350 which is a selectable checkbox. Other UI elements may include a checkmark 4360 to indicate that the user has completed the selections and an X 4370 to indicate a cancellation of a selection.

[0157] In FIG. 4D, further interactions between a chat agent (query agent) and a user are depicted on the display device associated with the user. In the fourth screenshot 4400, the chat agent outputs a dialogue operation 4410 requesting that the user provide additional information to complete the task associated with the query. Here, in accordance with example embodiments of the disclosure, the query plan determiner 314 may be configured to determine to provide and receive information relating to the query (e.g., the pizza order) via a plurality of UI elements of a UI screen, rather than through a dialogue operation. For example, the UI generator 326 may be configured to implement the UI generator machine- learned model 328 to dynamically generate (e.g., in real-time) a UI screen 4420 including a plurality of UI elements including a time option 4430 where the user can select a time for picking up the pizza by a touch input 4440 to a UI element which corresponds to a selectable scrollbar (for selecting the hour of delivery). In some implementations, the UI generator machine-learned model 328 may be configured to limit or bound selectable values according to a context of the query or user preferences (e.g., limiting values associated with the time for picking up the pizza, omitting toppings as being selectable which the user is allergic to, etc.). In some implementations, the UI generator machine-learned model 328 may be configured topreselect selectable values according to a context of the query (e.g., pre-selecting the “PM” value based on a current time of the query or pre-selecting toppings known to be a user’s favorite), s associated with the time for picking up the pizza).

[0158] In FIG. 4E, further interactions between a chat agent (query' agent) and a user are depicted on the display device associated with the user. In the fifth screenshot 4500, the chat agent outputs a dialogue operation 4510 requesting whether the user is ready to submit their order. Here, in accordance with example embodiments of the disclosure, the query plan determiner 314 may be configured to determine to receive information responsive to the chat agent’s query via the dialogue operation. The chat agent may output another dialogue operation indicating completion of the processing of the query.

[0159] The example interaction for processing a query via a chat agent depicted in FIGS. 4A through 4E is merely one example, and the disclosure is not limited to this example. Aspects of the disclosure can be applied in various other contexts, including travel reservations, restaurant reservations, operating entertainment systems, operating home appliances, operating heating and cooling sy stems, navigation systems, and any other environment in which a chat agent may be employed or implemented to assist a user with a task.Additional Disclosure

[0160] Another example aspect of the disclosure is directed to a computer-implemented method for processing (or responding to) a query (or task). The method includes receiving a query from a user; generating a plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of a dialogue operation or an input to a user interface (UI) element provided on a UI screen; when determining to receive a first portion of the information from the user via the UI element: providing, for presentation on a display device associated with the user, the UI screen including the UI element, and receiving the first portion of the information for processing the query via the input to the UI element; when determining to receive a second portion of the information from the user via the dialogue operation: receiving the second portion of the information for processing the query via the dialogue operation; and processing the query to respond to the query based on the first portion of the information received from the user via the input to the UI element provided on the UI screen and the second portion of the information received from the user via the dialogue operation.

[0161] In some implementations, the method includes, in response to dynamically determining to receive the first portion of the information from the user for processing the query via the input to the UI element provided on the UI screen, dynamically generating the UI element.

[0162] In some implementations, dynamically generating the UI element comprises: selecting the UI element from a library' including a plurality of UI elements based on a context of the query.

[0163] In some implementations, dynamically generating the UI element comprises: dynamically generating, via a machine-learned model, code for creating the UI element to be included in the UI screen.

[0164] In some implementations, the code includes code for creating the UI element using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

[0165] In some implementations, the method includes, in response to dynamically determining to receive the first portion of the information from the user for processing the query via the input to the UI element provided on the UI screen, dynamically generating the UI screen and the UI element.

[0166] In some implementations, dynamically generating the UI screen comprises: selecting the UI screen from a first library including a plurality of UI screens based on a context of the query, and selecting the UI element from a second library including a plurality of UI elements based on the context of the query.

[0167] In some implementations, dynamically generating the UI screen and UI element comprises: dynamically generating, via a machine-learned model, code for creating the UI screen and the UI element to be included in the UI screen.

[0168] In some implementations, the code for creating the UI screen and the UI element to be included in the UI screen includes code for creating the UI screen and the UI element to be included in the UI screen using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

[0169] In some implementations, generating the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen comprises: determining a complexity value associated with processing the query; when the complexity value exceeds a threshold complexity value, dynamically determining to receive the first portion of the information from the user for processing the query via theinput to the UI element provided on the UI screen; and dynamically generating at least one of the UI screen or the UI element.

[0170] In some implementations, the complexity value is based on a number of expected dialogue exchanges between the user and a computing system for processing the query entirely via dialogue operations.

[0171] In some implementations, the complexity value is based on anumber of expected dialogue exchanges between the user and a computing system for performing an intermediate operation via dialogue operations to respond to the query.

[0172] In some implementations, generating the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen comprises: determining a context associated with the user; determining a suitability for receiving at least some information for processing the query via the dialogue operation based on the context; in response to determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than a threshold dialogue suitability level, dynamically determining to receive the at least some of the information from the user for processing the query via the input to the UI element provided on the UI screen; and dynamically generating at least one of the UI screen or the UI element to enable the user to provide at least some of the information via the input to the UI element provided on the UI screen.

[0173] In some implementations, the context associated with the user includes environmental information about a surroundings of the user, and when the environmental information about the surroundings of the user indicates dialogue exchanges between the user and a computing system for processing the query via dialogue operations would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than the threshold dialogue suitability level.

[0174] In some implementations, generating the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen comprises: determining a context associated with the user; determining a suitability for receiving at least some information for processing the query via the input to the UI element provided on the UI screen based on the context; and in response to determining the suitability for receiving the at least some information for processing the query via the input tothe UI element provided on the UI screen is less than a threshold UI suitability level, dynamically determining to receive the at least some of the information from the user for processing the query via the dialogue operation.

[0175] In some implementations, the context associated with the user includes environmental information about a surroundings of the user, and when the environmental information about the surroundings of the user indicates providing an input to the UI element provided on the UI screen would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the input to the UI element provided on the UI screen is less than the threshold UI suitability level.

[0176] In some implementations, the method includes implementing a first machine- learned model to generate the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen; and in response to the first machine-learned model dynamically determining to receive at least some of the information from the user for processing the query via the input to the UI element provided on the UI screen, implementing a second machine-learned model to dynamically generate at least one of the UI screen or the UI element.

[0177] Another example aspect of the disclosure is directed to a computing system for processing a query. The computing system includes at least one memory device configured to store instructions; and at least one processor configured to execute the instructions to perform operations, the operations comprising: receiving a query from a user; generating a plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of a dialogue operation or an input to a user interface (UI) element provided on a UI screen; when determining to receive a first portion of the information from the user via the UI element: providing, for presentation on a display device associated with the user, the UI screen including the UI element, and receiving the first portion of the information for processing the query via the input to the UI element; when determining to receive a second portion of the information from the user via the dialogue operation: receiving the second portion of the information for processing the query via the dialogue operation; and processing the query to respond to the query based on the first portion of the information received from the user via the input to the UI element provided on the UI screen and the second portion of the infomration received from the user via the dialogue operation.

[0178] In some implementations, the operations further comprise implementing a first machine-learned model to generate the plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of the dialogue operation or the input to the UI element provided on the UI screen; and in response to the first machine-learned model dynamically determining to receive at least some of the information from the user for processing the query via the input to the UI element provided on the UI screen, implementing a second machine-learned model to dynamically generate at least one of the UI screen or the UI element.

[0179] The at least one processor of the computing system can also be configured to execute the instructions to perform operations corresponding to any of the methods of operation, as described herein.

[0180] Other aspects of the disclosure are directed to various systems, apparatuses, non- transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer- readable medium) which stores instructions that are executable by one or more processors of a computing system or computing device is provided. In some implementations the computer-readable medium stores instructions which may include instructions to cause the one or more processors to perform one or more operations of any of the methods described herein (e.g., operations of the computing system). The computer-readable medium may store additional instructions to execute other aspects of the computing system and corresponding methods of operation, as described herein.

[0181] For example, an aspect of the disclosure is directed to one or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving a query from a user; generating a plan to process the query by dynamically determining whether to receive information from the user for processing the query via at least one of a dialogue operation or an input to a user interface (UI) element provided on a UI screen; when determining to receive a first portion of the information from the user via the UI element: providing, for presentation on a display device associated with the user, the UI screen including the UI element, and receiving the first portion of the information for processing the query via the input to the UI element; when determining to receive a second portion of the information from the user via the dialogue operation: receiving the second portion of the information for processing the query via the dialogue operation; and processing the query to respond to the query based on the first portion of theinformation received from the user via the input to the UI element provided on the UI screen and the second portion of the information received from the user via the dialogue operation.

[0182] The technology discussed herein makes reference to servers, databases, software applications, and other computer-based systems, as well as actions taken and information sent to and from such systems. The inherent flexibility of computer-based systems allows for a great variety of possible configurations, combinations, and divisions of tasks and functionality between and among components. For instance, processes discussed herein can be implemented using a single device or component or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0183] Aspects of the above-described example embodiments may be recorded in non-transitory computer-readable media including program instructions to implement various operations embodied by a computer. The media may also include, alone or in combination with the program instructions, data files, data structures, and the like. Examples of non- transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tape; optical media such as CD ROM disks, Blue-Ray disks, and DVDs; magneto-optical media such as optical discs; and other hardware devices that are specially configured to store and perform program instructions, such as semiconductor memory, readonly memory (ROM), random access memory (RAM), flash memory, USB memory, and the like. Examples of program instructions include both machine code, such as produced by a compiler, and files containing higher level code that may be executed by the computer using an interpreter. The program instructions may be executed by one or more processors. The described hardware devices may be configured to act as one or more software modules in order to perform the operations of the above-described embodiments, or vice versa. In addition, a non-transitory computer-readable storage medium may be distributed among computer systems connected through a network and computer-readable codes or program instructions may be stored and executed in a decentralized manner. In addition, the non- transitory computer-readable storage media may also be embodied in at least one application specific integrated circuit (ASIC) or Field Programmable Gate Array (FPGA).

[0184] Each block of the flowchart illustrations may represent a unit, module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks may occur out of order. Forexample, two blocks shown in succession may in fact be executed substantially concurrently (simultaneously) or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved.

[0185] While the subject matter has been described in detail with respect to various example embodiments thereof, each example is provided by way of explanation, not limitation of the disclosure. Those skilled in the art, upon attaining an understanding of the foregoing, can readily produce alterations to, variations of, and equivalents to such embodiments. Accordingly, the subject disclosure does not preclude inclusion of such modifications, variations and / or additions to the subject matter as would be readily apparent to one of ordinary skill in the art. For instance, features illustrated or described as part of one embodiment can be used with another embodiment to yield a still further embodiment. Thus, it is intended that the disclosure covers such alterations, variations, and equivalents.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method, comprising: receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

2. The computer-implemented method of claim 1, wherein dynamically generating, via the machine-learned model, the UI element, comprises: selecting the UI element from a library including a plurality of UI elements based on a context of the query.

3. The computer-implemented method of claim 1, wherein dynamically generating, via the machine-learned model, the UI element, comprises: dynamically generating, via the machine-learned model, code for creating the UI element.

4. The computer-implemented method of claim 3, wherein the code includes code for creating the UI element using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

5. The computer-implemented method of claim 1, further comprising dynamically generating, via the machine-learned model, a UI screen which includes the UI element.

6. The computer-implemented method of claim 5, wherein dynamically generating, via the machine-learned model, the UI screen, comprises: selecting the UI screen from a first library including a plurality of UI screens based on a context of the query, andselecting the UI element from a second library including a plurality of UI elements based on the context of the query.

7. The computer-implemented method of claim 5, dynamically generating, via the machine-learned model, the UI screen, comprises: dynamically generating, via the machine-learned model, code for creating the UI screen.

8. The computer-implemented method of claim 7, wherein the code includes code for creating the UI screen using at least one of HyperText Markup Language, Cascading Style Sheets, or JavaScript.

9. The computer-implemented method of claim 1 , further comprising: in response to receiving the query, generating a plan to process the query by dynamically determining whether to: receive information from the user for processing the query via a dialogue operation, or dynamically generate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query'.

10. The computer-implemented method of claim 9, wherein generating the plan to process the query' further comprises: determining a complexity value associated with processing the query; and when the complexity value exceeds a threshold complexity value, dynamically determining to dynamically generate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query.

11. The computer-implemented method of claim 10, wherein the complexity value is based on a number of expected dialogue exchanges between the user and a computing system for processing the query entirely via dialogue operations.

12. The computer-implemented method of claim 10, wherein the complexity value is based on a number of expected dialogue exchanges between the user and a computing system for performing an intermediate operation via dialogue operations to respond to the query.

13. The computer-implemented method of claim 9, wherein generating the plan to process the query further comprises: determining a context associated with the user; determining a suitability for receiving at least some information for processing the query via the dialogue operation based on the context; and in response to determining the suitability' for receiving the at least some information for processing the query via the dialogue operation is less than a threshold dialogue suitability level, dynamically determining to dynamically generate, via the machine-learned model, the UI element to receive the at least some of the information via the input to the UI element for processing the query.

14. The computer-implemented method of claim 13, wherein the context associated with the user includes environmental information about a surroundings of the user, and when the environmental information about the surroundings of the user indicates dialogue exchanges between the user and a computing system for processing the query via dialogue operations would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the dialogue operation is less than the threshold dialogue suitability level.

15. The computer-implemented method of claim 9, wherein generating the plan to process the query further comprises: determining a context associated with the user; determining a suitability for receiving at least some information for processing the query via the input to the UI element based on the context; and in response to determining the suitability' for receiving the at least some information for processing the query via the input to the UI element is less than a threshold UI suitability level, dynamically determining to receive the at least some of the information from the user for processing the query via the dialogue operation.

16. The computer-implemented method of claim 15, wherein the context associated with the user includes environmental information about a surroundings of the user, andwhen the environmental information about the surroundings of the user indicates providing an input to the UI element would be inappropriate, determining the suitability for receiving the at least some information for processing the query via the input to the UI element is less than the threshold UI suitability level.

17. The computer-implemented method of claim 9, further comprising: implementing a further machine-learned model to generate the plan to process the query by dynamically determining whether to: receive information from the user for processing the query via the dialogue operation, or dynamically generate, via the machine- learned model, the UI element to receive the infonnation via the input to the UI element for processing the query; and in response to the further machine-learned model dynamically determining to receive the information from the user for processing the query' via the input to the UI element, dynamically generating, via the machine-learned model, the UI element configured to receive the information from the user.

18. A computing system, comprising: at least one memory device configured to store instructions; and at least one processor configured to execute the instructions to perform operations, the operations comprising: receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

19. The computing system of claim 18, wherein the operations further comprise: in response to receiving the query, implementing a further machine-learned model to generate a plan to process the query by dynamically determining whether to: receive information from the user for processing the query via a dialogue operation, or dynamicallygenerate, via the machine-learned model, the UI element to receive the information via the input to the UI element for processing the query; and in response to the further machine-learned model dynamically determining to receive the information from the user for processing the query via the input to the UI element, dynamically generating, via the machine-learned model, the UI element configured to receive the information from the user.

20. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations comprising: receiving a query from a user; dynamically generating, via a machine-learned model, a user interface (UI) element configured to receive information from the user; providing, for presentation on a display device associated with the user, the UI element; receiving information from the user via an input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

Images