Dynamic determination of exchanging information to process queries via user interface elements or dialog operations

Abstract

A computer-implemented method includes receiving a query from a user; dynamically generating a user interface (UI) element via a machine learning model, the UI element configured to receive information from the user; providing the UI element for presentation on a display device associated with the user; receiving information from the user via an input of the UI element; generating a response to the query based on information received from the user via an input of the UI element; and providing the response to a display device associated with the user.

Description

Technical Field

[0001] This disclosure relates to computing systems and computer-implemented methods for processing queries. More specifically, this disclosure relates to computing systems and computer-implemented methods for processing queries by dynamically generating user interface (UI) elements via machine learning models, the UI elements being configured to receive information from a user. Background Technology

[0002] Recently, large language models (LLMs) have been implemented to simulate human conversation and even human reasoning through language generation. Chatbots can be used to realize human-computer interaction scenarios through dialogue. User interfaces can also be used to enable specific workflows to handle queries or tasks. Summary of the Invention

[0003] Various aspects and advantages of embodiments of this disclosure will be set forth in part in the description which follows, or may be learned from the description, or may be learned by practice of the embodiments.

[0004] An exemplary aspect of this disclosure relates to a computer-implemented method for processing (or responding to) a query (task). The method includes: receiving a query from a user; dynamically generating a user interface (UI) element via a machine learning model, the UI element being configured to receive information from the user; providing the UI element for presentation on a display device associated with the user; receiving information from the user via input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

[0005] In some implementations, dynamically generating the UI element via the machine learning model involves selecting the UI element from a library containing multiple UI elements based on the context of the query.

[0006] In some implementations, dynamically generating the UI element via the machine learning model includes dynamically generating code for creating the UI element via the machine learning model.

[0007] In some implementations, the code includes code for creating the UI element using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

[0008] In some implementations, the method includes dynamically generating a UI screen that includes the UI element via the machine learning model.

[0009] In some implementations, dynamically generating the UI screen via the machine learning model includes: selecting the UI screen from a first library containing multiple UI screens based on the context of the query, and selecting the UI element from a second library containing multiple UI elements based on the context of the query.

[0010] In some implementations, dynamically generating the UI screen via the machine learning model includes dynamically generating code for creating the UI screen via the machine learning model.

[0011] In some implementations, the code includes code for creating the UI screen using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

[0012] In some implementations, the method further includes, in response to receiving the query, generating a plan for processing the query by dynamically determining whether to receive information from the user via a dialog operation to process the query, or to dynamically generate the UI element via the machine learning model to receive information via input to the UI element to process the query.

[0013] In some implementations, generating a plan to process the query further includes: determining a complexity value associated with processing the query; and when the complexity value exceeds a threshold complexity value, dynamically determining to dynamically generate the UI element via the machine learning model to receive input information from the UI element for processing the query.

[0014] In some implementations, this complexity value is based on the expected number of dialogue exchanges between the user and the computing system to process the query entirely via dialogue.

[0015] In some implementations, this complexity value is based on the expected number of dialogue exchanges between the user and the computing system to perform intermediate operations via dialogue in response to the query.

[0016] In some implementations, generating a plan to process the query further includes: determining a context associated with the user; determining, based on the context, the suitability of receiving at least some information via the dialogue operation to process the query; and, in response to determining that the suitability of receiving at least some information via the dialogue operation to process the query is below a threshold dialogue suitability level, dynamically determining to dynamically generate the UI element via the machine learning model to receive at least some of the information via input to the UI element to process the query.

[0017] In some implementations, the context associated with the user includes environmental information about the user's surrounding environment, and when the environmental information about the user's surrounding environment indicates that a dialogue exchange between the user and the computing system for processing the query via a dialogue operation would be inappropriate, it is determined that the suitability of receiving at least some information via the dialogue operation for processing the query is below a threshold dialogue suitability level.

[0018] In some implementations, generating a plan to process the query further includes: determining a context associated with the user; determining, based on the context, the suitability of receiving at least some information via input to a UI element to process the query; and, in response to determining that the suitability of receiving at least some information via input to a UI element to process the query is below a threshold UI suitability level, dynamically determining at least some of the information received from the user via the dialog operation to process the query.

[0019] In some implementations, the context associated with the user includes environmental information about the user's surrounding environment, and when the environmental information about the user's surrounding environment indicates that providing input to the UI element would be inappropriate, it is determined that the suitability of receiving at least some information via input to the UI element for processing the query is below a threshold UI suitability level.

[0020] In some implementations, the method includes: implementing a further machine learning model to generate a plan for processing the query by dynamically determining whether to receive information from the user via the dialog operation to process the query, or to dynamically generate the UI element via the machine learning model to receive information via input to the UI element to process the query; and in response to the further machine learning model dynamically determining that information is received from the user via input to the UI element to process the query, dynamically generating the UI element via the machine learning model, the UI element being configured to receive the information from the user.

[0021] Another example aspect of this disclosure relates to a computing system for processing queries. The computing system includes at least one memory device configured to store instructions; and at least one processor configured to execute the instructions to perform operations including: receiving a query from a user; dynamically generating a user interface (UI) element via a machine learning model, the UI element being configured to receive information from the user; providing the UI element for rendering on a display device associated with the user; receiving information from the user via input to the UI element; generating a response to the query based on the information received from the user via the input to the UI element; and providing the response to the display device associated with the user.

[0022] In some implementations, the operation further includes: in response to receiving the query, implementing a further machine learning model to generate a plan for processing the query by dynamically determining whether to receive information from the user via the dialog operation to process the query, or to dynamically generate the UI element via the machine learning model to receive information via input to the UI element to process the query; and in response to the further machine learning model dynamically determining to receive information from the user via input to the UI element to process the query, dynamically generating the UI element via the machine learning model, the UI element being configured to receive the information from the user.

[0023] At least one processor of the computing system may also be configured to execute the instructions to perform an operation corresponding to any of the methods described herein.

[0024] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions executable by one or more processors of a computing system or computing device. In some implementations, the computer-readable medium stores instructions that may include instructions to cause one or more processors to perform one or more operations (e.g., operations of a computing system) of any of the methods described herein. The computer-readable medium may store additional instructions to perform other aspects of the computing system and corresponding methods of operation, as described herein.

[0025] For example, one aspect of this disclosure relates to one or more non-transitory computer-readable media that commonly stores instructions that, when executed by one or more processors, cause the processors to perform operations including: receiving a query from a user; dynamically generating a user interface (UI) element via a machine learning model, the UI element being configured to receive information from the user; providing the UI element for presentation on a display device associated with the user; receiving information from the user via input to the UI element; generating a response to the query based on the information received from the user via input to the UI element; and providing the response to the display device associated with the user.

[0026] These and other features, aspects, and advantages of the various embodiments of this disclosure will be better understood with reference to the following description and the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate exemplary embodiments of the disclosure and, together with the description, serve to explain the relevant principles. Attached Figure Description

[0027] Referring to the accompanying drawings, a detailed discussion of embodiments is set forth in this specification for those skilled in the art, in which:

[0028] Figure 1A A block diagram of an example computing system that uses a machine learning model to perform various tasks according to an example embodiment of the present disclosure is depicted.

[0029] Figure 1B A block diagram of an example computing device that performs various tasks using a machine learning model according to an example embodiment of the present disclosure is depicted.

[0030] Figure 1C A block diagram of an example computing device that performs various tasks using a machine learning model according to an example embodiment of the present disclosure is depicted.

[0031] Figure 2A A flowchart depicts an example method performed according to an example embodiment of the present disclosure.

[0032] Figure 2B A flowchart depicts an example method performed according to an example embodiment of the present disclosure.

[0033] Figure 3A An example query processor for processing queries is described according to an example embodiment of the present disclosure.

[0034] Figure 3B An example system for dynamically generating query plans according to an example embodiment of the present disclosure is described.

[0035] Figure 3C An example system for dynamically generating a user interface according to an example embodiment of the present disclosure is described.

[0036] Figures 4A to 4E An example interaction for processing a query via a chat agent, according to an example embodiment of the present disclosure, is depicted.

[0037] The repeated reference numerals across multiple figures are intended to identify the same features in various implementations. Detailed Implementation

[0038] Overview

[0039] Reference will now be made to embodiments of the present disclosure, one or more examples of which are illustrated in the accompanying drawings, wherein the same reference numerals across the drawings are intended to denote the same features in various implementations. Each example is provided by way of interpretation of the present disclosure and is not intended to limit the present disclosure.

[0040] The terminology used herein is for describing exemplary embodiments and is not intended to limit and / or constrain this disclosure. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” are intended to include the plural forms as well. In this disclosure, terms such as “comprising,” “having,” and “including” are used to specify features, numbers, steps, operations, elements, components, or combinations thereof, but do not preclude the presence or addition of one or more features, elements, steps, operations, components, or combinations thereof.

[0041] It will be understood that although the terms first, second, third, etc., may be used herein to describe various elements, these elements are not limited by these terms. Rather, these terms are used to distinguish one element from another. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and a second element may be referred to as a first element.

[0042] It will be understood that when an element is referred to as being “connected” to another element, the expression covers examples of direct connection or direct coupling, as well as connections or couplings with one or more other elements inserted between them.

[0043] The term “and / or” includes a combination of multiple related listed items or any one of multiple related listed items. For example, the scope of the expression or phrase “A and / or B” includes item “A”, item “B”, and a combination of items “A and B”.

[0044] Furthermore, the scope of the expression or phrase “at least one of A or B” is intended to include all of the following: (1) at least one of A, (2) at least one of B, and (3) at least one of A and at least one of B. Similarly, the scope of the expression or phrase “at least one of A, B or C” is intended to include all of the following: (1) at least one of A, (2) at least one of B, (3) at least one of C, (4) at least one of A and at least one of B, (5) at least one of A and at least one of C, (6) at least one of B and at least one of C, and (7) at least one of A, at least one of B and at least one of C.

[0045] While current conversational interfaces may be flexible or adaptable, in many situations, a user interface screen can be more efficient than engaging in back-and-forth conversations with a chatbot or chat (conversational) agent. For example, a chatbot might be used to book movie tickets for a user; however, it might need to know how many people will be attending, time constraints, where the user wants to sit, potential budget constraints, and where the user wants to pick up their tickets. A well-designed user interface allows the user to answer all these questions with just a few clicks and button presses, whereas obtaining and providing the same information conversationally via a chatbot could require several back-and-forth interactions over several minutes. Furthermore, a well-designed user interface screen can limit the type and scope of information to what is permissible. Additionally, there may be scenarios where engaging in conversation with a chatbot is inappropriate (e.g., socially unacceptable or physically inconvenient). For example, if the user is in a noisy environment or is moving around in an office where others are busy working, it might not be a good time to type or speak in response to the chatbot's questions.

[0046] According to examples of this disclosure, a computing system is configured to determine whether a query (e.g., a task) provided by a user can be processed (e.g., responded to or completed) more simply or efficiently by implementing a user interface (UI) screen (including UI elements) to receive at least some information for processing the query.

[0047] In some implementations, machine learning models can be trained and implemented to generate UI elements (e.g., dynamically generated UI elements, such as in real time) to facilitate efficient task completion, where information can be more easily provided through on-screen elements (e.g., via combinations of checkboxes, calendar widgets, number sliders, etc.).

[0048] According to examples in this disclosure, a computing system is configured to receive requests (queries) from a user to perform an operation or task (e.g., booking a flight, controlling an appliance, purchasing concert tickets, etc.). This request may be received by a query agent of the computing system (a chatbot or chatbot). For example, the query agent may be configured to communicate with the user via chat through text or voice dialogue / exchange.

[0049] The query agent can be configured to generate or formulate a plan to complete an operation or task. This plan may include determining information (or potentially needed information) from the user that the query agent needs to complete the operation or task. The plan may include identifying potential methods for receiving information from the user (e.g., through conversational interaction, through input received via a user interface, etc.). The plan may also include confirming with the user whether the user can provide information conversationally and / or via input to a user interface, which may be provided on a display device associated with the user.

[0050] As an example, in response to a user's query to book a flight, the query agent can be configured to determine that it will need information including the departure airport, destination airport, the date associated with the flight, the number of passengers, and other optional information including preferred airline, date and time, number of stops, class of service, whether the flight is one-way or round-trip, etc. The query agent can be configured to determine the best method for receiving information from the user, for example, conversationally and / or via input to a user interface, which can be provided on a display device associated with the user.

[0051] In the example implementation, the query agent can be configured to determine whether certain information is better collected through dialogue or through a simple user interface (e.g., more efficient collection, more secure collection, etc.).

[0052] In some implementations, the query agent can determine, based on efficiency analysis, whether certain information is better collected through dialogue or a simple user interface. For example, the query agent can be configured to determine, as part of processing a user query, the expected number of dialogue exchanges between the user and the query agent for various sub-operations (e.g., determining which airport the user wishes to depart from, the desired departure time from the departure airport, etc.), and / or the expected number of dialogue exchanges between the user and the query agent for all operations required to respond to the task (e.g., all operations required to book a flight). As an example, the query agent can determine that receiving various information (e.g., receiving preferred airlines via dialogue) is more efficient than receiving various information (e.g., departure and destination airports, flight dates, etc.) through a user interface with one or more user interface elements that can be manipulated to provide information.

[0053] In some implementations, the user interface, or user interface elements of a generic user interface that can be used to collect information, may not exist in advance. That is, the query agent can be configured to dynamically generate, or dynamically generate, the user interface including at least one user interface element, in response to determining that at least some information should be collected via the user interface. In some implementations, the query agent is configured to implement a machine learning model (e.g., a generative machine learning model) to dynamically generate, or dynamically generate, a user interface screen including at least one user interface element, or dynamically generate the at least one user interface element.

[0054] For example, the generation of user interface (UI) screens and / or UI elements can be achieved in a variety of different ways. For example, a computing system (e.g., a machine learning model) can be configured to select UI elements from a pre-specified library of UI elements through the generation of some limited markup language, through the generation of code (e.g., CSS, HTML, and JavaScript), or through the arbitrary code generation of a module that can be used as a plugin for an environment that provides a query (chat) agent.

[0055] In some implementations, the computing system may be configured to provide UI screens and / or UI elements that can be used to represent or convey information that may be inconvenient or difficult to convey or describe via dialogue (e.g., via plain text format). For example, a UI screen may include images representing various options (e.g., an image of an airline logo or an image of different types of dishes) instead of providing a text or audio dialogue description of each of the options (e.g., a text or audio description of each airline or each type of dish).

[0056] This disclosure provides several technical effects and benefits. According to examples of this disclosure, the methods and computational systems described herein enable the collection and display of at least some types of information more efficiently than through conversational interfaces used by users interacting with query (chat) agents through various exchanges. Therefore, computational resources (e.g., processing resources, bandwidth, and network resources) can be utilized more efficiently. According to examples of this disclosure, by implementing one or more machine learning models to determine when certain information is better received or provided via a UI, and by dynamically generating UI and / or UI elements to collect or provide that information (e.g., without relying on a pre-existing or predefined UI previously defined and created by the developer), the methods and computational systems described herein do not require the developer to anticipate all possible scenarios in which the UI will be useful. Therefore, the computational systems described herein are adaptable to each (real-time) situation in which the user provides a query, and can generate customized UIs to obtain the information needed to process the query more accurately and in a more timely manner. Furthermore, computational resources (e.g., memory resources) can be saved by dynamically generating user interface elements and user interface screens as needed and in real time, rather than storing pre-generated user interface elements and user interface screens pre-designed based on developer assumptions.

[0057] Exemplary embodiments of this disclosure will now be discussed in further detail with reference to the accompanying drawings.

[0058] Figure 1A A block diagram of an example computing system 100 according to an exemplary embodiment of the present disclosure is depicted, which uses one or more machine learning models (e.g., one or more machine learning models for determining whether to implement a user interface screen to provide and / or receive information associated with a query, one or more machine learning models for dynamically generating user interface screens and / or user interface elements of user interface screens, etc.) to perform various tasks. System 100 includes a user computing system 102, a server computing system 130, and a training computing system 150 communicatively coupled via a network 180.

[0059] User computing system 102 can be any type of computing device, such as, for example, a personal computing device (e.g., a laptop computer or desktop computer), a mobile computing device (e.g., a smartphone or tablet computer), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0060] User computing system 102 includes one or more processors 112 and memory 114. The one or more processors 112 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 114 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 114 can store data 116 and instructions 118 that are executed by processor 112 to cause user computing system 102 to perform operations.

[0061] In some implementations, the user computing system 102 may store or include one or more machine learning models 120 (e.g., one or more machine learning models for determining whether to implement a user interface screen to provide and / or receive information associated with a query, one or more machine learning models for dynamically generating the user interface screen and / or user interface elements of the user interface screen, etc.). For example, machine learning model 120 may be, or may otherwise include, various machine learning models, such as neural networks (e.g., deep neural networks) or other types of machine learning models, including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models). For example, some machine learning model examples may include generative machine learning models. Example machine learning models 120 are discussed with reference to the accompanying drawings.

[0062] In some implementations, one or more machine learning models 120 may be received from server computing system 130 via network 180, stored in memory 114, and then used or otherwise implemented by one or more processors 112. In some implementations, user computing system 102 may implement multiple parallel instances of a single machine learning model 120 (e.g., to perform parallel tasks across multiple instances of one or more machine learning models 120).

[0063] More specifically, the machine learning models disclosed herein (e.g., one or more machine learning models for determining whether to implement a user interface screen to provide and / or receive information associated with a query, one or more machine learning models for dynamically generating a user interface screen and / or user interface elements of the user interface screen, etc.) can be implemented to perform various tasks related to an input query that is initially provided to a chat or query agent for processing via a dialogue operation between a user and a chat or query agent (e.g., via voice audio and / or via text). According to examples of this disclosure, a first machine learning model can be configured and implemented by a computing system to determine whether some or all of the information needed to process the query will be better obtained via input to user interface elements provided on a user interface screen. According to examples of this disclosure, a second machine learning model can be configured and implemented by a computing system to generate a user interface screen and / or one or more user interface elements provided on a user interface screen, which can be used to obtain information for processing and responding to queries.

[0064] The machine learning model trained according to the method described herein can be used to determine or predict whether implementing a user interface screen to obtain certain information for processing a query is more efficient, appropriate, or suitable than obtaining information via dialogue. The machine learning model trained according to the method described herein can also be used to generate user interface screens and / or user interface elements to obtain or provide information for processing queries based on various factors, such as user preferences, the context of the query, the query itself, known or possible intermediate operations required to process the query, the characteristics of the computing device associated with the user, and the appropriate method of presenting and receiving information via the user interface.

[0065] Additionally or alternatively, one or more machine learning models 140 may be included in or otherwise stored and implemented by server computing system 130, which communicates with user computing system 102 according to a client-server relationship. For example, machine learning model 140 may be implemented by server computing system 130 as part of a web service (e.g., recommendation service, search service, image analysis service, etc.). Thus, one or more machine learning models 120 may be stored and implemented at user computing system 102, and / or one or more machine learning models 140 may be stored and implemented at server computing system 130.

[0066] User computing system 102 may also include one or more user input components 122 that receive user input. For example, user input component 122 may be a touch-sensitive component (e.g., a touch-sensitive display or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may be used to implement a virtual keyboard. Other example user input components include a microphone, a traditional keyboard, a mouse, or other means by which the user provides input (e.g., a camera that captures images).

[0067] Server computing system 130 includes one or more processors 132 and memory 134. The one or more processors 132 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. Memory 134 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. Memory 134 can store data 136 and instructions 138, which are executed by processor 132 to cause server computing system 130 to perform operations.

[0068] In some implementations, the server computing system 130 includes one or more server computing devices or is otherwise implemented by such one or more server computing devices. Where the server computing system 130 includes multiple server computing devices, these server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0069] As described above, server computing system 130 may store or otherwise include one or more machine learning models 140. For example, machine learning model 140 may be, or may otherwise include, various machine learning models. Example machine learning models include neural networks or other multi-layered nonlinear models. Example neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Some example machine learning models may utilize attention mechanisms, such as self-attention. For example, some example machine learning models may include multi-head self-attention models (e.g., transformer models). For example, some machine learning model examples may include generative machine learning models. Example machine learning model 140 is discussed herein with reference to the accompanying drawings.

[0070] User computing system 102 and / or server computing system 130 can train machine learning models 120 and / or 140 via interaction with training computing system 150, which is communicatively coupled through network 180. Training computing system 150 may be separate from server computing system 130 or may be part of server computing system 130.

[0071] The training computing system 150 includes one or more processors 152 and memory 154. The one or more processors 152 can be any suitable processing device (e.g., processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and can be a single processor or multiple processors operatively connected. The memory 154 can include one or more non-transitory computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, disks, etc., and combinations thereof. The memory 154 can store data 156 and instructions 158, which are executed by the processor 152 to cause the training computing system 150 to perform operations. In some implementations, the training computing system 150 includes one or more server computing devices or is otherwise implemented by such one or more server computing devices.

[0072] The training computation system 150 may include a model trainer 160 that uses various training or learning techniques, such as, for example, backpropagation of error, to train machine learning models 120 and / or 140 stored at the user computation system 102 and / or the server computation system 130. For example, one or more parameters of the model can be updated (e.g., gradients based on the loss function) via model backpropagation loss functions. Various loss functions can be used, such as mean squared error, likelihood loss, cross-entropy loss, hinge loss, and / or various other loss functions. Gradient descent techniques can be used to iteratively update parameters across multiple training iterations.

[0073] In some implementations, backpropagation of execution error may include backpropagation of execution truncation over time. The model trainer 160 may perform various generalization techniques (e.g., weight decay, dropout, etc.) to improve the generalization ability of the model being trained.

[0074] Specifically, model trainer 160 can train machine learning models 120 and / or 140 based on a set of training data 162. Training data 162 can include various datasets, such as those that can be stored remotely or at the training computing system 150. For example, the data can include data related to the expected dialogue interaction and possible information needed to respond to a given query. For example, the data can include data related to user interface screens and user interface elements adapted to obtain and provide information needed to respond to a given query. For example, the data can include data related to previous user interactions via dialogue actions for processing the query. The data can include data associated with the number of dialogue exchanges for processing the query, data associated with the duration of query execution, data associated with the number of dialogue exchanges for processing a portion of the query (e.g., intermediate dialogue actions), data associated with the duration of that portion of the query execution, etc. For example, the data can include data related to previous user interactions via user interface actions for processing the query. The data may include data associated with the number of inputs to the user interface screen used to process the query, data associated with the duration of executing the query via input to the user interface screen, data associated with the number of inputs to the user interface screen used to process a portion of the query (e.g., intermediate user interface operations), and data associated with the duration of that portion of the query executed via input to the user interface screen, etc. The data may include data associated with contextual information surrounding the query (e.g., current date, current year, weather information, noise information, other environmental information, including whether the query is associated with a home environment, office environment, entertainment environment (e.g., cinema, stadium, etc.)). The training data 162 may include other data besides the non-limiting example data described above.

[0075] In some implementations, training examples can be provided by the user computing system 102 if the user has provided consent. Therefore, in such implementations, the machine learning model 120 provided to the user computing system 102 can be trained by the training computing system 150 on user-specific data received from the user computing system 102. In some cases, this process may be referred to as model personalization.

[0076] Model trainer 160 includes computer logic for providing desired functionality. Model trainer 160 can be implemented using hardware, firmware, and / or software that controls a general-purpose processor. For example, in some implementations, model trainer 160 includes a program file stored on a storage device, loaded into memory, and executed by one or more processors. In other implementations, model trainer 160 includes one or more sets of computer-executable instructions stored in a tangible computer-readable storage medium (such as RAM, hard disk, or optical or magnetic media).

[0077] Network 180 can be any type of communication network, including Local Area Network (LAN), Wireless Local Area Network (WLAN), Wide Area Network (WAN), Personal Area Network (PAN), Virtual Private Network (VPN), etc. For example, wireless communication between the components of the examples described herein can be performed via wireless LAN, Wi-Fi, Bluetooth, ZigBee, Wi-Fi Direct (WFD), Ultra Wideband (UWB), Infrared Data Association (IrDA), Bluetooth Low Energy (BLE), Near Field Communication (NFC), radio frequency (RF) signals, etc. For example, wired communication between the components of the examples described herein can be performed via two-strand cable, coaxial cable, fiber optic cable, Ethernet cable, etc. Communication over a network can use a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formats (e.g., HTML, XML), and / or protection schemes (e.g., VPN, Secure HTTP, SSL).

[0078] The machine learning models described in this specification can be used for a variety of tasks, applications, and / or use cases.

[0079] In some implementations, the input to the machine learning model of this disclosure can be text or natural language data (or voice input that is then converted to text). The machine learning model can process the text or natural language data to generate output. As an example, the machine learning model can process text or natural language data to generate classification output. As another example, the machine learning model can process text or natural language data to generate semantic intent output. As another example, the machine learning model can process text or natural language data to generate predictive output. As another example, the machine learning model can process text or natural language data to generate search output. As an example, the machine learning model can be configured to classify queries into a certain type of request for performing a task (e.g., reserving a seat at a restaurant, booking a flight, remotely controlling a home appliance, controlling an entertainment system, etc.). The output of the machine learning model can be used to formulate a plan for collecting and providing information via dialogue and / or user interface operations to process queries.

[0080] In some implementations, the input to the machine learning model of this disclosure may be statistical data. Statistical data may be, represent, or otherwise include data calculated and / or computed from another data source. The machine learning model may process the statistical data to generate output. As an example, the machine learning model may process the statistical data to generate an identification output. As another example, the machine learning model may process the statistical data to generate a prediction output. As another example, the machine learning model may process the statistical data to generate a classification output. As another example, the machine learning model may process the statistical data to generate a visualization output. As an example, statistical data related to dialogic actions associated with various queries and statistical data related to user interface actions associated with various queries may be provided as input to a machine learning model, which may be configured to determine, at least in part, based on the statistical data (e.g., in real time), whether information should be provided or received for processing the query via dialogic actions or via input to one or more user interface elements provided on a user interface screen, wherein one or more machine learning models may also be configured to dynamically (e.g., in real time) generate one or more user interface elements and / or user interface screens based on this determination.

[0081] In some implementations, the input to the machine learning model of this disclosure can be sensor data. The machine learning model can process the sensor data to generate an output. As an example, the machine learning model can process the sensor data to generate an identification output. As another example, the machine learning model can process the sensor data to generate a prediction output. As another example, the machine learning model can process the sensor data to generate a classification output. As another example, the machine learning model can process the sensor data to generate a detection output. As an example, sensor data obtained by one or more sensors (e.g., a position sensor, a microphone, an image sensor, etc.) can indicate the user's environment (e.g., whether the user is in an office environment, a noisy environment, etc.), and such data can be provided as input to a machine learning model that can be configured to determine, at least in part, based on the sensor data (e.g., in real time), whether information should be provided or received for processing a query via dialogue or via input to one or more user interface elements provided on a user interface screen, wherein one or more machine learning models can also be configured to dynamically (e.g., in real time) generate one or more user interface elements and / or user interface screens based on this determination.

[0082] Figure 1AAn example computing system that can be used to implement various aspects of this disclosure is shown. Other computing systems may also be used. For example, in some implementations, user computing system 102 may include model trainer 160 and training data 162. In such implementations, machine learning model 120 may be trained locally at user computing system 102 and both may be used. In some implementations of this type, user computing system 102 may implement model trainer 160 to personalize machine learning model 120 based on user-specific data.

[0083] Figure 1B A block diagram is depicted illustrating an example computing device 10 performing operations according to an exemplary embodiment of the present disclosure. The computing device 10 may be a user computing device or a server computing device.

[0084] The computing device 10 includes multiple applications (e.g., application 1 to application N). Each application contains its own machine learning library and machine learning model. For example, each application may include a machine learning model. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, social media applications, infotainment applications, browser applications, etc.

[0085] like Figure 1B As shown, each application can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, each application can communicate with each device component using an API (e.g., a public API). In some implementations, the API used by each application is application-specific.

[0086] Figure 1C A block diagram depicts an example computing device 50 performing operations according to an exemplary embodiment of the present disclosure. The computing device 50 may be a user computing device or a server computing device.

[0087] The computing device 50 includes multiple applications (e.g., application 1 to application N). Each application communicates with a central intelligence layer. Example applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, social media applications, infotainment applications, browser applications, etc. In some implementations, each application may use an API (e.g., a common API across all applications) to communicate with the central intelligence layer (and the models stored therein).

[0088] The central intelligence layer comprises multiple machine learning models. For example, such as... Figure 1CAs shown, a corresponding machine learning model can be provided for each application, and the corresponding machine learning model can be managed by a central intelligent layer. In other implementations, two or more applications can share a single machine learning model. For example, in some implementations, the central intelligent layer can provide a single model for all applications. In some implementations, the central intelligent layer is included within the operating system of the computing device 50 or otherwise implemented by the operating system.

[0089] The central intelligence layer can communicate with the central device data layer. The central device data layer can be a centralized data repository for computing device 50. For example... Figure 1C As shown, the central device data layer can communicate with multiple other components of the computing device, such as, for example, one or more sensors, a context manager, a device state component, and / or additional components. In some implementations, the central device data layer can communicate with each device component using an API (e.g., a private API).

[0090] Figure 2A A flowchart illustrating an example method performed according to an example embodiment of this disclosure is provided. Although Figure 2A For illustrative and discussion purposes, operations performed in a particular order are depicted; however, the methods of this disclosure are not limited to the order or arrangement specifically shown. Various operations of method 2000 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0091] According to some examples of this disclosure, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to process a query by dynamically determining, (e.g., via an implementation of a first machine learning model), whether the query is processed via input to user interface elements of a user interface screen or via a dialog operation (e.g., via a query agent) receiving or providing some information for processing the query. Additionally, the computing system can be configured to dynamically generate user interface elements and / or user interface screens (e.g., via an implementation of a second machine learning model) when it determines that at least some of the information received via input to user interface elements of a user interface screen has been processed.

[0092] For example, at 2100, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives a query from a user. For example, the computing system may be configured to receive queries from the user via a query agent. For example, a query agent may refer to a chatbot or chatbot, which may be implemented as a computer program or application configured to simulate human conversation through text or voice interaction. The query agent may be configured to utilize artificial intelligence (AI) and natural language processing (NLP) technologies to understand user input and generate relevant responses. For example, the query agent may be configured to automate and facilitate communication between the user and the computer system, thereby providing information, answering questions, and performing tasks in a conversational manner.

[0093] For example, a user's query input can be a question (e.g., "Where can I get good Chinese food tonight near John's house?") or a command (e.g., "I want to book a flight" or "Tell me when the tide is out"). For example, a user can input a query into the computing system via voice or text (e.g., via user input component 122). The query agent can be configured to respond to a query by determining the information needed to process it. Query processing can include various intermediate operations between the user and the query agent before fully responding to the query and completing the task. For example, the query agent can ask the user various questions (e.g., what type of Chinese food the user prefers, confirming John's identity, confirming the user's preferred dining time, confirming whether the user wants to dine in or take away, etc.). Once the query agent has enough information, it can be configured to respond to the query (e.g., the query agent can provide a short list of Chinese restaurants that meet the user's query and other conditions explicitly specified by the user based on the user's known preferences; the query agent can transmit an order request electronically to the specified restaurant, etc.).

[0094] For example, at 2200, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) generates a plan for processing the query by dynamically determining whether to receive information from the user for processing the query via at least one of dialogue operation or input to UI elements provided on the user interface (UI) screen.

[0095] For example, a computing system can be configured to determine, in response to receiving the query, the information that may be needed to process the query.

[0096] Figure 3A A query processor 310 according to an example embodiment of the present disclosure is illustrated, the query processor being configured to process queries. For example, the query processor 310 may include a query agent 312, a query plan determiner 314, a query plan generator 320, and a UI generator 326. The query processor 310 may be configured (e.g., via implementations of the query plan determiner 314, the query plan machine learning model 316, and / or the query plan complexity determiner 318) to dynamically determine whether information for processing the query is received or provided via input to UI elements on a UI screen or via a dialogical action (e.g., through the query agent). Additionally, the query processor 310 may be configured to dynamically generate UI elements and / or UI screens (e.g., via implementations of the UI generator 326 and the UI generator machine learning model 328) when it is determined that at least some of the information received via input to UI elements on a UI screen has been received. For example, the query processor 310 may be included in a computing system (e.g., user computing system 102, server computing system 130, training computing system 150).

[0097] For example, query agent 312 can be configured to receive queries from a user. As described above, query agent 312 can refer to a chatbot or chatbot, which can be implemented as a computer program or application configured to simulate human conversation through text or voice interaction.

[0098] Query plan determiner 314 can be configured to analyze the query, for example, through one or more of predefined rules, machine learning techniques, and natural language processing (NLP) algorithms, to determine a dialogue plan for responding to the query. Query plan determiner 314 can be configured to determine the context of the query and the user's intent, extract key information, identify keywords, and discern the user's goal. For example, query plan determiner 314 can be configured to (e.g., through one or more machine learning models including query plan machine learning model 316, text classification model, etc.) categorize the query into predefined categories or intents.

[0099] For example, the query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges between a user and the computing system to complete a task or respond to a query. The query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges based on historical information associated with the user, statistical information associated with multiple users who have performed similar queries, the user's known preferences, the context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges between a user and the computing system to order food from a restaurant, book a flight, remotely control a home appliance, etc. As a supplement (or alternative) to predicting or estimating the number of dialogue exchanges, the query plan determiner 314 can be configured to predict or estimate another metric associated with completing a task or responding to a query via dialogue (e.g., duration, processing load, etc.).

[0100] For example, the query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges between a user and a computing system to obtain certain information needed to complete a task or respond to a query. The query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges based on historical information associated with the user, statistical information associated with multiple users who have provided similar information, the user's known preferences, the context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of dialogue exchanges between a user and a computing system to identify pizza toppings, identify a desired airline, control the volume of stereo speakers, etc. As a supplement (or alternative) to predicting or estimating the number of dialogue exchanges, the query plan determiner 314 can be configured to predict or estimate another metric (e.g., duration, processing load, etc.) associated with obtaining certain information via dialogue operations.

[0101] Similarly, the query plan determiner 314 can be configured to predict or estimate the number of user inputs used to complete a task or respond to a query via one or more UI elements on a UI screen. The query plan determiner 314 can be configured to predict or estimate the number of user inputs based on historical information associated with the user, statistical information associated with multiple users who have performed similar queries, the user's known preferences, the context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of user inputs used to order food from a restaurant, book a flight, remotely control a home appliance, etc. As a supplement (or alternative) to predicting or estimating the number of user inputs, the query plan determiner 314 can be configured to predict or estimate another metric associated with completing a task or responding to a query via a dialog operation (e.g., duration, processing load, etc.).

[0102] Similarly, the query plan determiner 314 can be configured to predict or estimate the number of user inputs required to obtain certain information needed to complete a task or respond to a query via one or more UI elements on a UI screen. The query plan determiner 314 can be configured to predict or estimate the number of user inputs based on historical information associated with the user, statistical information associated with multiple users who provided similar information, the user's known preferences, the context of the query, etc. For example, the query plan determiner 314 can be configured to predict or estimate the number of user inputs required to identify pizza toppings, identify a desired airline, control the volume of stereo speakers, etc. As a supplement (or alternative) to predicting or estimating the number of user inputs, the query plan determiner 314 can be configured to predict or estimate another metric associated with obtaining certain information via user input (e.g., duration, processing load, etc.).

[0103] refer to Figure 3B An example system for dynamically generating query plans according to an example embodiment of this disclosure is shown. Figure 3B In this process, the query plan determiner 314 can receive one or more of the following as inputs: user data 332, device data 334, query data 336, context data 338, query plan statistics data 342, external source data 344, and feedback data 346.

[0104] In some example embodiments, user data 332 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, user data 332 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, user data 332 may include information about one or more user profiles, including various types of user data such as user preference data, user demographic data, user calendar data, user social network data, user historical travel data, user historical query agent data, etc. For example, user data 332 may include, but is not limited to: email data, including text content, images, email-associated calendar information, or contact information; social media data, including comments, ratings, check-ins, likes, invitations, contacts, or appointments; calendar application data, including dates, times, events, descriptions, or other content; virtual wallet data, including purchases, e-tickets, coupons, or transactions; scheduling data; location data; SMS data; or other suitable data associated with a user account. According to one or more examples of this disclosure, the data may be analyzed to determine user preferences for queries, for example, to automatically suggest or automatically determine preferred restaurants, airlines, navigation routes, navigation modes, etc.

[0105] User data 332 is provided to illustrate potential data that, in some embodiments, can be analyzed by a computing system to identify user preferences, determine the best method for exchanging information with the user (via dialogue operations and / or input to a UI screen), generate appropriate UI screens with appropriate UI elements, etc. However, such user data may not be collected, used, or analyzed unless the user has given consent after being informed of what data is being collected and how such data will be used. Furthermore, in some embodiments, tools may be provided to the user (e.g., in a query agent application or via a user account) to revoke or modify the scope of permissions. Additionally, certain information or data may be processed in one or more ways before being stored or used, such that personally identifiable information is removed or stored in an encrypted manner. Therefore, specific user information stored as user data 332 may or may not be accessible to the computing system based on permissions granted by the user, or such data may not be stored as user data 332 at all.

[0106] In some example embodiments, device data 334 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, device data 334 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, device data 334 may include specification information about a computing device (e.g., user computing system 102) associated with a user, used to provide input via dialog operations and / or input to a UI screen, which may be provided for display on a monitor. For example, the specification information may include display information associated with the display device of the computing device. Display information may include the display resolution, display size, display color information, display font information, display type, display layout information, tolerance information associated with the display, text position information of the display, etc. Device data 334 may also include information related to the processing power of the computing device, information related to the storage capacity of the computing device, compatibility information associated with the computing device, etc.

[0107] In some example embodiments, query data 336 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, query data 336 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, query data 336 may include information about or corresponding to a user query. For example, query data 336 may include characteristic or semantic information associated with the query. Query data 336 may include intent information, entity information (e.g., date or time associated with the appointment query), contextual information (e.g., previous user input, previous responses from the query agent, user preferences, time information, etc.), language information, the tone of the user query, information related to the user's emotions or feelings, etc.

[0108] In some example embodiments, context data 338 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, context data 338 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, context data 338 may include environmental information about the user's surroundings (e.g., whether the query is associated with a home environment, office environment, entertainment environment (e.g., a movie theater, stadium, etc.), the current day time associated with the query, the current year time associated with the query, weather information, noise information, etc. Context data 338 may include other contextual information, including whether the user is in a vehicle, indoors, outdoors, the user's location information, etc. Context data 338 may further include previous query history information, user preferences, user sentiment and emotion information, etc.

[0109] In some example embodiments, query plan statistics 342 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, query plan statistics 342 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, query plan statistics 342 may include statistics about previous dialogue exchanges and / or previous inputs to the user interface associated with the same or similar queries provided by the user. The statistics about previous dialogue exchanges and / or previous inputs to the user interface associated with the same or similar queries provided by the user may include metrics including time information associated with responding to the query or for completing tasks associated with the query, information about the number of user inputs for processing the query in the dialogue exchanges and / or UI screens, information about the computing resources used to process the query, etc.

[0110] Statistical information regarding previous conversational exchanges and / or previous inputs to the user interface associated with the same or similar queries provided by the user can include information about expected inputs or responses received from the user for a particular query. For example, the statistics can include data reflecting the options a user is most likely to choose to complete a task. For instance, statistics could indicate that 95% of users want to identify the pizza toppings they want, or that based on previous statistics, a particular user is unlikely to request a specific topping, and therefore that topping may not be offered as an option on the main menu. Query planning statistics 342 can further include data tracking historical interactions between the query agent 312 and the user, indicating a preference for exchanging certain information associated with the query using conversational actions versus UI screen input. For example, query planning statistics 342 could indicate that most users (or a particular user) prefer user input via the UI screen to providing certain information (e.g., pizza toppings) through back-and-forth conversational exchanges. For example, querying the plan statistics 342 can indicate that most users (or specific users) prefer to receive certain information via graphics provided on the UI screen (e.g., a map showing the location of a restaurant) rather than receiving large amounts of information in the form of text or audio (e.g., a list of addresses corresponding to the restaurant locations).

[0111] In some example embodiments, external source data 344 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, external source data 344 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, external source data 344 may include any form of external content, including news articles, web pages, video files, audio files, written descriptions, ratings, game content, social media content, photos, business offers, transportation methods, weather conditions, sensor data obtained from various sensors, or other suitable external content. External source data 344 may be searched by the computing system using known search methods, and search results may be ranked based on relevance, popularity, or other suitable attributes, including location-specific filtering or promotion. For example, external source data 344 may be used to predict or estimate information about the expected conversational exchange used to process a query, the expected input to a UI screen used to process a query, etc.

[0112] In some example embodiments, feedback data 346 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, feedback data 346 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, feedback data 346 may include information provided by one or more users related to a query plan determined by query plan determiner 314. For example, feedback data 346 may include information instructing users to approve or agree (or disapprove or object) to: a proposed plan for processing the query using a combination of dialogue and input to UI elements provided to the UI screen; a proposed plan for processing the query using only dialogue; or a proposed plan for processing the query using only input to UI elements provided to the UI screen. For example, feedback data 346 may include information indicating the user's approval or consent (or disapproval or objection) to: processing the query using a combination of dialogue and input to UI elements provided to the UI screen; processing the query using only dialogue; or processing the query using only input to UI elements provided to the UI screen. This information can be used to further improve the operation of query plan determiner 314, which includes query plan machine learning model 316 and query plan complexity determiner 318. Feedback data 346 may include other information, including feedback on the quality or appropriateness of the computational system's response to the dialogue, the quality or appropriateness of the UI elements provided for user selection, and the quality or appropriateness of the response to or completion of the task associated with the query.

[0113] like Figure 3B As shown, the query plan determiner 314 may include a query plan machine learning model 316 and a query plan complexity determiner 318.

[0114] In some implementations, the query plan machine learning model 316 can be configured to process one or more of the following input data (e.g., user data 332, device data 334, query data 336, context data 338, query plan statistics 342, external source data 344, feedback data 346, etc.) to generate output. For example, the output may include classifying the query as a type of request for performing a task (e.g., booking a table at a restaurant, booking a flight, remotely controlling a home appliance, controlling an entertainment system, etc.). For example, the output may include a query plan that identifies anticipated or possible actions for processing the query. For example, the output may include determinations about which actions will be better handled via dialogue with the user (exchange) and which actions will be better handled via input to UI elements on a UI screen.

[0115] For example, the query plan machine learning model 316 can be configured to process input data based on statistical data to generate a predictive output. Alternatively, the query plan machine learning model 316 can be configured to process input data based on sensor data indicating the context of the query to generate a predictive output. For example, sensor data indicating that the user is in a noisy environment could cause the query plan machine learning model 316 to output a query plan that includes at least some query operations performed via user input through a UI screen. For example, sensor data indicating that the user is driving could cause the query plan machine learning model 316 to output a query plan that includes at least some query operations performed via a dialogue exchange, unless the user stops and it is safe to provide user input through a UI screen. Therefore, the query plan machine learning model 316 can be configured to dynamically change the query plan based on the user's state.

[0116] In some implementations, the computing system (more specifically, query plan machine learning model 316) can be configured to generate a query processing plan by dynamically determining whether to receive information from the user for processing the query via a dialog operation or input to UI elements provided on a UI screen, by means of: determining the context associated with the user and, based on that context, determining the suitability of receiving at least some information via the dialog operation for processing the query. In response to the computing system (more specifically, query plan machine learning model 316) determining that the suitability of receiving at least some information via the dialog operation for processing the query is below a threshold dialog suitability level, the computing system (more specifically, query plan machine learning model 316) can be configured to dynamically determine whether to receive at least some information from the user for processing the query via input to UI elements provided on a UI screen, and the computing system (more specifically, UI generator 326) can be configured to dynamically generate at least one of a UI screen or UI elements such that the user can provide at least some of the information via input to UI elements provided on the UI screen. For example, the context associated with a user may include environmental information about the user's surroundings (e.g., information indicating the user is in an office environment, elevator, movie theater, concert, or other environmental information indicating the user's state in a particular environment). When the environmental information about the user's surroundings indicates that providing input via a dialogue operation would be inappropriate (e.g., noise levels are too high, a particular location is known to be unsuitable for speaking loudly, etc.), the computational system (more specifically, query planning machine learning model 316) may be configured to determine that the suitability for receiving at least some information via a dialogue operation to process the query is below a threshold dialogue suitability level. Here, the threshold dialogue suitability level may correspond to the probability level (probability level) that the user can successfully provide input to the UI screen. In some implementations, specific environments may be excluded from the environments used to provide dialogue operations, or they may have a suitability level set to zero. As an example, when the context information indicates that the user is in an elevator or at a concert with noise levels above a threshold level, the computational system (more specifically, query planning machine learning model 316) may be configured to determine that the suitability for receiving at least some information via a dialogue operation to process the query is below a threshold dialogue suitability level. As another example, when contextual information indicates that the user is in a quiet home environment, the computational system (more specifically, query plan machine learning model 316) can be configured to determine that the suitability of receiving at least some information via dialogue for processing the query is greater than a threshold dialogue suitability level. Therefore, query plan machine learning model 316 can be configured to dynamically change the query plan based on the user's state.

[0117] In some implementations, the computing system (more specifically, query plan machine learning model 316) can be configured to generate a query processing plan by dynamically determining whether to receive information from the user for processing the query via at least one of a dialog operation or input to UI elements provided on the UI screen, in order to: determine the context associated with the user and, based on that context, determine the appropriateness of receiving at least some information via input to UI elements provided on the UI screen for processing the query. In response to the computing system (more specifically, query plan machine learning model 316) determining that the appropriateness of receiving at least some information via input to UI elements provided on the UI screen for processing the query is below a threshold UI appropriateness level, the computing system (more specifically, query plan machine learning model 316) can be configured to dynamically determine at least some of the information received from the user via a dialog operation for processing the query. For example, the context associated with the user may include environmental information about the user's surroundings (e.g., environmental information indicating that the user is driving a car or environmental information indicating the user's state in a particular environment). When environmental information about the user's surroundings indicates that providing input to UI elements displayed on the UI screen would be inappropriate (e.g., unsafe or impractical), the computing system (more specifically, query planning machine learning model 316) can be configured to determine that the suitability of receiving at least some information via input to UI elements displayed on the UI screen for processing the query is less than a threshold UI suitability level. Here, the threshold UI suitability level can correspond to the probability (probability level) that the user can safely or successfully provide input to the UI screen. As an example, when contextual information indicates that the user is driving a vehicle at high speed or at a speed greater than a certain threshold level (e.g., ten miles per hour), the computing system (more specifically, query planning machine learning model 316) can be configured to determine that the suitability of receiving at least some information via input to UI elements displayed on the UI screen for processing the query is less than the threshold UI suitability level. As another example, when contextual information indicates that the user is in a vehicle but the vehicle is stationary, the computing system (more specifically, query plan machine learning model 316) can be configured to determine that the suitability for processing the query by receiving at least some information via input to UI elements provided on the UI screen is greater than a threshold UI suitability level. Therefore, query plan machine learning model 316 can be configured to dynamically change the query plan based on the user's state.

[0118] In some implementations, instead of a query plan machine learning model 316, a query plan complexity determiner 318 can be configured to determine a plan for processing the query based on statistics associated with the query. For example, the query plan complexity determiner 318 can be configured to determine or estimate the expected complexity of processing the query or a portion of the query, along with its corresponding complexity value or metric, and determine whether to utilize a dialog interaction (exchange) or a UI screen to process the query or that portion of the query. In some implementations, the query plan complexity determiner 318 can be implemented as a classifier model that categorizes expected dialog interactions related to the information that may be needed to process the query as complex or uncomplex. When the expected dialog interaction is categorized as complex, the query plan determiner 314 can be configured to determine to process the query by implementing a UI screen that includes one or more UI elements to exchange information with the user. When the expected dialog interaction is categorized as uncomplex, the query plan determiner 314 can be configured to determine to process the query by exchanging information with the user via a dialog interaction.

[0119] For example, the complexity value or metric may be based on the expected number of dialogue exchanges between a user or a computing system for processing a query entirely via dialogue or for performing intermediate operations via dialogue in response to a query. In some implementations, when the complexity value used to process the query exceeds a threshold complexity value (e.g., a threshold number of dialogue exchanges), the query plan complexity determiner 318 may be configured to dynamically determine, via input to UI elements provided on a UI screen, to receive at least some information from the user for processing the query, and the computing system (e.g., UI generator 326) may be configured to dynamically generate at least one of the UI screens or UI elements. In some implementations, when the complexity value used to process a portion of the query (e.g., intermediate operations) exceeds a threshold complexity value, the query plan complexity determiner 318 may be configured to dynamically determine, via input to UI elements provided on a UI screen, to receive information from the user associated with that portion of the query, and the computing system (e.g., UI generator 326) may be configured to dynamically generate at least one of the UI screens or UI elements.

[0120] For example, as a supplement to or alternative to the expected number of dialogue exchanges, the complexity value or metric may be based on the duration of the expected dialogue exchanges between the user or computing system used to process the query entirely via dialogue or to perform intermediate operations via dialogue in response to the query. In some implementations, when the complexity value used to process the query exceeds a threshold complexity value (e.g., a threshold duration), the query plan complexity determiner 318 may be configured to dynamically determine at least some information received from the user via input to UI elements provided on the UI screen for processing the query. In some implementations, when the complexity value used to process a portion of the query exceeds a threshold complexity value (e.g., five dialogue exchanges, ten dialogue exchanges, etc.), the query plan complexity determiner 318 may be configured to dynamically determine information received from the user via input to UI elements provided on the UI screen associated with that portion of the query.

[0121] The output of the query plan machine learning model 316 or the query plan complexity determiner 318 can be provided to the query plan generator 320 to generate one or more dialog actions 322 for processing the query (or a part of the query), and / or one or more UI inputs 324 for processing the query (or a part of the query).

[0122] As described herein, at operation 2200, query plan determiner 314 and query plan generator 320 can dynamically and in real time generate query plans in response to user queries.

[0123] For example, the query plan determiner 314 can determine that various information needs to be obtained to process the query, including a first part of the information and a second part of the information.

[0124] At operation 2300, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to determine whether to receive the first part of the information via UI elements of the UI screen. More specifically, query plan determiner 314 can determine whether to receive the first part of the information via UI elements of the UI screen. For example, query plan determiner 314 can determine whether to receive the first part of the information via UI elements of the UI screen based on the output of query plan machine learning model 316 or query plan complexity determiner 318, as described herein.

[0125] When query plan determiner 314 determines that the first portion of information is received via a UI element on a UI screen, at operation 2310, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to provide a UI screen including the UI element, and at operation 2320, the computing system can be configured to receive the first portion of information via input to the UI element. For example, the computing system can be configured to dynamically generate the UI element and / or a UI screen including the UI element in response to dynamically determining that the first portion of information received from the user via input to a UI element provided on the UI screen is for processing a query. The generation of the UI element and / or the UI screen including the UI element will be related to... Figure 3C Describe it.

[0126] refer to Figure 3C This illustrates an example system for dynamically generating UI screens and UI elements according to an example embodiment of the present disclosure. Figure 3C In this process, the UI generator 326 can receive one or more of the following as input: user data 362, device data 364, query data 366, context data 368, UI library 372, UI code 374, external source data 376, feedback data 378, etc.

[0127] User data 362 can correspond to user data 362, device data 364 can correspond to, query data 366 can correspond to query data 336, context data 368 can correspond to context data 338, external source data 376 can correspond to external source data 344, feedback data 378 can correspond to feedback data 346, and for the sake of brevity, a detailed description of each of these features will be omitted.

[0128] In short, user data 362 may include data that can be used to generate appropriate UI elements and / or UI screens. For example, UI generator 326 may be configured to generate UI and / or UI elements based on user preferences (e.g., preference for sliders relative to selecting buttons with specific values) or user characteristics (e.g., setting font size based on the user's eyesight).

[0129] For example, UI generator 326 can be configured to generate UI and / or UI elements based on information about the computing device associated with the user to display a UI screen, which can be obtained from device data 364.

[0130] For example, UI generator 326 can be configured to generate UI and / or UI elements based on information included in query data 366. For example, UI and / or UI elements may include information directly related to the query (e.g., including selectable UI elements associated with a date or time mentioned in the query), or information related to the query (e.g., using appropriate colors to soothe the user when query data 366 indicates that the user is emotionally agitated).

[0131] For example, UI generator 326 can be configured to generate UI and / or UI elements based on information included in context data 368. For instance, UI and / or UI elements can be provided based on environmental information (e.g., lighting information instructing a user to view the display in a dark environment could cause UI generator 326 to generate a UI screen that is easier to view in the dark). As another example, UI and / or UI elements can be provided based on movement information associated with the user (e.g., movement information instructing a user to move quickly could cause UI generator 326 to generate a UI screen with fewer (and larger) UI elements that are easier to view and select while the user is moving).

[0132] In some example embodiments, the UI library 372 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, the UI library 372 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, the UI library 372 may include a pre-designed collection of components, templates, and tools, and the UI generator 326 may be configured to use this collection of components, templates, and tools to generate UI elements and / or UI screens to obtain or provide information to the user for processing a query. For example, the UI library 372 may include buttons, forms, navigation menus, cards, modals, etc. For example, the UI library 372 may include CSS frames, HTML frames, etc. For example, the UI generator 326 may be configured to dynamically generate a UI screen including one or more UI elements for processing a query by selecting various components from the UI library. For example, the UI generator 326 may be configured to select one or more UI elements from the UI library 372, which includes multiple UI elements, based on the context of the query.

[0133] In some example embodiments, UI code 374 may be stored in a single database accessible to the computing system (e.g., via network 180). In some embodiments, UI code 374 may be stored in multiple different databases accessible to the computing system (e.g., via network 180). In some examples, UI code 374 may include Hypertext Markup Language (HTML) code that provides the structure and content of the UI screen using markup elements that define various components of the UI, such as headings, paragraphs, lists, forms, images, links, etc. In some examples, UI code 374 may include Cascading Style Sheets (CSS) code that can be implemented to control the rendering and styling of HTML elements and define how UI elements behave, including aspects such as color, font, spacing, positioning, and responsiveness. In some examples, UI code 374 may include JavaScript code that can be used to add animations, validate form data, retrieve data from a server, and perform real-time updates to the UI. For example, UI generator 326 may be configured to implement UI code 374 to dynamically generate UI screens that include one or more UI elements for processing queries.

[0134] For example, UI generator 326 can be configured to generate UI and / or UI elements based on information included in external source data 376. For example, when UI generator 326 dynamically generates a UI screen with UI elements, external source data 344 can be used to populate the UI elements with information obtained from websites or other electronic resources for presentation on the UI screen (e.g., pricing information, menu information, travel information, etc.).

[0135] For example, UI generator 326 can be configured to generate UI and / or UI elements based on the information included in feedback data 378. For example, feedback data 346 may include feedback on the quality or suitability of UI elements provided for user selection, the quality or suitability of UI screens, etc.

[0136] like Figure 3C As shown, UI generator 326 may include UI generator machine learning model 328. In some implementations, UI generator 326 may be configured to dynamically (e.g., in real-time) generate (e.g., generate in real-time) UI elements and / or UI screens as output based on one or more of various inputs (e.g., user data 362, device data 364, query data 366, context data 368, UI library 372, UI code 374, external source data 376, feedback data 378, etc.).

[0137] For example, UI generator machine learning model 328 can be implemented by UI generator 326 to generate one or more UI elements 384 and / or UI screen 382 for UI screen 382. For example, UI generator machine learning model 328 can be implemented as a generative machine learning model to dynamically generate UI screen 382 including one or more UI elements 384 included in UI screen 382, ​​or to dynamically generate one or more UI elements 384. For example, UI generator machine learning model 328 can be configured to dynamically generate code for creating one or more UI elements to be included in UI screen, and / or dynamically generate code for creating UI screen. For example, this code can be generated based on training data including code from UI code 374. The generated code for creating UI screen and / or one or more UI elements can include Hypertext Markup Language, Cascading Style Sheets, JavaScript, etc.

[0138] In some implementations, UI generator 326 can be configured to select UI screens from a first library containing multiple UI screens based on the context of a query, and UI generator 326 can be configured to select one or more UI elements from a second library containing multiple UI elements based on the context of a query.

[0139] For example, the UI generator machine learning model 328 can be trained to generate UI elements and UI screens based on training data, which may include existing UI designs containing various UI elements such as buttons, text fields, icons, layouts, color schemes, etc. For example, the UI generator machine learning model 328 can be trained to generate UI elements and UI screens based on UI elements obtained from UI library 372 and code provided from UI code 374. The UI generator machine learning model 328 may include neural networks, such as generative adversarial networks (GANs) or variational autoencoders (VAEs), which have interconnected layers of nodes that process information and learn patterns from input data. The UI generator machine learning model 328 can be configured to improve over time based on an iterative process involving generating multiple designs, receiving feedback, and making adjustments.

[0140] The dynamically generated UI screen 382 and one or more UI elements 384 can be provided for presentation on a display device associated with the user to process queries. In some implementations, the user can provide input to one or more UI elements 384 to obtain information for query processing in a more efficient and / or convenient manner than obtaining information for query processing via dialogical operations. In some implementations, the UI screen 382, ​​including one or more UI elements 384, can provide information for query processing to the user in a more efficient and / or convenient manner than providing information for query processing via dialogical operations. For example, the UI screen 382 can include graphical images that can quickly convey a large amount of information to the user, rather than the query agent 312 providing such information through multiple dialogical operations that may be difficult for the user to remember, inconvenient to read in text form, or may take longer.

[0141] Return to reference Figure 2A When query plan determiner 314 determines that the first portion of information has not been received by a UI element on the UI screen, at operation 2330, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to receive the first portion of information via a dialogue operation. For example, the computing system can be configured to, in response to dynamically determining that the first portion of information received from the user via a dialogue operation is for processing a query, provide the output of the first portion of the requested information (e.g., via text or audio), and receive the first portion of information from the user via text input or voice input.

[0142] At operation 2400, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to determine whether to receive the second part of the information via a dialogue operation (exchange). More specifically, query plan determiner 314 can determine whether to receive the second part of the information via a dialogue operation. For example, query plan determiner 314 can determine whether to receive the second part of the information via a dialogue operation based on the output of query plan machine learning model 316 or query plan complexity determiner 318, as described herein.

[0143] When the query plan determiner 314 determines the second part of the information received via a dialogue operation, at operation 2410, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to receive the second part of the information via the dialogue operation. For example, the computing system can be configured to, in response to dynamically determining the second part of the information received from the user via the dialogue operation for processing the query, provide the output of the second part of the requested information (e.g., via text or audio), and receive the second part of the information from the user via text input or voice input.

[0144] When query plan determiner 314 determines that the second part of the information has not been received by the dialogue operation, at operation 2420, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to provide a UI screen including UI elements, and at operation 2430, the computing system can be configured to receive the second part of the information via input to the UI elements. For example, the computing system can be configured to dynamically generate UI elements and / or a UI screen including those UI elements in response to dynamically determining that the second part of the information received from the user via input to UI elements provided on the UI screen is for processing the query. The generation of UI elements included in the UI screen and / or a UI screen including those UI elements is related to... Figure 3C Described.

[0145] At operation 2500, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to process queries based on a first part of the information and a second part of the information (whether received via dialogue or input to UI elements on a UI screen).

[0146] Figure 2B A flowchart is depicted for another example method performed according to an example embodiment of this disclosure. Although Figure 2B For illustrative and discussion purposes, operations performed in a particular order are depicted; however, the methods of this disclosure are not limited to the order or arrangement specifically shown. Various operations of method 2000' may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0147] According to some examples of this disclosure, computing systems (e.g., user computing system 102, server computing system 130, training computing system 150) can be configured to process queries by dynamically generating user interface (UI) elements (e.g., via an implementation of a machine learning model), the UI elements being configured to receive information from the user for query processing.

[0148] For example, similar to Figure 2A Operation 2100, in Figure 2BAt point 2600, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives a query from the user. For example, the computing system can be configured to receive queries from the user via a query agent. For example, a query agent can refer to a chatbot or chatbot, which can be implemented as a computer program or application configured to simulate human conversation through text or voice interaction. The query agent can be configured to utilize artificial intelligence (AI) and natural language processing (NLP) technologies to understand user input and generate relevant responses. For example, the query agent can be configured to automate and facilitate communication between the user and the computer system, thereby providing information, answering questions, and performing tasks in a conversational manner.

[0149] For example, a user's query input can be a question (e.g., "Where can I find good Chinese food near John's house tonight?") or a command (e.g., "I want to book a flight" or "Tell me when the tide is out"). For example, a user can input a query into the computing system via voice or text (e.g., via user input component 122). The query agent can be configured to respond to the query by determining the information needed to process it. Query processing can include various intermediate operations between the user and the query agent before fully responding to the query and completing the task.

[0150] For example, at 2650, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) dynamically generates user interface (UI) elements via a machine learning model, which are configured to receive information from the user. Figure 3C An example system for dynamically generating UI elements is shown. For example, Figure 3C The example system can be implemented to dynamically generate UI elements via a UI generator machine learning model 328, which are configured to receive information from the user. Figure 3C The detailed operation of the sample system used to generate UI elements and UI screens containing UI elements is described in this article and will not be repeated for the sake of brevity.

[0151] For example, at 2700, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) provides UI elements for presentation on a display device associated with the user. For example, one or more UI elements 384 may be provided for presentation on a display device associated with the user to process queries.

[0152] For example, at 2750, a computing system (e.g., user computing system 102, server computing system 130, training computing system 150) receives information from a user via input to UI elements. For example, a user can provide input to one or more UI elements 384 to obtain information for query processing in a more efficient and / or convenient manner than obtaining information via dialogue. For example, if a UI element is configured as a selectable checkbox, a user can select the checkbox (e.g., via touch input); if a UI element is configured as a scrollbar, a user can manipulate the scrollbar, etc. UI elements can be configured as interactive or selectable UI elements to allow the user to easily provide input that provides information to the computing system for query processing.

[0153] For example, at 2800, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) generates a response to the query based on information received from the user via input to UI elements. For example, the computing system can generate any kind of output for responding to the query, such as that determined via query processor 310, query agent 312, query plan determiner 314, etc. For example, if the user selects a UI element corresponding to a restaurant at a specific location, the computing system can be configured to generate a response including directions or addresses to that restaurant, or dynamically generate further UI elements to continue processing the query to receive further information (e.g., choices of food, choices of suppliers to deliver food to the user, etc.). For example, if the user selects a UI element corresponding to a restaurant at a specific location, the computing system can be configured to generate a response including a menu of that restaurant with selectable UI elements. In some implementations, the generated response can be a dialogic action. For example, the dialogic action can confirm a previous selection, confirm task completion, inquire about further information needed to process the query, etc. The response can be any kind of output used to perform, complete, or advance query processing.

[0154] For example, at 2850, the computing system (e.g., user computing system 102, server computing system 130, training computing system 150) provides the response to the display device associated with the user. For example, the computing system may provide the response via another UI element (including in a UI screen) provided for presentation on the display device, via dialogue, or via another method (e.g., haptic feedback, audio feedback, etc.).

[0155] Figures 4A to 4E An example interaction for processing a query via a chat agent, according to an example embodiment of this disclosure, is depicted. Figure 4AIn this context, the interaction between the chat agent (query agent) and the user is depicted on a display device associated with the user. In the first screenshot 4100, the user orders pizza by entering a query 4130 via a dialogue action. The chat agent responds with another dialogue action 4120, requesting certain information from the user (e.g., which restaurant the user wants to order pizza from). As described herein, dialogue actions correspond to text input or text output, or voice input via a microphone (e.g., by selecting a UI element 4110 depicted as a microphone to enable the microphone), or audio output via the speakers of the computing device associated with the user.

[0156] exist Figure 4B In this context, further interactions between the chat agent (query agent) and the user are depicted on a display device associated with the user. In the second screenshot 4200, the chat agent outputs a dialogue action 4210 indicating three restaurants that match the user's preferred restaurant options. Here, according to an example embodiment of this disclosure, the query plan determiner 314 can be configured to determine whether to provide and receive information related to these three restaurants via multiple UI elements on the UI screen rather than through a dialogue action. For example, the UI generator 326 can be configured to implement a UI generator machine learning model 328 to dynamically (e.g., in real-time) generate an interactive map 4220 depicting the locations of the three restaurants relative to the user, where the user can select a preferred restaurant location via touch input 4230 on UI elements 4240 corresponding to the restaurant.

[0157] exist Figure 4C In this context, further interactions between the chat agent (query agent) and the user are depicted on a display device associated with the user. In the third screenshot 4300, the chat agent outputs a dialogue action 4310, which requests additional information from the user to complete a task associated with the query. Here, according to an example embodiment of this disclosure, the query plan determiner 314 may be configured to determine whether to provide and receive query-related information (e.g., a pizza order) via multiple UI elements of a UI screen rather than through a dialogue action. For example, the UI generator 326 may be configured to implement a UI generator machine learning model 328 to dynamically (e.g., in real-time) generate a UI screen 4320 that includes multiple UI elements, including a tortilla option 4330, where the user can select a preferred tortilla via touch input 4340 on a UI element 4350 that is a selectable checkbox. Other UI elements may include a checkmark 4360 indicating that the user has made a selection; and an X 4370 indicating the cancellation of the selection.

[0158] exist Figure 4DIn this context, further interactions between the chat agent (query agent) and the user are depicted on a display device associated with the user. In the fourth screenshot 4400, the chat agent outputs a dialogue action 4410, which requests additional information from the user to complete a task associated with the query. Here, according to an example embodiment of this disclosure, the query plan determiner 314 can be configured to determine whether to provide and receive query-related information (e.g., pizza order) via multiple UI elements of a UI screen rather than through a dialogue action. For example, the UI generator 326 can be configured to implement a UI generator machine learning model 328 to dynamically (e.g., in real-time) generate a UI screen 4420 including multiple UI elements, including a time option 4430, whereby the user can select a pizza pickup time via touch input 4440 on the UI elements, the touch input corresponding to a selectable scroll bar (for selecting a delivery time). In some implementations, the UI generator machine learning model 328 can be configured to restrict or define selectable values ​​based on the context of the query or user preferences (e.g., restricting values ​​associated with pizza pickup time, omitting selectable toppings that the user is allergic to, etc.). In some implementations, the UI generator machine learning model 328 can be configured to pre-select selectable values ​​based on the context of the query (e.g., pre-selecting the "PM" value based on the current time of the query, or pre-selecting toppings known to be the user's favorite, associated with pizza pickup time).

[0159] exist Figure 4E Further interactions between the chat agent (query agent) and the user are depicted on a display device associated with the user. In the fifth screenshot 4500, the chat agent outputs a dialogue action 4510, which requests whether the user is ready to submit their order. Here, according to an example embodiment of this disclosure, the query plan determiner 314 can be configured to receive information via a dialogue action in response to a query from the chat agent. The chat agent may output another dialogue action indicating the completion of processing the query.

[0160] Figures 4A to 4E The example interaction depicted in the document, which involves processing queries via a chat agent, is merely an example, and this disclosure is not limited to that example. Aspects of this disclosure can be applied to a wide variety of other contexts, including travel booking, restaurant reservations, operating entertainment systems, operating home appliances, operating heating and cooling systems, navigation systems, and any other environment in which chat agents can be employed or implemented to assist users in completing tasks.

[0161] Additional Publication

[0162] Another exemplary aspect of this disclosure relates to a computer-implemented method for processing (or responding to) a query (task). The method includes: receiving a query from a user; generating a plan for processing the query by dynamically determining whether information is received from the user for processing the query via at least one of a dialog operation or input to a UI element provided on a user interface (UI) screen; when it is determined that a first portion of the information is received from the user via the UI element: providing a UI screen including the UI element for presentation on a display device associated with the user, and receiving the first portion of the information for processing the query via input to the UI element; when it is determined that a second portion of the information is received from the user via the dialog operation: receiving the second portion of the information for processing the query via the dialog operation; and processing the query in response to the query based on the first portion of the information received from the user via input to a UI element provided on the UI screen and the second portion of the information received from the user via the dialog operation.

[0163] In some implementations, the method includes dynamically generating the UI element in response to dynamically determining a first portion of information received from the user via input to a UI element provided on the UI screen for processing the query.

[0164] In some implementations, dynamically generating the UI element involves selecting the UI element from a library containing multiple UI elements based on the context of the query.

[0165] In some implementations, dynamically generating the UI element includes dynamically generating code via a machine learning model to create the UI elements included in the UI screen.

[0166] In some implementations, the code includes code for creating the UI element using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

[0167] In some implementations, the method includes dynamically generating the UI screen and the UI element in response to dynamically determining a first portion of information received from the user via input to a UI element provided on the UI screen for processing the query.

[0168] In some implementations, dynamically generating the UI screen includes: selecting the UI screen from a first library containing multiple UI screens based on the context of the query, and selecting the UI element from a second library containing multiple UI elements based on the context of the query.

[0169] In some implementations, dynamically generating the UI screen and the UI elements includes dynamically generating code via a machine learning model to create the UI screen and the UI elements included in the UI screen.

[0170] In some implementations, the code used to create the UI screen and the UI elements included in the UI screen includes code used to create the UI screen and the UI elements included in the UI screen using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

[0171] In some implementations, generating a plan to process a query by dynamically determining whether to receive information from the user via the dialog operation or input to a UI element provided on the UI screen for processing the query includes: determining a complexity value associated with processing the query; dynamically determining, when the complexity value exceeds a threshold complexity value, a first portion of information received from the user via input to a UI element provided on the UI screen for processing the query; and dynamically generating the UI screen or at least one of the UI elements.

[0172] In some implementations, this complexity value is based on the expected number of dialogue exchanges between the user and the computing system to process the query entirely via dialogue.

[0173] In some implementations, this complexity value is based on the expected number of dialogue exchanges between the user and the computing system to perform intermediate operations via dialogue in response to the query.

[0174] In some implementations, generating a plan to process a query by dynamically determining whether to receive information from the user via the dialog operation or input to a UI element provided on the UI screen to process the query includes: determining a context associated with the user; determining, based on the context, the suitability of receiving at least some information via the dialog operation to process the query; dynamically determining, in response to determining that the suitability of receiving at least some information via the dialog operation to process the query is below a threshold dialog suitability level, to receive at least some information from the user via input to a UI element provided on the UI screen to process the query; and dynamically generating the UI screen or at least one of the UI elements such that the user can provide at least some information via input to a UI element provided on the UI screen.

[0175] In some implementations, the context associated with the user includes environmental information about the user's surrounding environment, and when the environmental information about the user's surrounding environment indicates that a dialogue exchange between the user and the computing system for processing the query via a dialogue operation would be inappropriate, it is determined that the suitability of receiving at least some information via the dialogue operation for processing the query is below a threshold dialogue suitability level.

[0176] In some implementations, generating a plan to process a query by dynamically determining whether to receive information from the user via the dialog operation or input to UI elements provided on the UI screen for processing the query includes: determining a context associated with the user; determining, based on the context, the suitability of receiving at least some information via input to UI elements provided on the UI screen for processing the query; and dynamically determining, in response to determining that the suitability of receiving at least some information via the dialog operation for processing the query is below a threshold UI suitability level, to receive at least some of the information from the user via the dialog operation for processing the query.

[0177] In some implementations, the context associated with the user includes environmental information about the user's surrounding environment, and when the environmental information about the user's surrounding environment indicates that providing input to UI elements provided on the UI screen would be inappropriate, it is determined that the suitability for processing the query by receiving at least some information via input to UI elements provided on the UI screen is below a threshold UI suitability level.

[0178] In some implementations, the method includes: implementing a first machine learning model to generate a plan for processing the query by dynamically determining whether information is received from the user via the dialogue operation or input to a UI element provided on the UI screen for processing the query; and implementing a second machine learning model to dynamically generate the UI screen or at least one of the UI elements in response to the first machine learning model dynamically determining that at least some of the information received from the user via input to a UI element provided on the UI screen for processing the query.

[0179] Another exemplary aspect of this disclosure relates to a computing system for processing queries. The computing system includes at least one memory device configured to store instructions; and at least one processor configured to execute the instructions to perform an operation including: receiving a query from a user; generating a plan for processing the query by dynamically determining whether information is received from the user via a dialog operation or input to a UI element provided on a user interface (UI) screen for processing the query; when a first portion of the information is determined to be received from the user via the UI element: providing a UI screen including the UI element for presentation on a display device associated with the user, and receiving the first portion of the information via input to the UI element for processing the query; when a second portion of the information is determined to be received from the user via the dialog operation: receiving the second portion of the information via the dialog operation for processing the query; and processing the query in response to the query based on the first portion of the information received from the user via input to a UI element provided on the UI screen and the second portion of the information received from the user via the dialog operation.

[0180] In some implementations, the operation further includes: implementing a first machine learning model to generate a plan for processing the query by dynamically determining whether information is received from the user via the dialogue operation or input to UI elements provided on the UI screen for processing the query; and implementing a second machine learning model to dynamically generate the UI screen or at least one of the UI elements in response to the first machine learning model dynamically determining that at least some of the information received from the user via input to UI elements provided on the UI screen for processing the query.

[0181] At least one processor of the computing system may also be configured to execute the instructions to perform an operation corresponding to any of the methods described herein.

[0182] Other aspects of this disclosure relate to various systems, devices, non-transitory computer-readable media, user interfaces, and electronic devices. In one or more example embodiments, a computer-readable medium (e.g., a non-transitory computer-readable medium) is provided that stores instructions executable by one or more processors of a computing system or computing device. In some implementations, the computer-readable medium stores instructions that may include instructions to cause one or more processors to perform one or more operations (e.g., operations of a computing system) of any of the methods described herein. The computer-readable medium may store additional instructions to perform other aspects of the computing system and corresponding methods of operation, as described herein.

[0183] For example, one aspect of this disclosure relates to a non-transitory computer-readable medium that commonly stores instructions that, when executed by one or more processors, cause the processors to perform operations including: receiving a query from a user; generating a plan for processing the query by dynamically determining whether information is received from the user via a dialog operation or input to a UI element provided on a user interface (UI) screen for processing the query; when a first portion of the information is determined to be received from the user via the UI element: providing a UI screen including the UI element for presentation on a display device associated with the user, and receiving the first portion of the information via input to the UI element for processing the query; when a second portion of the information is determined to be received from the user via the dialog operation: receiving the second portion of the information via the dialog operation for processing the query; and processing the query in response to the query based on the first portion of the information received from the user via input to a UI element provided on the UI screen and the second portion of the information received from the user via the dialog operation.

[0184] This paper discusses technologies related to servers, databases, software applications, and other computer-based systems, as well as the actions taken and the information sent to and from such systems. The inherent flexibility of computer-based systems allows for a wide range of possible configurations, combinations, and partitions of tasks and functionality between and within components. For example, the processes discussed herein can be implemented using a single device or component, or multiple devices or components working in combination. Databases and applications can be implemented on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0185] The aspects of the above-described example embodiments can be recorded in a non-transitory computer-readable medium, which includes program instructions for implementing various operations embodied by a computer. The medium may also include data files, data structures, etc., alone or in combination with the program instructions. Examples of non-transitory computer-readable media include magnetic media such as hard disks, floppy disks, and magnetic tapes; optical media such as CD-ROMs, Blu-ray discs, and DVDs; magneto-optical media such as optical discs; and other hardware devices specifically configured to store and execute program instructions, such as semiconductor memories, read-only memories (ROMs), random access memories (RAMs), flash memory, USB storage, etc. Examples of program instructions include machine code (such as that generated by a compiler) and files containing higher-level code that can be executed by a computer using an interpreter. The program instructions can be executed by one or more processors. The described hardware devices can be configured to act as one or more software modules to perform the operations of the above embodiments, and vice versa. Furthermore, non-transitory computer-readable storage media can be distributed among computer systems connected via a network, and computer-readable code or program instructions can be stored and executed in a distributed manner. In addition, non-transitory computer-readable storage media may also be embodied in at least one application-specific integrated circuit (ASIC) or field-programmable gate array (FPGA).

[0186] Each box in a flowchart can represent a unit, module, segment, or section of code, which includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions mentioned in the boxes may occur out of order. For example, two boxes shown consecutively may actually execute substantially concurrently (simultaneously), or these boxes may sometimes execute in reverse order, depending on the functionality involved.

[0187] While the subject matter has been described in detail with respect to various exemplary embodiments, each example is provided by way of explanation and not limitation. Modifications, variations, and equivalents of such embodiments will be readily apparent to those skilled in the art upon understanding the foregoing. Therefore, this disclosure does not exclude such modifications, alterations, and / or additions to the subject matter that will be readily understood by those of ordinary skill in the art. For example, features shown or described as part of one embodiment may be used with another embodiment to produce yet another embodiment. Therefore, this disclosure is intended to cover such modifications, alterations, and equivalents.

Claims

1. A computer-implemented method, comprising: Receive queries from users; User interface (UI) elements are dynamically generated via a machine learning model, and these UI elements are configured to receive information from the user. The UI elements are provided for presentation on a display device associated with the user; Receive information from the user via input to the UI elements; A response to the query is generated based on the information received from the user via the input to the UI element; as well as The response is provided to the display device associated with the user.

2. The computer-implemented method of claim 1, wherein dynamically generating the UI elements via the machine learning model comprises: The UI element is selected from a library containing multiple UI elements based on the context of the query.

3. The computer-implemented method of claim 1, wherein dynamically generating the UI elements via the machine learning model comprises: The machine learning model dynamically generates code for creating the UI elements.

4. The computer-implemented method of claim 3, wherein the code includes code for creating the UI element using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

5. The computer-implemented method of claim 1, further comprising dynamically generating a UI screen including the UI elements via the machine learning model.

6. The computer-implemented method of claim 5, wherein dynamically generating the UI screen via the machine learning model comprises: The UI screen is selected from a first library comprising multiple UI screens based on the context of the query, and The UI element is selected from a second library containing multiple UI elements based on the context of the query.

7. The computer-implemented method of claim 5, wherein dynamically generating the UI screen via the machine learning model comprises: The machine learning model dynamically generates code for creating the UI screen.

8. The computer-implemented method of claim 7, wherein the code includes code for creating the UI screen using at least one of Hypertext Markup Language, Cascading Style Sheets, or JavaScript.

9. The computer-implemented method of claim 1, further comprising: In response to receiving the query, a plan for processing the query is generated by dynamically determining the following: Is information received from the user via a dialogue operation to process the query, or The UI elements are dynamically generated via the machine learning model to receive the information for processing the query via the input to the UI elements.

10. The computer-implemented method of claim 9, wherein generating the plan for processing the query further comprises: Determine the complexity value associated with processing the query; as well as When the complexity value exceeds the threshold complexity value, the UI element is dynamically generated via the machine learning model to receive the information for processing the query via the input to the UI element.

11. The computer-implemented method of claim 10, wherein the complexity value is based on the number of anticipated dialogue exchanges between the user and the computing system for processing the query entirely via dialogue operations.

12. The computer-implemented method of claim 10, wherein the complexity value is based on the expected number of dialogue exchanges between the user and the computing system for performing intermediate operations via dialogue operations in response to the query.

13. The computer-implemented method of claim 9, wherein generating the plan for processing the query further comprises: Determine the context associated with the user; The suitability of receiving at least some information via the dialogue operation for processing the query is determined based on the context. as well as In response to determining that the suitability for receiving the at least some information via the dialogue operation to process the query is below a threshold dialogue suitability level, the UI element is dynamically generated via the machine learning model to receive at least some of the information via the input to the UI element to process the query.

14. The computer-implemented method of claim 13, wherein... The context associated with the user includes environmental information about the user's surrounding environment, and When environmental information about the user's surroundings indicates that the dialogue exchange between the user and the computing system for processing the query via dialogue operation would be inappropriate, it is determined that the suitability of receiving the at least some information via the dialogue operation for processing the query is below the threshold dialogue suitability level.

15. The computer-implemented method of claim 9, wherein generating the plan for processing the query further comprises: Determine the context associated with the user; The suitability of receiving at least some information via the input to the UI element for processing the query is determined based on the context. as well as In response to determining that the suitability for processing the query by receiving at least some information via the input to the UI element is below a threshold UI suitability level, at least some of the information received from the user via the dialogue operation is dynamically determined for processing the query.

16. The computer-implemented method of claim 15, wherein... The context associated with the user includes environmental information about the user's surrounding environment, and When the environmental information about the user's surrounding environment indicates that providing input to the UI element would be inappropriate, it is determined that the suitability for receiving the at least some information via the input to the UI element to process the query is below the threshold UI suitability level.

17. The computer-implemented method of claim 9, further comprising: Implement a further machine learning model to generate the plan for processing the query by dynamically determining whether to receive information from the user via the dialogue operation to process the query, or to dynamically generate the UI elements via the machine learning model to receive the information via the input to the UI elements to process the query; as well as In response to the further machine learning model dynamically determining the information received from the user via the input to the UI element for processing the query, the UI element is dynamically generated via the machine learning model and configured to receive the information from the user.

18. A computing system, comprising: At least one memory device, the at least one memory device being configured to store instructions; as well as At least one processor, configured to execute the instructions to perform an operation, the operation including: Receive queries from users; User interface (UI) elements are dynamically generated via a machine learning model, and these UI elements are configured to receive information from the user. The UI elements are provided for presentation on a display device associated with the user; Receive information from the user via input to the UI elements; A response to the query is generated based on the information received from the user via the input to the UI element; and The response is provided to the display device associated with the user.

19. The computing system of claim 18, wherein the operation further comprises: In response to receiving the query, a further machine learning model is implemented to generate a plan for processing the query by dynamically determining whether to receive information from the user via a dialog operation to process the query, or to dynamically generate the UI elements via the machine learning model to receive the information to process the query via input to the UI elements. as well as In response to the further machine learning model dynamically determining the information received from the user via the input to the UI element for processing the query, the UI element is dynamically generated via the machine learning model and configured to receive the information from the user.

20. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations, the operations including: Receive queries from users; User interface (UI) elements are dynamically generated via a machine learning model, and these UI elements are configured to receive information from the user. The UI elements are provided for presentation on a display device associated with the user; Receive information from the user via input to the UI elements; A response to the query is generated based on the information received from the user via the input to the UI element; as well as The response is provided to the display device associated with the user.

Images