A method for generating dynamic interactive user interface based on sketch function

Abstract

The present invention discloses a method for generating a dynamic interactive user interface based on a sketch function, and belongs to the technical field of electronic digital data processing. The method is implemented through an interactive dialogue between a hub service deployed on the server side and a user; after the hub service is turned on, a function tool set is registered with the LLM, user input tasks are received, the message list of this round is forwarded to the LLM, and the target function is called according to the LLM reasoning result. When further user input is required, the sketch structure describing the target user interface component is forwarded to the rendering engine, and the interactive result of the user clicking the target user interface component is added to the message list of this round and then forwarded to the LLM, and the LLM reasoning result is echoed to the user when the user input task is completed. The present invention realizes the dynamic generation of user interface components that can be clicked by the user's mouse on demand during the dialogue with the generative AI, simplifies user operations, protects user privacy, and improves the consistency of user experience.

Description

Technical Field

[0001] The present invention relates to generative AI application technology in the field of artificial intelligence, discloses a method for generating a dynamic interactive user interface based on a sketch function, belongs to the technical field of electronic digital data processing, and specifically relates to the technical field of interactive technology based on a graphical user interface. Background Art

[0002] Generative AI has been widely used in scenarios such as text generation, text generation images, text generation audio, and text generation video. Among them, text generation scenarios include chat, code, etc. Large Language Model (LLM) is an important subset of generative AI, which is specifically used to process and generate natural language.

[0003] In order to expand the capabilities of large language models, mainstream vendors such as OpenAI provide API services to developers, enabling them to use custom tools to enable large language models to interact with external systems or perform specific operations. Function tools are the most widely used tool type. The basic working principle of expanding the capabilities of large language models through function tools is as follows: Step 1, developers predefine a set of functions and deploy them on the server side, and provide the function specifications to the large language model; Step 2, the user sends a message to the large language model to start a round of dialogue; Step 3, the large language model infers and determines the function to be called, and sends the function call instruction to the server side; Step 4, the server side executes the function and returns the execution result to the large language model; Step 5, the large language model converts the function execution result into natural language and echoes it to the user, and this round of dialogue ends; Step 6, if you want to start a new round of dialogue, repeat steps 2 to 5.

[0004] At present, developer-defined functions are usually traditional functions that focus on processing background tasks, such as pure logical calculations or querying databases. If further user input is required during the execution process, the user can only be prompted with a text response message to start a new round of dialogue. At this time, the user needs to enter text through the keyboard or voice through the microphone. This type of interactive experience has the following defects: text input requires users to type manually, which is cumbersome and easy to fatigue; although voice input is natural, it is inconvenient to use in public places, and when private information is involved, the use of voice is prone to privacy risks. How to dynamically generate components that users can choose from on demand during the dialogue with generative AI, so as to simplify user operations while protecting user privacy, is a problem that needs to be solved at present. Summary of the invention

[0005] The purpose of the present invention is to address the deficiencies of the above-mentioned background technology and provide a dynamic interactive user interface generation method based on sketch functions, which dynamically generates components that can be clicked by the user on demand during the dialogue with generative AI, and replaces text or voice input with point-selection actions, thereby solving the technical problems of poor user experience and privacy leakage risks in generative AI dialogues, and achieving the purpose of the invention of improving the consistency of user experience.

[0006] The present invention adopts the following technical solutions to achieve the above-mentioned invention object:

[0007] A method for generating a dynamic interactive user interface based on sketch functions is implemented through an interactive dialogue between a hub service deployed on the server and a user. The hub service defines a function tool set, management logic for implementing communication with a large language model, and management logic for implementing a dialogue context with a large language model. The function tool set includes but is not limited to sketch functions and traditional functions. After starting the hub service, the hub service performs the following steps:

[0008] S1: Establish a connection with the large language model and register the function tool set to the large language model;

[0009] S2: Receive the user input task of this round of dialogue, initialize the message list of this round, and forward the message list of this round to the large language model;

[0010] S3: Receive the natural language results obtained by the large language model inference of the message list in this round, and echo the inference results of the large language model to the user when the user input task is completed. When the user input task is not completed, call the objective function according to the inference results of the large language model;

[0011] S4: Obtain the target function execution result, add the target function execution result to the message list of this round, forward the message list of this round to the large language model, and return to S3. The target function execution result includes but is not limited to the sketch function execution result and the traditional function execution result. The sketch function execution result includes but is not limited to the user interface component identifier and the interaction result of the user clicking the target user interface component. The traditional function execution result includes but is not limited to the execution result of the user input task.

[0012] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch functions, the sketch function obtains a structured description of the user interface component represented by a sketch structure after processing the input parameters through processing logic; the traditional function obtains a structured description of irrelevant user interface interaction text after processing the input parameters through processing logic; the sketch function and the traditional function are called through a calling interface that unifies the sketch function return value type and the traditional function return value type. The calling interface maps the function toolset into a mapping table from function name to function implementation and function description specification. The function description specification is described in natural language.

[0013] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch functions, the hub service defines the management logic for implementing communication with the large language model. Specifically, the same communication protocol as the large language model is adopted to build a large language model communication client that communicates with the large language model. The large language model communication client establishes a connection with the large language model after the hub service is turned on.

[0014] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch function, the message list of this round includes: an information list recording the interaction between the hub service and the large language model in this round of conversation and a pending sketch list, and the pending sketch list includes at least one sketch structure that has not been clicked by the user or inferred by the large language model.

[0015] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch functions, in S1, the hub service registers the function toolset with the large language model. Specifically, the hub service traverses each function name in the function toolset, encapsulates the function description specification corresponding to each function name into a data format recognized by the large language model, and submits the function description specifications corresponding to all function names to the large language model through the large language model client, and corrects the function toolset when the large language model feedbacks error information.

[0016] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch function, S3 is specifically:

[0017] When the natural language result obtained by the hub service from the large language model reasoning of the current round of message list is to complete the user input task, the reasoning result of completing the user input task is forwarded to the rendering engine, and the rendering engine is used to reversely display the reasoning result of completing the user input task on the user interface;

[0018] When the natural language result obtained by the hub service from receiving the large language model reasoning of the message list in this round is an unfinished user input task, the objective function is called according to the reasoning result of the large language model.

[0019] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch functions, in S3, the hub service calls the target function according to the inference results of the large language model. Specifically, when the inference results of the large language model include the target function and its calling parameters, the hub service adds the inference results of the large language model to the message list of this round, extracts the target function from the function tool set, and executes the target function according to the target function call parameters inferred by the large language model.

[0020] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch function, S4 is specifically as follows:

[0021] When the target function execution result is a traditional function return value type, the hub service adds the traditional function execution result to the current round message list and forwards the current round message list to the large language model;

[0022] When the execution result of the target function is the return value type of the sketch function, the hub service converts the execution result of the sketch function into a sketch structure, adds the sketch structure to the pending sketch list in the current round of message list, and submits the sketch structure to the rendering engine. The rendering engine is used to generate the target user interface component on the user side according to the received sketch structure, update the pending sketch list in the current round of message list according to the interaction result of the user clicking the target user interface component, and forward the current round of message list to the large language model.

[0023] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch functions, when the target function execution result is a traditional function return value type, the traditional function return value is obtained by calling other third-party services.

[0024] As a further optimization scheme of the dynamic interactive user interface generation method based on sketch function, the pending sketch list in the current round message list is updated according to the interaction result of the user clicking the target user interface component. Specifically: in the current round message list, find the pending sketch structure that matches the identifier of the target user interface component selected by the user, and update the matching pending sketch structure to the interaction result of the user clicking the target user interface component.

[0025] The present invention adopts the above technical solution and has the following beneficial effects: compared with the prior art,

[0026] (1) The present invention introduces two key concepts, sketch function and hub service, to achieve an innovative method for dynamically generating an interactive user interface during a generative AI dialogue.

[0027] (2) The present invention is used in the process of dialogue with generative AI. By calling the sketch function, the interface components that can be clicked by the user are dynamically generated on demand, and the clicking action is used instead of text or voice input, which simplifies the user operation, protects the user privacy, and improves the consistency of the user experience.

[0028] (3) The sketch function proposed in the present invention can generate a structured description of an interactive user interface, namely a sketch structure, so that the user end can dynamically create clickable interface components; in addition, the sketch can be parsed and rendered by various UI frameworks, thereby enhancing the cross-platform compatibility of the present invention.

[0029] (4) The hub service proposed in the present invention unifies the calling interfaces of sketch functions and traditional functions based on the functional programming paradigm and manages the function tool set in a consistent manner, thereby reducing the calling complexity of function callers and improving the maintainability of the code; by creating and maintaining message lists, the context information management of multiple rounds of dialogue with generative AI is realized, ensuring the integrity and timeliness of context information during multiple rounds of interaction, and providing a more complete and accurate basis for the reasoning process of the large language model; by defining a common internal interface specification and implementing the interface specification in multiple large language model communication clients, compatibility with multiple large language models is achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is a flow chart of the method for generating a dynamic interactive user interface based on sketch functions proposed in the present invention.

[0031] Figure 2 The figure is a schematic diagram of information interaction between the server and the user in the dynamic interactive user interface generation method proposed in the present invention. DETAILED DESCRIPTION

[0032] The technical solution of the invention is described in detail below with reference to the accompanying drawings.

[0033] The method for generating a dynamic interactive user interface based on a sketch function proposed in the present invention is as follows: Figure 2 As shown in FIG. 1 , the hub service deployed on the server is used to implement multiple rounds of interactive conversations with the user. The hub service defines a function tool set, management logic for implementing communication with the large language model, and management logic for implementing the conversation context with the large language model.

[0034] The functions in the function tool set include traditional functions and the sketch functions customized by the present invention. The difference between traditional functions and sketch functions is that the return result of traditional functions can be directly submitted to the large language model for the large language model to continue the reasoning process; while the return result type of the sketch function is a sketch, which needs to be submitted to the rendering engine and converted into a clickable interactive interface element. After obtaining the user interaction result, the user interaction result is submitted to the large language model to continue the large language model reasoning process.

[0035] Defining a sketch function includes function name, input parameters, processing logic and output sketch structure. The sketch structure is a structured description of the user interface represented by the sketch structure obtained by processing the input parameters through the processing logic, which is used to instruct the rendering engine to generate the corresponding interface component. The present invention exemplarily defines the sketch structure through a three-element array including a tool call ID, a target user interface component name, and a sketch function call parameter list, which is a parameter list required for rendering the target user interface component.

[0036] The definition of a traditional function includes the function name, input parameters, processing logic and output results. The output result is obtained by processing the input parameters through the processing logic to obtain a structured description that does not involve user interface interaction text.

[0037] In order to reduce the calling complexity of function callers and improve the maintainability of the code, the present invention adopts the concept of coproduct in category theory to unify the representation of traditional functions and sketch functions, so that traditional functions and sketch functions can be processed uniformly under the same framework, and can provide a consistent function calling interface to external callers. Specifically, a new data structure consisting of two different types of parameters, supporting two morphism operations of left injection and right injection, and conforming to the coproduct characteristic is used. The first type of parameter of the two different types of parameters represents the return value type of the traditional function, and the second type of parameter represents the return value type of the sketch function. The function tool set is unified into a mapping table from function name to function implementation and function description specification, and the function description specification is described in natural language so that the large language model can accurately understand the purpose of the function when it receives the function tool set registration information of the hub service.

[0038] Hub services can be developed using any framework that has RPC (Remote Procedure Call) server capabilities. RPC server types include but are not limited to: local or remote HTTP services, WebSocket services, gRPC services, etc.

[0039] The communication management logic is implemented in the hub service, and the same communication protocol as the large language model is adopted to build a large language model client that communicates with the large language model to establish a connection between the hub service and the large language model, and complete the encoding of the request message and the decoding of the response message. In the case where the hub service is connected to multiple large language models, it is necessary to build multiple large language model communication clients. The hub service in the present invention defines a common internal interface specification, and implements the interface specification in each large language model communication client, thereby achieving unified management of multiple large language model communication clients.

[0040] In order to realize a single intention expressed by a user through natural language, the large language model may need to interact with the user for multiple rounds. Therefore, the present invention creates and maintains a message list in the hub service to manage the context information of multiple rounds of interaction to implement the management logic of the conversation context with the large language model. The context information includes a message list and a pending sketch list for multiple rounds of interaction with the large language model in the current session. Each message in the message list includes a message ID, a message role, and a message content. The message role is used to identify the purpose of the interaction. Among them, the system message is used to provide system prompt words to the large language model, the user message is used to provide user input to the large language model, the assistant message is used to display the reasoning or execution results of the large language model, and the tool message is used to provide the execution results of the tool function to the large language model. The present invention determines whether the session is over by judging whether a non-empty assistant message from the large language model is received. When the hub service receives the user's intention expressed in natural language, it converts it into a user message, adds it to the message list of the current conversation context, and then submits the entire message list to the large language model through the specified large language model communication client, thereby achieving effective management of the conversation context and ensuring the integrity and timeliness of multi-round interaction context information.

[0041] In terms of deployment methods, the deployment methods of hub services include but are not limited to: local deployment, physical server deployment, containerized deployment, serverless deployment and many other methods.

[0042] Start the hub service on the server to ensure that the hub service can respond to the call requests initiated by the large language model at any time, execute the corresponding function operations, and coordinate the presentation of the rendering results.

[0043] The method for generating a dynamic interactive user interface based on a sketch function is as follows: Figure 1 As shown, after the hub service is started, the hub service executes S1 to S4.

[0044] S1: Establish a connection with the large language model and register the function tool set to the large language model.

[0045] After completing the connection between the hub service and the big language model, the hub service will register all the functions in the defined function tool set with the big language model. Specifically, the hub service will traverse each function in the function tool set, convert the function description specification of each function into a format that the big language model can understand, usually in JSON format, and submit it to the big language model through the registration interface of the big language model communication client. If the big language model returns an error message, you need to troubleshoot according to the error message and correct the error in the function tool set.

[0046] S2: Receive the user input task of this round of dialogue, initialize the message list of this round, and forward the message list of this round to the large language model.

[0047] Since the hub service has registered all functions in the function tool set with the large language model in S1, the large language model now has the ability to infer based on user input and decide whether to call a function in the function tool set. After receiving the user input, the hub service generates a message list for this round, and the large language model communication client transmits the message list for this round to the large language model. The large language model infers whether the sketch function or the traditional function needs to be called based on the registered function tool set. If a call is required, the large language model converts the function call instruction into an assistant message containing the target function, and the assistant message is returned to the hub service via the large language model communication client.

[0048] S3: Receive the natural language results obtained by the large language model inference of the current round of message list, and echo the inference results of the large language model to the user when the user input task is completed. When the user input task is not completed, call the target function according to the inference results of the large language model. If the assistant message currently received by the hub service is a session termination message, the hub service will clear the current session context and send the assistant message to the rendering engine, which will be reversed on the user interface to prompt the user that the current session has been completed; if the assistant message currently received by the hub service contains the target function and its calling parameters, indicating that the current session has not ended, the hub service will add the assistant message to the message list in the current session context, extract the corresponding function from the function tool set according to the function name of the target function, and execute the function using the calling parameters specified by the large language model.

[0049] S4: Get the target function execution result, add the target function execution result to the message list of this round, forward the message list of this round to the large language model, and return to S3. The target function execution result includes the sketch function execution result and the traditional function execution result. The sketch function execution result includes the user interface component identifier and the interaction result of the user clicking the target user interface component. The traditional function execution result includes the execution result of the user input task.

[0050] S4.1: The hub service checks the return value type of the target function. If it is a left injection type, indicating that the target function is a traditional function, the hub service converts the call result of the target function into a tool message, adds the tool message to the message list of the current session context, and submits the message list to the large language model through the specified large language model communication client, and the large language model continues the reasoning process. For example, the hub service executes a traditional function, calls other third-party services to complete tasks such as flight information query and ticket booking, and other third-party services return the task execution results to the hub service after completing the requested task. The hub service converts the task execution results into a tool message, adds it to the current round message list, and transmits the current round message list to the large language model. The large language model continues to reason. If the large language model reasoning believes that the user input task has been completed, the reasoning result is encapsulated into an assistant message and fed back to the hub service. The hub service checks that the assistant message is a session termination message, clears the current session context, and sends the assistant message to the rendering engine, which is displayed on the user interface to prompt the user that the current task has been completed.

[0051] S4.2: The hub service checks the return value type of the target function. If it is a right injection type, it means that the target function is a sketch function. The hub service converts the call result of the target function into a sketch structure, adds the sketch structure to the pending sketch list of the current session context, and submits the sketch structure to the rendering engine. The rendering engine renders the target user interface component according to the component name and parameter list in the sketch structure, and displays the target user interface component on the interface. After the user interacts with the user interface component, the identifier of the user interface component and the interaction result of the user's click operation will be submitted to the hub service. The hub service checks the message list of the current context to find out whether there is a sketch structure that matches the identifier of the user interface component. If a match is found, it means that the corresponding sketch function has been executed. The hub service updates the matching pending sketch structure to the interaction result of the user's click operation on the target user interface component, encapsulates the interaction result of the user's click operation on the target user interface component and the left injection type into a tool message, and adds the tool message as the target function execution result to the message list of the current session context.

[0052] The target UI component corresponding to each UI component name can be abstracted into a function that receives the parameter list in the sketch structure as input and outputs a UI component type data. UI component is an abstract concept that can be implemented using any framework with UI rendering capabilities, including but not limited to: Web frameworks such as React, Vue, Angular, or mobile frameworks such as Flutter, SwiftUI, Android View, etc.

[0053] The above shows and describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above specific embodiments. The above specific embodiments and the description in the specification are only for further illustrating the principles and preparation effects of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of the present invention is defined by the claims and their equivalents.

Claims

1. A method for generating a dynamic interactive user interface based on a sketch function, characterized in that: This is achieved through an interactive dialogue between a hub service deployed on the server and the user. The hub service defines a function tool set, management logic for communicating with a large language model, and management logic for implementing a dialogue context with a large language model. The function tool set includes but is not limited to sketch functions and traditional functions. After the hub service is started, the hub service performs the following steps: S1: Establish a connection with the large language model and register the function tool set to the large language model; S2: Receive the user input task of this round of dialogue, initialize the message list of this round, and forward the message list of this round to the large language model; S3: Receive the natural language results obtained by the large language model inference of the message list in this round, and echo the inference results of the large language model to the user when the user input task is completed. When the user input task is not completed, call the objective function according to the inference results of the large language model; S4: Obtain the target function execution result, add the target function execution result to the message list of this round, forward the message list of this round to the large language model, and return to S3. The target function execution result includes but is not limited to the sketch function execution result and the traditional function execution result. The sketch function execution result includes but is not limited to the user interface component identifier and the interaction result of the user clicking the target user interface component. The traditional function execution result includes but is not limited to the execution result of the user input task.

2. The method for generating a dynamic interactive user interface based on a sketch function according to claim 1, characterized in that: The sketch function is to obtain a structured description of the user interface component represented by a sketch structure after processing input parameters through processing logic; The traditional function is a structured description of obtaining irrelevant user interface interaction text after processing input parameters through processing logic; The sketch function and traditional function are called through a calling interface that unifies the sketch function return value type and the traditional function return value type. The calling interface maps the function toolset into a mapping table from function name to function implementation and function description specification. The function description specification is described in natural language.

3. The method for generating a dynamic interactive user interface based on a sketch function according to claim 2, characterized in that: The hub service defines management logic for implementing communication with the large language model, specifically: using the same communication protocol as the large language model, building a large language model communication client that communicates with the large language model, and the large language model communication client establishes a connection with the large language model after the hub service is started.

4. The method for generating a dynamic interactive user interface based on a sketch function according to claim 3, characterized in that: The message list of this round includes: an information list recording the interaction between the hub service and the large language model in this round of conversation and a pending sketch list, wherein the pending sketch list includes at least one sketch structure that has not been clicked by the user or inferred by the large language model.

5. The method for generating a dynamic interactive user interface based on a sketch function according to claim 4, characterized in that: In S1, the hub service registers the function toolset with the large language model as follows: the hub service traverses each function name in the function toolset, encapsulates the function description specification corresponding to each function name into a data format recognized by the large language model, submits the function description specifications corresponding to all function names to the large language model through the large language model client, and corrects the function toolset when the large language model feeds back error information.

6. The method for generating a dynamic interactive user interface based on a sketch function according to claim 5, characterized in that: The S3 is specifically: When the natural language result obtained by the hub service from the large language model reasoning of the current round of message list is to complete the user input task, the reasoning result of completing the user input task is forwarded to the rendering engine, and the rendering engine is used to reversely display the reasoning result of completing the user input task on the user interface; When the natural language result obtained by the hub service from receiving the large language model reasoning of the message list in this round is an unfinished user input task, the objective function is called according to the reasoning result of the large language model.

7. The method for generating a dynamic interactive user interface based on a sketch function according to claim 6, characterized in that: In S3, the hub service calls the target function according to the inference result of the large language model. Specifically, when the inference result of the large language model includes the target function and its calling parameters, the hub service adds the inference result of the large language model to the message list of this round, extracts the target function from the function tool set, and executes the target function according to the target function calling parameters inferred by the large language model.

8. The method for generating a dynamic interactive user interface based on a sketch function according to claim 7, characterized in that: The S4 is specifically: When the target function execution result is a traditional function return value type, the hub service adds the traditional function execution result to the current round message list and forwards the current round message list to the large language model; When the execution result of the target function is the return value type of the sketch function, the hub service converts the execution result of the sketch function into a sketch structure, adds the sketch structure to the pending sketch list in the current round message list, and submits the sketch structure to the rendering engine. The rendering engine is used to generate a target user interface component on the user side according to the received sketch structure, update the pending sketch list in the current round message list according to the interaction result of the user clicking the target user interface component, and forward the current round message list to the large language model.

9. The method for generating a dynamic interactive user interface based on a sketch function according to claim 8, characterized in that: When the target function execution result is a traditional function return value type, the traditional function return value is obtained by calling other third-party services.

10. The method for generating a dynamic interactive user interface based on a sketch function according to claim 8, characterized in that: The method of updating the pending sketch list in the current round message list according to the interaction result of the user clicking the target user interface component is specifically as follows: in the current round message list, searching for the pending sketch structure that matches the identifier of the target user interface component selected by the user, and updating the matching pending sketch structure to the interaction result of the user clicking the target user interface component.

Images