AI-driven UI generation and interaction methods, systems, and devices

Abstract

This disclosure provides an AI-driven user interface (UI) generation method. The method includes: responding to a received service request containing user natural language requirements and capability constraints; parsing the service request to obtain a request context, the request context containing semantic parsing results of the user's natural language requirements and capability constraints; invoking an AI agent based on the semantic parsing results; assembling a runtime constraint model according to the capability constraints, the runtime constraint model being a model used to constrain the output range of the AI ​​agent; generating a UI message through the AI ​​agent based on the runtime constraint model; performing first and second checks on the UI message; and splitting the UI message that passes the checks into first and second data streams, outputting the first and second data streams through at least two parallel output channels. The first data stream is used for front-end display of natural language response content for the natural language requirements, and the second data stream is used for front-end rendering of the UI interface.

Description

Technical Field

[0001] This disclosure relates to the field of storage technology, specifically to an AI-driven UI generation and interaction method and system, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the rapid development of artificial intelligence technology, especially the continuous maturation of large language model technology, generative artificial intelligence (AI) has been widely applied in many fields. In the field of user interaction, generating user interfaces (UIs) using natural language has become an emerging technological trend. Traditional UI development processes typically require designers to create the interface, and then developers to convert the design drafts into executable code. The entire process is time-consuming, labor-intensive, and difficult to respond quickly to users' personalized needs.

[0003] Currently, UI generation solutions based on large models mainly include generating natural language descriptions of the UI, generating UI design draft images, and directly generating front-end code. These solutions have improved UI development efficiency to some extent, but there are still many areas for improvement in practical applications. As AI-driven application scenarios become increasingly diverse, higher demands are being placed on the accuracy, consistency, and interactivity of UI generation. Summary of the Invention

[0004] This disclosure provides an AI-driven UI interaction method and system, electronic device, computer-readable storage medium, and computer program product.

[0005] According to one aspect of this disclosure, an AI-driven user interface (UI) generation method is provided, executed by a server. The method includes: responding to receiving a service request containing user natural language needs and capability constraint information; parsing the service request to obtain a request context, the request context containing semantic parsing results of the user's natural language needs and capability constraint information; invoking an AI agent based on the semantic parsing results; assembling a runtime constraint model according to the capability constraint information, the runtime constraint model being a model used to constrain the output range of the AI ​​agent; generating a UI message through the AI ​​agent based on the runtime constraint model; performing a first verification and a second verification on the UI message; and splitting the verified UI message into a first data stream and a second data stream, outputting the first data stream and the second data stream through at least two parallel output channels, wherein the first data stream is used for front-end display of natural language response content for the natural language needs, and the second data stream is used for front-end rendering of the UI interface.

[0006] According to one aspect of this disclosure, an AI-driven user interface (UI) interaction method is provided, executed by a user terminal. The method includes: receiving a first data stream and a second data stream output by a server after processing by an AI agent, wherein the first data stream is used to display natural language response content and the second data stream is used to render the UI interface; maintaining the creation and state of the UI instance based on the unique identifier of the UI instance, wherein the UI instance is created based on the second data stream; and completing the rendering of the UI interface based on the second data stream, wherein the rendering result of the UI interface corresponds to the state of the UI instance.

[0007] According to one aspect of this disclosure, an AI-driven user interface (UI) generation system is provided, deployed on a server. The system includes: a request parsing module configured to respond to a received service request containing user natural language requirements and capability constraint information, parse the service request to obtain a request context, the request context containing semantic parsing results of the user's natural language requirements and capability constraint information; an AI agent invocation module configured to invoke an AI agent based on the semantic parsing results; a constraint model assembly module configured to assemble a runtime constraint model according to the capability constraint information, the runtime constraint model being a model used to constrain the output range of the AI ​​agent; a UI message generation module configured to generate UI messages through the AI ​​agent based on the runtime constraint model; a message verification module configured to perform a first verification and a second verification on the UI messages; and a multi-channel output module configured to split the verified UI messages into a first data stream and a second data stream, and output the first data stream and the second data stream through at least two parallel output channels, wherein the first data stream is used for front-end display of natural language response content for natural language requirements, and the second data stream is used for front-end rendering of the UI interface.

[0008] According to one aspect of this disclosure, an AI-driven user interface (UI) interaction system is provided, deployed on a user terminal. The system includes: a data receiving module configured to receive a first data stream and a second data stream output by a server after processing by an AI agent, wherein the first data stream is used to display natural language response content and the second data stream is used to render the UI interface; a UI state management module configured to maintain the creation and state of UI instances based on the unique identifier of the UI instance, wherein the UI instance is created based on the second data stream; and a UI rendering module configured to complete the rendering of the UI interface based on the second data stream, wherein the rendering result of the UI interface corresponds to the state of the UI instance.

[0009] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0010] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0011] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0012] By utilizing one or more embodiments provided in this disclosure, through runtime constraint model dynamic assembly, structure and state dual verification mechanism and dual-channel parallel output technology, AI-driven standardized interactive UI generation can be realized, effectively solving the problem of uncontrollable output of large models, reducing the cost of repeated development for multi-service access, while ensuring low vulnerability of front-end links and natural language interaction experience, and taking into account the high reliability of the system and the ability to quickly replicate and promote across services.

[0013] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0014] The accompanying drawings exemplify embodiments and form part of the specification, serving together with the textual description to explain exemplary implementations of the embodiments. The illustrated embodiments are for illustrative purposes only and do not limit the scope of the claims. Throughout the drawings, the same reference numerals refer to similar but not necessarily identical elements.

[0015] Figure 1 This is a flowchart illustrating the AI-driven UI generation method 100 in some embodiments of this disclosure.

[0016] Figure 2 This is a flowchart illustrating the AI-driven UI interaction method 200 in some embodiments of this disclosure.

[0017] Figure 3 This is a schematic diagram of the AI-driven UI generation and interaction process 300 in some embodiments of this disclosure.

[0018] Figures 4A to 4C The diagram illustrates the application flow of an AI-driven UI generation and interaction method according to some embodiments of the present disclosure in a form generation and submission scenario.

[0019] Figure 5 , Figure 6 and Figure 7 Several application examples of the AI-driven UI generation and interaction method according to some embodiments of the present disclosure in the data visualization chart generation scenario are shown.

[0020] Figure 8 This is a schematic diagram of the AI-driven UI generation and interaction system architecture process 800 in some embodiments of this disclosure.

[0021] Figure 9 This is a schematic diagram of an AI-driven UI generation system 900 according to some embodiments of this disclosure.

[0022] Figure 10 This is a schematic diagram of an AI-driven UI interaction system 1000 according to some embodiments of this disclosure.

[0023] Figure 11 A structural block diagram of an exemplary electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0024] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0025] In this disclosure, unless otherwise stated, the use of terms such as "first," "second," etc., to describe various elements is not intended to limit the positional, temporal, or importance relationships of these elements; such terms are merely used to distinguish one element from another. In some examples, the first element and the second element may refer to the same instance of that element, while in other cases, based on the context, they may refer to different instances.

[0026] The terminology used in the description of the various examples in this disclosure is for the purpose of describing particular examples only and is not intended to be limiting. Unless the context explicitly indicates otherwise, an element may be one or more unless the number of elements is specifically limited. Furthermore, the term "and / or" as used in this disclosure covers any one of the listed items and all possible combinations thereof.

[0027] It should be noted that, in any part of this disclosure involving the collection, storage, use, transmission, and processing of data, each stage strictly adheres to the laws, regulations, industry standards, and regulatory requirements of the data source, usage location, and relevant countries and regions to ensure the legality and compliance of data activities. In the collection stage, the purpose, method, and scope of collection are clearly communicated to the data subject in a prominent manner. Collection is conducted only after obtaining the data subject's legal authorization, ensuring that the collection process follows the "minimum necessary" principle and does not exceed the scope of data collection. In the storage stage, storage periods are limited, and data is promptly deleted or anonymized / encrypted after the storage purpose is achieved. In the usage stage, a strict data security protection mechanism is implemented, using field-level desensitization technology and processing the original data according to preset desensitization rules. For different types of data, multiple desensitization strategies, such as data generalization, data anonymization, and data encryption, are employed to effectively mitigate the risk of sensitive information leakage and ensure that all data used is securely processed and desensitized, comprehensively protecting the rights and interests of data subjects and data security. In the transmission and processing stages, the confidentiality and security of data are ensured during transmission and processing.

[0028] As mentioned earlier, with the rapid development of artificial intelligence technology, especially the continuous maturation of large language model technology, generative artificial intelligence (AI) has been widely applied in many fields. In the field of user interaction, using natural language to generate user interfaces (UIs) has become an emerging technological trend. Traditional UI development processes usually require designers to design the interface, and then developers to convert the design drafts into executable code. The entire process is time-consuming, labor-intensive, and difficult to respond quickly to users' personalized needs.

[0029] Currently, UI generation solutions based on large models mainly include generating natural language descriptions of the UI, generating UI design draft images, and directly generating front-end code. These solutions have improved UI development efficiency to some extent, but there are still many areas for improvement in practical applications. As AI-driven application scenarios become increasingly diverse, higher demands are being placed on the accuracy, consistency, and interactivity of UI generation.

[0030] UI generation solutions based on large models generally lack effective runtime constraint mechanisms. The output of large models has inherent randomness and uncontrollability, which can easily generate component types or interactive actions that exceed the scope allowed by the business, leading to system malfunctions. At the same time, different businesses usually develop their own UI generation logic independently, without a unified capability abstraction and sharing mechanism. As a result, different businesses gradually develop incompatible protocol dialects, which significantly increases the repetitive development costs of multi-business access and is also not conducive to unified system maintenance and upgrades.

[0031] These solutions have significant drawbacks in their output methods. Most employ a single data stream that mixes natural language and UI data, requiring the front-end to perform complex semantic parsing and format conversion on the natural language text to extract UI information. This not only significantly increases the complexity of front-end development but also makes rendering prone to errors due to parsing, resulting in high fragility throughout the data transmission and processing chain. Furthermore, while some solutions can generate UI structures, they lack robust layered validation mechanisms, failing to simultaneously guarantee the format validity of UI messages and the consistency of business states. This leads to frequent issues such as state confusion, duplicate rendering, and misaligned updates in multi-round interaction scenarios.

[0032] These solutions generally have low levels of standardization, with a lack of unified specifications for output formats and service interfaces. This makes it difficult to reuse and migrate UI capabilities across different businesses, and fails to support a long-term evolution path that combines the stability of basic protocols with the expansion of business capabilities. At the same time, most of these solutions limit AI capabilities to a purely conversational output format, failing to elevate them to the engineering level of generating structured and interactive UIs. This restricts the diversity of interaction methods in AI applications and prolongs the user's completion path for complex tasks.

[0033] To address the aforementioned technical issues, this disclosure provides an AI-driven UI generation and interaction method and system.

[0034] The present invention will now be described in further detail with reference to the accompanying drawings.

[0035] Figure 1 This is a flowchart illustrating an AI-driven UI generation method 100 according to some embodiments of this disclosure. This method 100 can be executed by a server to generate UI-related data conforming to specifications based on the user's natural language requirements and output it to the front end. For example... Figure 1 As shown, method 100 includes step S110: in response to receiving a service request containing user natural language needs and capability constraint information, parsing the service request to obtain a request context; step S120: based on the semantic parsing result contained in the request context, invoking an AI agent; step S130: assembling a runtime constraint model according to the capability constraint information, the runtime constraint model being a model used to constrain the output range of the AI ​​agent; step S140: generating a UI message through the AI ​​agent based on the runtime constraint model; step S150: performing a first verification and a second verification on the UI message; step S160: based on the verified UI message, splitting it into a first data stream and a second data stream, so as to output the first data stream and the second data stream through at least two parallel output channels, the first data stream can be used for front-end display of natural language response content for natural language needs, and the second data stream can be used for front-end rendering of the UI interface.

[0036] In step S110, in response to receiving a service request containing user natural language requirements and capability constraint information, the service request is parsed to obtain a request context. In some implementations, the service request may be sent by a business agent to a server (e.g., Google's A2UI service) for processing. As an example, and not a limitation, the parameters included in the service request may include natural language messages, thread_id, biz_code, agent_id, component_pack_ids, component_whitelist, and action_whitelist. The request context specifically includes the semantic parsing result of the user's natural language requirements and capability constraint information. The semantic parsing result is used for subsequent calls to the AI ​​agent, and the capability constraint information can be used to limit the types of UI components and interaction actions allowed to be generated in this request, thereby preventing the model from outputting content beyond the business scope. In the example, different businesses can achieve capability isolation by setting different component_whitelists and action_whitelists.

[0037] In step S120, the AI ​​agent is invoked based on the semantic parsing result contained in the request context. In some implementations, the semantic parsing result is obtained by the server after semantically decomposing and identifying the user's natural language requirements. Invoking the corresponding AI agent based on this accurate semantic parsing result can ensure that the AI ​​agent accurately matches the user's UI generation requirements, avoids aimless model calls, and improves the targeting and accuracy of UI generation.

[0038] In step S130, a runtime constraint model is assembled based on capability constraint information. As used herein, when referring to a runtime constraint model, it can refer to a component and action constraint model dynamically assembled according to the business context. This model is specifically designed to constrain the output range of the AI ​​agent and limit the boundaries of available capabilities for UI generation. In some implementations, the process of assembling the runtime constraint model may include: first loading a preset basic component package, then loading the corresponding business component package according to the component_pack_ids in the request context, merging the basic component package and the business component package, and then trimming the merged capability set according to the component_whitelist and action_whitelist in the capability constraint information. It is easy to understand that by dynamically assembling the runtime constraint model, the same service can support the different capability requirements of multiple services, while ensuring capability isolation between services, and the entire assembly process relies on capability constraint information, aligning with the overall technical logic.

[0039] In step S140, UI messages are generated by an AI agent based on a runtime constraint model. In some implementations, a large model guidance prompt containing descriptions of available components and actions can first be generated based on the runtime constraint model. Then, the user's natural language requirements and the guidance prompt are input into the large model corresponding to the AI ​​agent, and the UI message generation process is completed by the AI ​​agent. Additionally or alternatively, the UI messages generated by the AI ​​agent can conform to the specifications of a standardized UI description protocol to ensure that the front end can uniformly parse and process them. Here, the UI message is an intermediate message containing natural language response content and structured data required for UI rendering, not a directly output data stream.

[0040] In step S150, a first verification and a second verification are performed on the UI message. In some implementations, the first verification may be a structural validity verification, which can be used to verify the format specifications, field completeness, and data type correctness of the UI message; the second verification may be a state consistency verification, which can be used to verify the validity of the unique identifier of the UI instance contained in the UI message, the legality of state transitions, the existence of component references, and the integrity of component hierarchical reference relationships. It is understood that, due to the uncertainty of the output of the large model corresponding to the AI ​​agent, the dual verification mechanism can effectively filter UI messages that do not conform to the specifications, thereby improving the stability of the system. In the example, if any verification fails, the server can generate an error message and terminate the current operation, while also supporting the frontend to re-initiate the request.

[0041] In step S160, based on the validated UI message, a first data stream and a second data stream are split, and output through at least two parallel output channels. In some implementations, the first data stream may be a text stream, which can be used by the front end to progressively display natural language response content tailored to the user's natural language needs; the second data stream may be a structured data stream, which may contain complete UI structure definitions and state data, for the front end to directly parse and render the UI interface. As an example and not a limitation, the server can implement dual-channel output through SSE (Server-Sent Events) technology. Additionally, the server can also output control signals concurrently with the first and second data streams, which can be used to manage message reception, processing, and termination operations, such as outputting completion, interruption, cancellation, and error control signals.

[0042] It should be noted that the specific implementation forms of the above embodiments of this disclosure, which use the business agent and the service adopting the A2UI protocol as the execution subjects, are merely illustrative examples and do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art can adopt other suitable system architectures or functional module division methods to achieve the same or equivalent functions according to actual application needs. For example, it can be replaced with different AI-UI interaction systems such as the AG-UI protocol proposed by CopilotKit, the MCP-UI related protocol of Anthropic, or the UI interaction protocol in the OpenAI ecosystem. A service architecture with a custom private protocol can also be adopted. All of these do not depart from the technical concept of this disclosure, and this disclosure does not impose any limitations on them.

[0043] The AI-driven UI generation method provided in this disclosure enables multiple services to share the same service capabilities by dynamically assembling runtime constraint models, while ensuring capability isolation between services and avoiding the formation of different protocol dialects, thereby reducing the repetitive development costs of multi-service access. Through a dual verification mechanism of structural legality and state consistency verification, it effectively solves the problem of unstable output from large models, significantly improving the stability and reliability of the UI generation chain. The dual-channel output mechanism allows the front-end to directly consume structured UI data without relying on text parsing, reducing the vulnerability of the chain while retaining the display capabilities of natural language responses, balancing user experience and technical reliability. Furthermore, the standardized output format and unified service interface support a long-term evolution path combining basic protocol stability and business capability expansion, facilitating rapid cross-service replication and promotion. Ultimately, it elevates AI capabilities from pure dialogue output to an engineering capability capable of generating structured UIs, providing richer interaction methods for AI applications, effectively shortening the completion path of complex tasks, and improving user experience.

[0044] In some embodiments, method 100 further includes, in response to receiving interactive action data, parsing the interactive action data to obtain action type and action parameters; generating an incremental update message based on the parsed interactive action data; and outputting the incremental update message through at least two parallel output channels. In some implementations, when a user performs actions such as clicking, inputting, or selecting on the rendered UI interface, the user terminal can send interactive action data back to the server. As an example and not a limitation, this interactive action data may include an action type identifier, action parameters, and a unique identifier of the associated UI instance. After parsing this data, the server can generate an incremental update message containing only the changed portion of the UI interface, rather than the complete UI data. In the example, the incremental update message can be output through the same dual-channel mechanism as the initial UI message. This approach can significantly reduce data transmission volume, improve interaction response speed, and avoid the user experience degradation caused by a full card redraw.

[0045] In some embodiments, assembling a runtime constraint model based on the request context includes loading a preset first capability set, which contains general UI component definitions and general interaction action definitions; loading a corresponding second capability set based on the request context, which may contain dedicated UI component definitions and dedicated interaction action definitions adapted to the semantic parsing results and capability constraint information; merging the second capability set and the first capability set; trimming the merged capability set according to the capability constraint information; and assembling the runtime constraint model based on the trimmed capability set. In some embodiments, the first capability set may contain basic components and interaction actions common to all businesses, such as buttons, input boxes, selectors, switches, etc.; the second capability set may contain extended components and interaction actions specific to a particular business domain, such as order cards, payment components, logistics status display components, etc. It is understood that by merging first and then trimming, the range of usable UI capabilities can be flexibly customized for different business scenarios, while ensuring the consistency of basic capabilities and avoiding different businesses forming their own protocol dialects.

[0046] In some embodiments, a first verification and a second verification are performed on the UI message, including performing a first verification on the UI message; performing a second verification on the UI message in response to a pass result of the first verification; generating an error message and terminating the current operation in response to a fail result of the first verification; generating an error message and terminating the current operation in response to a fail result of the second verification; and obtaining a valid UI message in response to a pass result of the second verification. In some embodiments, this layered verification mechanism can quickly filter out messages that obviously do not meet the format requirements, and then perform a more in-depth state consistency check on messages that pass the initial verification. It is easy to understand that this approach can improve verification efficiency and reduce unnecessary computational resource consumption while ensuring comprehensive verification. Additionally or alternatively, when verification fails, the server can return standardized error messages to the front end so that the front end can perform appropriate error handling and user prompts.

[0047] In some embodiments, based on a validated UI message, a first data stream and a second data stream are output through at least two parallel output channels. This includes outputting the first data stream through the first output channel; outputting the second data stream through the second output channel, the second data stream containing complete UI structure definitions and state data; and outputting control signals concurrently with the first and second data streams. These control signals are used to manage message reception, processing, and termination operations. In some embodiments, the first and second data streams can be transmitted and processed independently, and the front-end can choose to display them simultaneously or separately as needed. As an example and not a limitation, the first data stream can be used to display natural language response content word by word, and the second data stream can be used to render the entire UI interface at once. The control signals can be used to coordinate the message transmission process between the front-end and back-end, ensuring the reliability and orderliness of data transmission.

[0048] In some embodiments, the merged capability set is pruned based on capability constraint information, including extracting component constraints and action constraints from the capability constraint information; removing component capabilities that do not meet the conditions in the merged capability set based on component constraints; removing interactive actions that do not meet the conditions in the merged capability set based on action constraints; completing the pruning of the merged capability set through the above two removal steps to detect whether there are unknown components or unknown actions in the pruned capability set; in response to the detection result indicating the presence of unknown components or unknown actions, an error message is generated and the current operation is terminated. In some embodiments, component constraints and action constraints are usually in the form of whitelists, and only components and actions in the whitelist are retained. It is understood that by detecting unknown components and actions and failing quickly, it is possible to prevent large models from generating content beyond the scope of business, ensuring the security and stability of the system. In the example, different businesses can achieve capability isolation by setting different whitelists.

[0049] In some embodiments, a first validation is performed on the UI message, including at least one of the following: validating the UI message format; validating the integrity of UI message fields; and validating the correctness of UI message data types. In some implementations, format validation can be used to check whether the UI message conforms to a preset JSON Schema specification; field integrity validation can be used to check whether the UI message contains all required fields; and data type correctness validation can be used to check whether the data types of each field in the UI message are as expected. It is easy to understand that these validations ensure the basic structure of the UI message is correct, laying the foundation for subsequent processing and rendering.

[0050] In some embodiments, a second verification is performed on the UI message, including at least one of the following: verifying the validity of the unique identifier of the UI instance contained in the UI message; verifying the legality of state transitions in the UI message; verifying the existence of all components referenced in the UI message; and verifying the integrity of the hierarchical reference relationship of components in the UI message. In some implementations, the UI instance unique identifier validity verification can be used to check whether the identifier corresponds to an existing UI instance; the state transition legality verification can be used to check whether the current state transition conforms to preset state machine rules; the component existence verification can be used to check whether the components referenced in the UI message exist in the runtime constraint model; and the component hierarchical reference relationship integrity verification can be used to check whether the reference relationship between components is complete and free of circular references. It is understood that these verifications ensure that the UI message is correct in business logic, avoiding problems such as inconsistent states or rendering errors.

[0051] In some embodiments, control signals are output concurrently with the first data stream and the second data stream, including at least one of the following: outputting a first control signal indicating message transmission completion; outputting a second control signal indicating message transmission interruption; outputting a third control signal indicating message transmission cancellation; and outputting a fourth control signal indicating an error occurred during message transmission. In some embodiments, these control signals can help the front end understand the status of message transmission in a timely manner and take appropriate action. By way of example and not limitation, when a completion signal is received, the front end can close the loading animation; when an error signal is received, the front end can display an error message and provide a retry option.

[0052] In some embodiments, generating incremental update messages based on interaction action data includes parsing the interaction action data to determine the action type and action parameters; generating incremental update messages based on the action type and action parameters; wherein the incremental update messages include at least one of a first type of update message and a second type of update message, the first type of update message being used to update UI state data, and the second type of update message being used to update the UI component hierarchy structure. In some implementations, the first type of update message is typically used to modify data content in the UI interface, such as the values ​​of form fields, the status of list items, the progress of progress bars, etc.; the second type of update message is used to modify the structure of the UI interface, such as adding or deleting components, adjusting the layout of components, replacing entire component nodes, etc. It is understood that by distinguishing between different types of update messages, more refined partial UI updates can be achieved, further improving interaction response speed and user experience.

[0053] Figure 2This is a flowchart illustrating an AI-driven UI interaction method 200 in some embodiments of this disclosure. This method 200 can be executed by a user terminal to receive UI-related data output from a server and complete UI rendering and basic interactions. Figure 2 As shown, method 200 includes step S210 receiving a first data stream and a second data stream output by the server. The first data stream can be used to display natural language response content, and the second data stream can be used to render the UI interface. Step S220 maintaining the creation and state of the UI instance based on the unique identifier of the UI instance. Step S230 completing the rendering of the UI interface based on the second data stream.

[0054] In step S210, the first data stream and the second data stream output by the receiving server after processing by the AI ​​agent are received. In some implementations, the first data stream may be a text stream, which can be used for the front-end to progressively display natural language response content; the second data stream may be a structured data stream, which may contain complete UI structure definitions and state data. As an example and not a limitation, the user terminal can simultaneously receive the above two data streams through SSE (Server-Sent Events) technology. It is understood that by adopting a dual-channel data transmission method, the front-end can simultaneously obtain a natural language interactive experience and structured UI rendering capabilities without mixing different types of data in a single data stream.

[0055] In step S220, the creation and status of UI instances are maintained based on their unique identifiers. UI instances are created based on the second data stream. In some implementations, the unique identifier of a UI instance is generated and distributed by the server, serving as a unique credential to distinguish different UI interfaces. It is understood that managing UI instances through a unified identifier system establishes a state association between the front-end and back-end, ensuring accurate identification and operation of the corresponding UI interface during multi-round interactions.

[0056] In step S230, the UI interface is rendered based on the second data stream, and the rendering result corresponds to the state of the UI instance. In some embodiments, the second data stream contains all the structured information required to render the UI interface, and the front end can directly generate visual interface elements based on this information. It is easy to understand that rendering the UI interface directly based on structured data avoids the front end's complex parsing and conversion of natural language text, significantly improving the reliability and consistency of the rendering process.

[0057] The AI-driven UI interaction method provided in this disclosure effectively ensures the consistency of UI states in multiple rounds of interaction through a full-link management mechanism with unique UI instance identifiers, avoiding issues such as duplicate rendering and state loss. Rendering by directly consuming the structured UI data stream output from the server significantly reduces the parsing complexity and vulnerability of the front-end, improving the stability and response speed of UI rendering. Furthermore, the standardized data format and interaction process support unified access from different types of front-end devices, facilitating cross-platform deployment and promotion of the system.

[0058] In some embodiments, method 200 further includes displaying the natural language response content corresponding to the first data stream. In some implementations, the first data stream may be a text stream, supporting a progressive, word-by-word display of the natural language response content by the front end. As an example and not a limitation, while a large model is generating a response, the front end can display the generated text fragments in real time, without waiting for the complete response to be generated. In the example, the front end can choose to stream or display all text content at once as needed. It is understood that this streaming display method can significantly improve the user's interactive experience and reduce the user's perceived waiting time.

[0059] In some embodiments, method 200 further includes detecting user interaction actions on the rendered UI interface; performing traffic splitting based on the action type of the interaction action, the traffic splitting being used to distinguish between interaction actions that need to be processed locally and interaction actions that need to be sent back to the server for processing; receiving incremental update data generated and returned by the server based on the interaction action in response to the interaction action being sent back to the server; and performing UI updates (e.g., partial UI updates) based on the incremental update data. In some implementations, the front end can monitor all user interaction operations on the UI interface in real time and classify these operations according to preset rules. As an example and not a limitation, the front end component can distinguish between local processing and backend flow by action prefixes; actions prefixed with "frontend:" can be processed directly locally, while other actions need to be sent back to the server for processing. It is easy to understand that this action splitting mechanism can effectively reduce the server load while improving the response speed of the interaction.

[0060] In some embodiments, the creation and state of UI instances are maintained based on their unique identifiers. This includes receiving a globally unique UI instance identifier from the server in response to a request to generate a UI interface for the first time; binding the UI instance identifier to the newly created UI instance; storing the UI instance identifier and its state information; locating the corresponding UI instance in response to receiving the UI instance identifier in subsequent interactions; and performing an update operation on the UI instance based on the stored state information to maintain its state. In some implementations, the UI instance identifier is generated by the server and issued during the first rendering, serving as a unique credential to distinguish different UI interfaces. In the example, the server automatically generates a new identifier when the front-end has no identifier, and the presence of an identifier on the front-end is considered an update to an existing UI interface. It is understood that this identifier management mechanism can effectively avoid state confusion and update errors that occur during multiple rounds of interaction.

[0061] In some embodiments, UI rendering is performed based on a second data stream, including parsing the second data stream to obtain UI structure information and state data; constructing a UI component hierarchy based on the UI structure information; mapping each node in the UI component hierarchy to a corresponding rendering component based on the component type; binding the rendering component to state data and the rendering component to interactive actions via a component interface; and rendering the UI component hierarchy to generate a visual UI interface. In some implementations, the second data stream contains all the structured information required for rendering the UI interface, eliminating the need for additional text parsing or semantic understanding by the front end. As an example and not a limitation, the front-end renderer can automatically match the corresponding wrapper component based on the node type and complete the binding of data and actions through a unified component interface. In the example, basic interactive elements such as buttons and input boxes can be automatically generated by the renderer. It is easy to understand that this rendering method based on structured data can significantly improve the accuracy and consistency of rendering.

[0062] In some embodiments, receiving the first and second data streams output by the server after processing by an AI agent further includes preprocessing the received data streams. This includes: deduplicating the received second data streams based on unique message identifiers; storing the updated data in a pending message set in response to receiving update data for UI instances that have not yet been created; and processing the updated data in the pending message set in the order of storage in response to the completion of the creation of the corresponding UI instance. In some implementations, each second data stream carries a unique message identifier, which the front end can use to identify and filter duplicate messages. Additionally or alternatively, due to the uncertainty of network transmission, update data may arrive before UI creation messages. The pending message set can effectively solve this out-of-order problem. In the example, if only data updates are received and the UI instance has not yet been created, the update is delayed in queuing and replayed after creation. It is understood that these mechanisms can significantly improve the reliability and stability of front-end data processing.

[0063] In some embodiments, traffic splitting is performed based on action type, including extracting the type identifier of the interaction action; extracting the action parameters of the interaction action, the action parameters being associated with the execution of the interaction action; determining the category to which the action belongs based on the type identifier; in response to the determination result being a first-category action, modifying the UI state data locally based on the action parameters; refreshing the UI interface based on the modified state data; and in response to the determination result being a second-category action, sending the interaction action and associated UI state data back to the server for processing. In some embodiments, first-category actions are typically pure front-end state operations that do not require interaction with the server; second-category actions involve business logic processing and must be executed by the server. As an example and not a limitation, operations such as modifying the value of a form field and toggling the state of a switch can be classified as first-category actions, while operations such as submitting a form and initiating a payment can be classified as second-category actions. It is easy to understand that this traffic splitting mechanism can maximize the response speed of the interaction while ensuring the correctness of the business logic.

[0064] In some embodiments, UI updates are performed based on incremental update data, including parsing the incremental update data returned by the server; determining the update type and update scope based on the incremental update data; updating the changed data items in the UI state model in response to a first-type update; updating the changed component nodes in the component-level UI structure in response to a second-type update; and refreshing the changed areas in the UI interface based on the updated state data or UI component hierarchy. In some implementations, the incremental update data only includes the changed parts of the UI interface, rather than the complete UI data. In the example, update messages are smaller than creation messages, and creation and update can be distinguished by the begin rendering flag. It is understood that this partial update mechanism can significantly reduce data transmission and rendering computation, improve the smoothness of interaction, and avoid flickering issues caused by full-page redraws.

[0065] In some embodiments, in response to a judgment result of a first type of action, the UI state data is modified locally, including at least one of the following: in response to a judgment result of a first subtype action, setting the value of a specified path in the UI state data according to the path information and value information in the action parameters; in response to a judgment result of a second subtype action, resetting the value of a specified path set in the UI state data according to the path set information in the action parameters; in response to a judgment result of a third subtype action, adding an element to a specified array path in the UI state data according to the array path information and value information in the action parameters; in response to a judgment result of a fourth subtype action, deleting a matching element from a specified array path in the UI state data according to the array path information, matching key information, and matching value information in the action parameters. In some implementations, these local actions cover most common UI state operation scenarios. As an example and not a limitation, setting the value of an input box can use the first subtype action, clearing the form can use the second subtype action, adding a list item can use the third subtype action, and deleting a list item can use the fourth subtype action. In the example, these actions are all handled directly by the front end without needing to be sent back to the back end. Understandably, by predefining these standard local actions, most simple interactions can be localized on the front end without interacting with the server.

[0066] In some embodiments, in response to a second type of update, the changed component nodes in the UI component hierarchy are updated, including parsing the second type of update data; determining the identifier and position of the changed component node; locating the corresponding original component node in the UI component hierarchy based on the component node's identifier; replacing the original component node with the new component node from the update data; and updating the reference relationships related to the component node in the UI component hierarchy. In some implementations, the second type of update message only contains information about the component nodes that need to be modified, and the front end does not need to rebuild the entire UI component hierarchy. As an example and not a limitation, when an item in the list needs to be replaced, the update message only contains the new data for that item, and the front end only needs to locate and replace the corresponding node. In the example, the front end only refreshes the changed parts in situ after receiving the message. It is easy to understand that this fine-grained structure update mechanism can significantly improve the efficiency of UI updates and reduce unnecessary rendering operations.

[0067] Figure 3 This is a schematic diagram of an AI-driven UI generation and interaction process 300 according to some embodiments of this disclosure. It is understood that this process 300 exemplarily demonstrates the entire chain from user input of natural language requirements to completion of UI interaction, encompassing the collaborative working mechanism between the front-end and back-end.

[0068] In step 301, the user inputs a natural language request. In some implementations, the user can input any natural language description through a dialog interface to express their operational intent or information retrieval needs. As an example, and not a limitation, the user can input natural language requests such as "Help me check my recent orders" or "I want to submit a reimbursement application."

[0069] In step 302, the business agent identifies whether a UI is needed. In some implementations, the business agent can perform semantic understanding of the user's natural language request to determine whether the request requires generating a visual UI to assist in completing the task. It is understood that for simple text-based question-and-answer requests, the business agent can directly return a text response; for requests requiring complex interaction or data display, a corresponding UI needs to be generated.

[0070] In step 303, the A2UI service is invoked, carrying the business whitelist. In some implementations, when a business agent determines that UI needs to be generated, it can initiate a call request to the A2UI service, carrying the whitelist information corresponding to the business in the request. As an example and not a limitation, the whitelist information may include a component whitelist and an action whitelist, used to limit the types of UI components and interaction actions allowed to be generated in this request.

[0071] In step 304, it is determined whether a surfaceId exists. This determination node is used to distinguish whether a new UI interface is being created or an existing UI interface is being updated.

[0072] In branch 304-1, if the surfaceId does not exist, step 305 is executed to create the UI. In some implementations, when a user makes a new request and there is no associated UI, the server will create a completely new UI instance and generate a globally unique surfaceId.

[0073] In branch 304-2, if a surfaceId exists, step 306 is executed to update the UI. In some implementations, when a user interacts with an existing UI and triggers a new request, the front-end sends the current UI's surfaceId along with the request to the server. The server then uses this surfaceId to locate the corresponding UI instance and performs the update operation.

[0074] In step 307, the server generates and validates the A2UI message. In some implementations, the server assembles a runtime constraint model based on a business whitelist, and then generates a compliant A2UI message based on this model. After generation, the server performs dual validation on the A2UI message: the first validation is a structural validity check, verifying the correctness of the message format, field integrity, and data type; the second validation is a state consistency check, verifying the validity of the surfaceId, the legality of state transitions, the existence of component references, and the integrity of component hierarchy references. It is understandable that this dual validation mechanism effectively filters out non-compliant messages, ensuring the reliability of the generated UI.

[0075] In step 308, the front-end receives the a2ui_message and renders the initial UI. In some implementations, the front-end simultaneously receives a text stream and a structured UI data stream from the server via dual channels. The text stream is used to display the natural language response content word by word, while the structured UI data stream contains complete UI structure definitions and state data. The front-end uses this data stream to construct the UI component hierarchy and render a visual UI interface. As an example, and not a limitation, the front-end can implement dual-channel data reception using SSE technology.

[0076] In step 309, the user clicks / inputs / selects. In some implementations, the user can perform various interactive operations on the rendered UI interface, including but not limited to clicking buttons, entering text, selecting options, and swiping pages.

[0077] In step 310, the action type is determined. This determination node is used to triage user interaction actions.

[0078] In branch 310-1, if the action type is a business action, then step 311, which sends the action back to the backend, is executed. In some implementations, a business action refers to an interactive action involving business logic processing, such as submitting a form, initiating a payment, or querying data. These actions need to be processed by the server, so the frontend packages the action data and associated UI state data and sends it back to the server.

[0079] In branch 310-2, if the action type is "frontend:*", then step 312, frontend local real-time processing, is executed. In some implementations, "frontend:*" actions refer to pure frontend state operations that do not require interaction with the server. As an example and not a limitation, these actions include setting the value of a specified path, resetting the value of a specified set of paths, adding an element to an array, deleting a matching element from an array, etc. The frontend can directly modify UI state data locally and refresh the interface without waiting for a server response.

[0080] In step 313, the backend returns an incremental update. In some implementations, after receiving the business action data returned by the frontend, the server executes the corresponding business logic processing, then generates an incremental update message and returns it to the frontend. The incremental update message only contains the parts of the UI that need to change, not the complete UI data. As an example and not a limitation, incremental update messages can be divided into a first type of update message and a second type of update message. The first type of update message is used to update UI state data, and the second type of update message is used to update UI structure.

[0081] In step 314, the front-end refreshes the UI in situ. In some implementations, whether an incremental update message is received from the server or local action processing is completed, the front-end performs a partial UI update operation. Depending on the update type and scope, the front-end only refreshes the areas of the UI that have changed, rather than re-rendering the entire interface. Understandably, this in-situ refresh mechanism can significantly improve interaction response speed and avoid flickering issues caused by full-page redraws.

[0082] In step 315, the user continues to interact until the task is completed. In some implementations, the user can continue to interact on the updated UI, repeating the process from steps 309 to 314 above until the entire task is completed.

[0083] It should be noted that the specific implementation forms of the above embodiments of this disclosure, which use the business agent and the service adopting the A2UI protocol as the execution subjects, are merely illustrative examples and do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art can adopt other suitable system architectures or functional module division methods to achieve the same or equivalent functions according to actual application needs. For example, it can be replaced with different AI-UI interaction systems such as the AG-UI protocol proposed by CopilotKit, the MCP-UI related protocol of Anthropic, or the UI interaction protocol in the OpenAI ecosystem. A service architecture with a custom private protocol can also be adopted. All of these do not depart from the technical concept of this disclosure, and this disclosure does not impose any limitations on them.

[0084] It is understood that the process 300 shown in this embodiment realizes the AI-driven UI generation and interaction function. Through technical means such as dynamic assembly of runtime constraint models, dual verification mechanism, dual-channel output, action diversion processing and incremental update, it effectively solves problems such as uncontrollable output of large models, inconsistent protocols, unstable rendering and inconsistent states, and significantly improves the stability, reliability and user experience of the system.

[0085] In some embodiments, an AI-driven UI generation and interaction method is provided, which is executed collaboratively by a server and a user terminal. The user terminal receives natural language input from the user and sends it to a business agent. After recognizing the need for UI generation, the business agent sends a service request containing the user's natural language requirements and capability constraints to the server. Upon receiving the service request, the server parses the request to obtain a request context; assembles a runtime constraint model based on the request context; generates a UI message based on the runtime constraint model; performs a first and second verification on the UI message; and outputs a first data stream and a second data stream to the user terminal through at least two parallel output channels based on the verified UI message. The user terminal receives the first and second data streams output by the server; maintains the creation and state of the UI instance based on its unique identifier; renders the UI interface based on the second data stream; and displays the natural language response content corresponding to the first data stream.

[0086] In some embodiments, the step of assembling the runtime constraint model according to the request context may include: the server loading a preset first capability set, which includes general UI component definitions and general interaction action definitions; loading a corresponding second capability set according to the request context; merging the second capability set and the first capability set; trimming the merged capability set according to capability constraint information; and assembling the runtime constraint model based on the trimmed capability set. In some embodiments, the first capability set may include basic components and interaction actions common to all businesses, such as buttons, input boxes, selectors, switches, etc.; the second capability set may include extended components and interaction actions specific to a particular business domain, such as order cards, payment components, logistics status display components, etc. By merging first and then trimming, the range of usable UI capabilities can be flexibly customized for different business scenarios, while ensuring the consistency of basic capabilities and avoiding different businesses forming their own protocol dialects.

[0087] In some embodiments, the steps of performing the first and second checks on the UI message described above may include: the server performing a first check on the UI message; in response to the first check being passed, performing a second check on the UI message; in response to the first check being failed, generating an error message and terminating the current operation; in response to the second check being failed, generating an error message and terminating the current operation; in response to the second check being passed, obtaining a valid UI message. In some embodiments, the first check may be a structural validity check, which can be used to verify the format specification, field integrity, and data type correctness of the UI message; the second check may be a state consistency check, which can be used to verify the validity of the unique identifier of the UI instance contained in the UI message, the validity of state transitions, the existence of component references, and the integrity of component hierarchical reference relationships. This layered verification mechanism can quickly filter out messages that obviously do not meet the format requirements, and then perform a more in-depth state consistency check on messages that pass the initial check, improving verification efficiency while ensuring comprehensiveness of the verification.

[0088] In some embodiments, the steps described above, which output a first data stream and a second data stream to a user terminal through at least two parallel output channels, may include: the server outputting the first data stream through the first output channel; outputting the second data stream through the second output channel, the second data stream containing complete UI structure definitions and status data; and outputting control signals concurrently with the first and second data streams, the control signals being used to manage message reception, processing, and termination operations. In some embodiments, the first data stream may be a text stream, which can be used by the front end to progressively display natural language response content; the second data stream may be a structured data stream, which can be used by the front end to render the complete UI interface at once. The control signals may include four types: completion, interruption, cancellation, and error, to help the front end understand the status of message transmission in a timely manner and take appropriate action.

[0089] In some embodiments, the method may further include: the user terminal detecting user interaction actions on the rendered UI interface; performing traffic splitting based on the action type; and sending interaction action data to the server in response to the interaction action being sent back to the server. The server, in response to receiving the interaction action data, parses the interaction action data; generates an incremental update message based on the interaction action data; and outputs the incremental update message to the user terminal through at least two parallel output channels. The user terminal receives the incremental update data returned by the server and performs a partial UI update based on the incremental update data. In some embodiments, the incremental update message only contains the parts of the UI interface that need to change, rather than the complete UI data, which can significantly reduce the amount of data transmitted and improve the interaction response speed.

[0090] In some embodiments, the steps of maintaining the creation and state of UI instances based on their unique identifiers may include: the user terminal receiving a globally unique UI instance identifier from the server in response to a request to generate a UI interface for the first time; binding the unique UI instance identifier to the newly created UI instance; storing the unique UI instance identifier and the state information of the UI instance; locating the corresponding UI instance in response to receiving the unique UI instance identifier in subsequent interactions; and performing an update operation based on the stored state information of the UI instance. In the example, the server automatically generates a new identifier when the front end has no identifier, and the presence of an identifier on the front end is considered an update to an existing UI interface. This identifier management mechanism effectively avoids state confusion and update errors that occur during multiple rounds of interaction.

[0091] In some embodiments, the steps for rendering the UI interface based on the second data stream may include: the user terminal parsing the second data stream to obtain UI structure information and state data; constructing a UI component hierarchy based on the UI structure information; mapping each node in the UI component hierarchy to a corresponding rendering component according to the component type; binding the rendering component with the state data and the interaction actions via the component interface; and rendering the UI component hierarchy to generate a visual UI interface. In some implementations, the front-end renderer can automatically match the corresponding wrapper component based on the node type and complete the binding of data and actions through a unified component interface. This rendering method based on structured data can significantly improve the accuracy and consistency of rendering.

[0092] In some embodiments, the steps of receiving the first data stream and the second data stream output by the server may further include: the user terminal performing deduplication processing on the received second data stream based on the unique message identifier; in response to receiving update data for a UI instance that has not yet been created, storing the update data in a message set to be processed; and in response to the completion of the creation of the corresponding UI instance, processing the update data in the message set to be processed in sequence. In the example, if only data updates are received and the UI instance has not been created, the update is delayed in queuing and replayed after creation. These mechanisms can effectively solve the problems of message duplication and out-of-order delivery that may occur in network transmission, significantly improving the reliability and stability of front-end data processing.

[0093] In some embodiments, the above-described step of performing traffic splitting based on action type may include: the user terminal extracting the type identifier and action parameters of the interactive action; determining the category to which the action belongs based on the type identifier; modifying the UI state data locally in response to the determination result being a first type of action; refreshing the UI interface based on the modified state data; and sending the interactive action and associated UI state data back to the server for processing in response to the determination result being a second type of action. In some implementations, the first type of action is typically a pure frontend state operation prefixed with "frontend:", which does not require interaction with the server; the second type of action involves business logic processing and must be executed by the server. This traffic splitting mechanism can maximize the response speed of the interaction while ensuring the correctness of the business logic.

[0094] In some embodiments, the steps of performing partial UI updates based on incremental update data may include: the user terminal parsing the incremental update data returned by the server; determining the update type and update scope; updating the changed data items in the UI state model in response to an update type of first type; updating the changed component nodes in the UI structure in response to an update type of second type; and refreshing the changed areas in the UI interface based on the updated state data or UI structure. In the example, update messages are smaller than creation messages, and creation and update can be distinguished by the begin rendering flag. This partial update mechanism can significantly reduce data transmission and rendering computation, improve the smoothness of interaction, and avoid flickering issues caused by full-page redraws.

[0095] Figures 4A to 4C The diagram illustrates the application flow of an AI-driven UI generation and interaction method according to some embodiments of the present disclosure in a form generation and submission scenario.

[0096] like Figure 4AAs shown, a user enters the natural language request "Generate a simple form" in the input box of the A2UI Assistant dialog interface. In some implementations, after receiving the user input, the business agent recognizes through semantic understanding that the request requires the generation of a visual UI interface to assist in completing the information collection task. The business agent then initiates a call request to the A2UI service, carrying the component whitelist and action whitelist corresponding to the business in the request parameters. As an example and not a limitation, the component whitelist in this request can include component types such as titles, input boxes, and buttons, and the action whitelist can include action types such as setting input values ​​and submitting forms. After receiving a request, the A2UI service first parses the request parameters to obtain the request context. Then, it assembles a runtime constraint model based on the request context, which restricts the UI generation to only using whitelisted components and actions. Next, it generates a UI message conforming to a standardized UI description protocol based on the runtime constraint model. After generation, it performs dual validation on the UI message: the first validation verifies the format specification, field completeness, and data type correctness of the UI message; the second validation verifies the validity of the unique identifier of the UI instance contained in the UI message, the legality of state transitions, the existence of component references, and the completeness of component hierarchy references. After passing the validation, it outputs the first and second data streams simultaneously through the SSE (Server-SentEvents) dual channel. The first data stream is a text stream, which the front-end receives and displays word-for-word, progressively, in natural language response content such as "A simple form interface has been generated, including name and email input boxes, and a submit button." The second data stream is a structured data stream, containing complete UI structure definitions and initial state data, which the front-end receives, parses, and renders to generate a visual UI interface containing a "User Information Form" title, a name input box, an email input box, and a submit button. Understandably, by using a dual-channel output mechanism, the front end can directly render the UI interface without performing complex parsing and conversion of natural language text, while preserving the natural language interaction experience, thus balancing technical reliability and user experience.

[0097] like Figure 4BAs shown, the user performs input operations in the rendered form interface. The user first clicks the name input box and enters "Xiaoming"; then clicks the email input box and enters "aaa@gmail.com". In some implementations, the front-end action processing module listens to the user's interaction in real time. When an input operation is detected, it extracts the action type identifier and action parameters. As an example, and not a limitation, the action type identifier for entering the name is "frontend:set", and the action parameters include the path information " / form / name" and the numerical information "Xiaoming"; the action type identifier for entering the email is also "frontend:set", and the action parameters include the path information " / form / email" and the numerical information "aaa@gmail.com". After recognizing that these actions are all local actions prefixed with "frontend:", the action processing module directly modifies the corresponding path value in the UI state data locally and refreshes the interface based on the modified state data. It is understandable that these simple state modification operations do not require interaction with the server, thus achieving instant response, significantly improving the user's interactive experience, and reducing the server load.

[0098] like Figure 4CAs shown, after filling in the form, the user clicks the "Submit" button. In some implementations, the front-end action processing module detects the click operation and extracts the action type identifier as "submit". This action does not belong to local actions prefixed with "frontend:", and is therefore determined to be a business action. The action processing module then packages the interaction action data with the associated UI state data (i.e., the user's name and email information) and sends it back to the business agent. After receiving the submitted data, the business agent executes the corresponding business logic processing, such as storing the user information in the database. After the business logic processing is completed, the business agent calls the A2UI service again to generate an incremental update message. The A2UI service generates the incremental update message and performs double verification before outputting it to the front end through SSE dual channels. This incremental update message is a second type of update message (surfaceUpdate), which only contains the component node information that needs to be replaced, i.e., a success message indicating the component's structure definition and state data. After receiving the incremental update message, the front-end performs a partial UI update: First, it parses the incremental update data to determine the identifier and location of the component node that needs to be updated; then, it locates the corresponding original form component node in the UI structure based on the component node identifier; it replaces the original form component node with the success message component node from the update data; finally, it updates the reference relationships related to this component node in the UI component hierarchy and refreshes the changed areas of the UI in situ, displaying a success message interface including a green checkmark icon, a "Submission Successful" title, the explanatory text "Your information has been successfully submitted, we will contact you as soon as possible," and a "Close" button. Simultaneously, the front-end also receives and displays the server's output through the first data stream: "Received, form submitted successfully! The following information has been recorded: 1. Name: Xiaoming 2. Email: aaa@gmail.com Do you need me to handle any other matters for you?" Understandably, through the incremental update and in-situ refresh mechanism, the front-end only refreshes the changed parts of the UI, avoiding the flickering problem caused by a full page redraw, improving the smoothness of the interaction, and significantly reducing the amount of data transmitted.

[0099] It should be noted that the specific implementation forms of the above embodiments of this disclosure, which use the business agent and the service adopting the A2UI protocol as the execution subjects, are merely illustrative examples and do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art can adopt other suitable system architectures or functional module division methods to achieve the same or equivalent functions according to actual application needs. For example, it can be replaced with different AI-UI interaction systems such as the AG-UI protocol proposed by CopilotKit, the MCP-UI related protocol of Anthropic, or the UI interaction protocol in the OpenAI ecosystem. A service architecture with a custom private protocol can also be adopted. All of these do not depart from the technical concept of this disclosure, and this disclosure does not impose any limitations on them.

[0100] Understandable. Figures 4A to 4C This paper demonstrates an exemplary workflow of the disclosed technical solution in a practical application. Specifically, by dynamically assembling runtime constraint models, multiple services can share the same service capabilities while ensuring capability isolation between services; a dual verification mechanism effectively solves the problem of unstable output from large models, significantly improving the stability and reliability of the UI generation chain; a dual-channel output mechanism balances user experience and technical reliability; a traffic splitting mechanism between front-end local action processing and business action feedback maximizes the response speed of interaction while ensuring the correctness of business logic; and incremental updates and in-situ refresh mechanisms improve the smoothness of interaction and reduce data transmission volume.

[0101] Figure 5 , Figure 6 and Figure 7 Several application examples of the AI-driven UI generation and interaction method according to some embodiments of the present disclosure in the data visualization chart generation scenario are shown.

[0102] like Figure 5 As shown, a user inputs a natural language request: "Generate a line chart of sales figures for 10 car companies, requiring 12 months of data." In some implementations, after the business agent recognizes that the request requires the generation of a data visualization UI, it calls the A2UI service, carrying a business whitelist containing line chart components. The A2UI service assembles the corresponding runtime constraint model, generates a UI message containing a chart title, horizontal and vertical axes, 10 line data series, and corresponding legends, and outputs it through dual channels after double verification. After receiving the structured data stream, the front end can directly render and generate an "Annual Sales Trend Chart of 10 Car Companies," while simultaneously displaying the corresponding natural language response content.

[0103] like Figure 6As shown, the user inputs a natural language request to "display a pie chart showing the percentage of different mobile phone brands." In some implementations, the A2UI service assembles a runtime constraint model containing a pie chart component based on the request context, generating a UI message containing data on the percentage of each brand, sector labels, and a legend. The front-end receives this message and renders a pie chart showing the percentage of different mobile phone brands, with each sector automatically labeled with its corresponding brand name and value.

[0104] like Figure 7 As shown, the user inputs a natural language request to "generate a stacked bar chart showing the popularity of different dishes." In some implementations, the A2UI service generates a UI message containing multiple series of stacked data, grouped axes, and category legends. After receiving the message, the front-end renders and generates a stacked bar chart showing the "Popularity Statistics of Different Dishes," which clearly displays the popularity comparison among online orders, dine-in services, and delivery.

[0105] The above examples demonstrate that the technical solution disclosed herein can support the automatic generation of various complex data visualization charts. Through a unified standardized protocol and dynamic capability assembly mechanism, it can quickly meet the data display needs of different business scenarios without the need to develop separate front-end rendering logic for each chart type.

[0106] Figure 8 This is a schematic diagram of an AI-driven UI generation and interaction system architecture process 800 according to some embodiments of this disclosure. It can be understood that this process 800 exemplarily demonstrates the collaborative working mechanism between various modules of the system, presenting the entire data flow and functional division from user input to action processing.

[0107] In step 801, the user provides input. In some implementations, the user can input any natural language description through a dialog interface to express their operational intent or information retrieval needs. As an example, and not a limitation, the user can input natural language requests such as "Help me check my orders for the last three months" or "I want to apply for an invoice."

[0108] In step 802, the business agent receives user input and performs preliminary processing. In some implementations, the business agent can perform semantic understanding of the user's natural language request to determine whether the request requires the generation of a visual UI to assist in completing the task. When the business agent determines that a UI needs to be generated, it can initiate a call request to the A2UI service, carrying relevant request parameters in the request. As an example and not a limitation, the request parameters may include natural language messages, thread_id, biz_code, agent_id, component_pack_ids, component_whitelist, and action_whitelist.

[0109] In step 803, the A2UI service receives request parameters sent by the business agent. In some implementations, the A2UI service can be a backend service of the system, responsible for handling all UI generation-related requests. After receiving the request, the A2UI service will pass the request parameters to the subsequent runtime schema assembly module for processing.

[0110] In step 804, the runtime schema assembly module assembles the runtime constraint model according to the request parameters. In some implementations, the assembly process may include: first loading a preset basic component package, which contains basic components and interaction action definitions common to all services; then loading the corresponding business component package according to the component_pack_ids in the request parameters; merging the basic component package and the business component package; then trimming the merged capability set according to the component_whitelist and action_whitelist; and finally generating the runtime constraint model based on the trimmed capability set. It can be understood that by dynamically assembling the runtime constraint model, the same service can support the different capability requirements of multiple services, while ensuring capability isolation between services.

[0111] In step 805, the protocol and transition verification module performs dual verification on the generated UI message. In some implementations, the first verification is a structural validity verification, used to verify the format standardization, field completeness, and data type correctness of the UI message; the second verification is a state consistency verification, used to verify the validity of the unique identifier of the UI instance contained in the UI message, the legality of state transitions, the existence of component references, and the integrity of component hierarchy reference relationships. If any verification fails, the server will generate an error message and terminate the current operation; if the verification passes, the valid UI message is passed to the SSE event flow module.

[0112] In step 806, the SSE event stream module outputs data through at least two parallel output channels. In some implementations, the SSE event stream module can simultaneously output a first data stream and a second data stream. The first data stream is a text stream used for the front-end to progressively display natural language response content word by word; the second data stream is a structured data stream containing complete UI structure definitions and state data. Additionally, the SSE event stream module can also output control signals concurrently with the first and second data streams to manage message reception, processing, and termination operations, such as outputting completion, interruption, cancellation, and error control signals.

[0113] In step 807, the A2UIMessage component receives structured messages output by the server. In some implementations, the A2UIMessage component is the entry point component of the front-end system, responsible for receiving and parsing all structured messages output by the server. After receiving a message, the A2UIMessage component first performs deduplication based on the message's unique identifier; if it receives update data for a UI instance that has not yet been created, it stores the update data in the message set to be processed and processes it sequentially after the corresponding UI instance is created.

[0114] In step 808, the message handler processes the parsed message. In some implementations, the message handler may execute different processing logic based on the message type. By way of example and not limitation, the message handler may process the beginRendering message to create a new UI instance, process the dataModelUpdate message to update UI state data, and process the surfaceUpdate message to update the UI structure. After processing, the message handler passes the result to the rendering layer for rendering.

[0115] In step 809, the rendering layer renders the UI interface based on the processing result of the message handler. In some implementations, the rendering layer can construct a UI component hierarchy based on UI structure information and map each node in the UI component hierarchy to the corresponding rendering component. After the rendering layer completes rendering, it passes the rendering result to the Wrapper component for further processing.

[0116] In step 810, the Wrapper component completes data binding and associates interactive actions. In some implementations, all Wrapper components uniformly receive node and surfaceId parameters and complete data parsing, data writing, and action construction through a unified component interface. The Wrapper component binds the rendering component with state data and associates interactive actions with corresponding processing functions, thereby generating an interactive UI.

[0117] In step 811, the action processing module processes user interactions on the UI interface. In some implementations, the action processing module can perform traffic splitting based on the action type: for local actions prefixed with "frontend:", the action processing module directly modifies the UI state data locally and refreshes the interface; for other business actions, the action processing module packages the interaction action and associated UI state data and sends it back to the business agent, which then performs further business logic processing. After the business agent completes its processing, it can call the A2UI service again to generate incremental update messages, thus forming a complete interaction loop.

[0118] It should be noted that the specific implementation forms of the above embodiments of this disclosure, which use the business agent and the service adopting the A2UI protocol as the execution subjects, are merely illustrative examples and do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art can adopt other suitable system architectures or functional module division methods to achieve the same or equivalent functions according to actual application needs. For example, it can be replaced with different AI-UI interaction systems such as the AG-UI protocol proposed by CopilotKit, the MCP-UI related protocol of Anthropic, or the UI interaction protocol in the OpenAI ecosystem. A service architecture with a custom private protocol can also be adopted. All of these do not depart from the technical concept of this disclosure, and this disclosure does not impose any limitations on them.

[0119] It is understandable that the system architecture process 800 demonstrated in this embodiment adopts a modular design approach, dividing the system functions into multiple independent modules, each responsible for a specific function, and communicating with each other through standardized interfaces. This design not only improves the maintainability and scalability of the system but also facilitates collaborative development between different teams. Furthermore, by employing techniques such as dynamically assembling runtime constraint models, dual verification mechanisms, dual-channel output, and action splitting, it effectively solves problems such as uncontrollable large model output, inconsistent protocols, unstable rendering, and inconsistent states, significantly improving the system's stability, reliability, and user experience.

[0120] Figure 9This is a schematic diagram of an AI-driven user interface (UI) generation system 900 according to some embodiments of this disclosure. The system 900 is deployed on a server and is used to generate UI-related data conforming to specifications based on the user's natural language requirements and output it to the front end. System 900 includes: a request parsing module 901, which can be used to respond to a service request containing user natural language needs and capability constraint information, parse the service request to obtain a request context, wherein the request context contains the semantic parsing result of the user's natural language needs and capability constraint information; an AI agent invocation module 902, which can be used to invoke an AI agent based on the semantic parsing result; a constraint model assembly module 903, which can be used to assemble a runtime constraint model according to the capability constraint information, wherein the runtime constraint model is a model used to constrain the output range of the AI ​​agent; a UI message generation module 904, which can be used to generate UI messages through the AI ​​agent based on the runtime constraint model; a message verification module 905, which can be used to perform a first verification and a second verification on the UI message; and a multi-channel output module 906, which can be used to split the verified UI message into a first data stream and a second data stream, and output the first data stream and the second data stream through at least two parallel output channels, wherein the first data stream can be used for front-end display of natural language response content for the natural language needs, and the second data stream can be used for front-end rendering of the UI interface.

[0121] In some embodiments, the system 900 further includes an incremental update module 907, which can be used to respond to received interactive action data, parse the interactive action data to obtain the action type and action parameters; generate an incremental update message based on the parsed interactive action data; and output the incremental update message through at least two parallel output channels. In some embodiments, the incremental update module 907 can generate incremental data that only includes the changed parts of the UI interface, rather than complete UI data, thereby significantly reducing the amount of data transmission.

[0122] In some embodiments, the constraint model assembly module 903 can be used to: load a preset first capability set, which includes general UI component definitions and general interaction action definitions; load a corresponding second capability set according to the request context; merge the second capability set and the first capability set; trim the merged capability set according to capability constraint information; and assemble a runtime constraint model based on the trimmed capability set. In some embodiments, the first capability set may contain basic components common to all businesses, while the second capability set may contain extended components specific to a particular business domain.

[0123] In some embodiments, the message verification module 905 can be used to: perform a first verification on the UI message; in response to the first verification result being successful, perform a second verification on the UI message; in response to the first verification result being unsuccessful, generate an error message and terminate the current operation; in response to the second verification result being unsuccessful, generate an error message and terminate the current operation; in response to the second verification result being successful, obtain a valid UI message. It is easy to understand that this layered verification mechanism can improve verification efficiency while ensuring comprehensive verification.

[0124] In some embodiments, the multi-channel output module 906 can be used to: output a first data stream through a first output channel; output a second data stream through a second output channel, the second data stream containing complete UI structure definitions and status data; and output control signals concurrently with the first and second data streams, the control signals being used to manage message reception, processing, and termination operations. As an example and not a limitation, the control signals may include four types: completion, interruption, cancellation, and error.

[0125] In some embodiments, the constraint model assembly module 903 can also be used to: extract component constraints and action constraints from the capability constraint information; remove component capabilities that do not meet the conditions in the merged capability set according to the component constraints; remove interactive actions that do not meet the conditions in the merged capability set according to the action constraints; detect whether there are unknown components or unknown actions in the trimmed capability set; and generate error information and terminate the current operation in response to the detection result indicating the existence of unknown components or unknown actions. It is understood that this strict trimming and detection mechanism can effectively prevent the AI ​​agent from generating content beyond the scope of business operations.

[0126] In some embodiments, the message verification module 905 performs a first verification including at least one of the following: verifying the standardization of the UI message format; verifying the integrity of the UI message fields; and verifying the correctness of the UI message data type. In some embodiments, the format standardization verification may be performed based on a preset JSON Schema specification.

[0127] In some embodiments, the message verification module 905 performs a second verification including at least one of the following: verifying the validity of the unique identifier of the UI instance contained in the UI message; verifying the legality of state transitions in the UI message; verifying the existence of all components referenced in the UI message; and verifying the integrity of the hierarchical reference relationship of components in the UI message. These verifications ensure the correctness of the UI message in terms of business logic.

[0128] In some embodiments, the multi-channel output module 906 outputs control signals including at least one of the following: outputting a first control signal to indicate that message transmission is complete; outputting a second control signal to indicate that message transmission is interrupted; outputting a third control signal to indicate that message transmission is canceled; and outputting a fourth control signal to indicate that an error occurred during message transmission. These control signals help the front end to understand the message transmission status in a timely manner and take appropriate action.

[0129] In some embodiments, the incremental update module 907 can be used to: parse interactive action data to determine the action type and action parameters; generate an incremental update message based on the action type and action parameters; the incremental update message includes at least one of a first type of update message and a second type of update message, wherein the first type of update message is used to update UI state data, and the second type of update message is used to update the UI component hierarchy. By distinguishing between different types of update messages, more refined partial UI updates can be achieved.

[0130] Figure 10 This is a schematic diagram of an AI-driven user interface (UI) interaction system 1000 according to some embodiments of this disclosure. The system 1000 is deployed on a user terminal and is used to receive UI-related data output by a server after processing by an AI agent, and to complete UI rendering and basic interaction. The system 1000 includes: a data receiving module 1001, which can be used to receive a first data stream and a second data stream output by the server after processing by an AI agent; the first data stream can be used to display natural language response content, and the second data stream can be used to render the UI interface; a UI state management module 1002, which can be used to maintain the creation and state of UI instances based on the unique identifier of the UI instance, the UI instance being created based on the second data stream; and a UI rendering module 1003, which can be used to complete the rendering of the UI interface based on the second data stream, the rendering result of the UI interface corresponding to the state of the UI instance.

[0131] In some embodiments, the system 1000 further includes a text display module 1004, which can be used to display natural language response content corresponding to the first data stream. In some embodiments, the text display module 1004 can support a progressive, word-by-word display of text content to enhance the user experience.

[0132] In some embodiments, the system 1000 further includes: an interaction detection module, used to detect user interaction actions on the rendered UI interface; an action routing module, used to perform routing processing according to the action type of the interaction action, the routing processing being used to distinguish between interaction actions that need to be processed locally and interaction actions that need to be sent back to the server for processing; an update receiving module, used to receive incremental update data generated and returned by the server according to the interaction action in response to the interaction action being sent back to the server; and a partial update module, used to perform partial UI updates according to the incremental update data. Through action routing processing, the interaction response speed can be improved while ensuring the correctness of business logic.

[0133] In some embodiments, the UI state management module 1002 can be used to: receive a globally unique UI instance identifier issued by the server in response to a request to generate a UI interface for the first time; bind the UI instance identifier to the newly created UI instance; store the UI instance identifier and the state information of the UI instance; locate the corresponding UI instance in response to receiving the UI instance identifier in subsequent interactions; and perform an update operation based on the stored state information of the UI instance. This identifier management mechanism can effectively avoid the state confusion problem in multi-round interactions.

[0134] In some embodiments, the UI rendering module 1003 can be used to: parse the second data stream to obtain UI structure information and state data; construct a UI component hierarchy based on the UI structure information; map each node in the UI component hierarchy to the corresponding rendering component according to the component type; bind the rendering component to the state data and the interaction actions via the component interface; and render the UI component hierarchy to generate a visual UI interface. This rendering method based on structured data can significantly improve the accuracy and consistency of rendering.

[0135] In some embodiments, the data receiving module 1001 can also be used to: perform deduplication processing on the received second data stream based on the unique identifier of the message; store the updated data into a message set to be processed in response to receiving updated data for a UI instance that has not yet been created; and process the updated data in the message set to be processed in sequence in response to the completion of the creation of the corresponding UI instance. These mechanisms can effectively solve the problems of message duplication and out-of-order delivery in network transmission.

[0136] In some embodiments, the action routing module can be used to: extract the type identifier and action parameters of the interactive action; determine the category to which the action belongs based on the type identifier; modify the UI state data locally in response to the determination result being a first type of action; refresh the UI interface based on the modified state data; and send the interactive action and associated UI state data back to the server for processing in response to the determination result being a second type of action. As an example and not a limitation, the first type of action is typically a pure frontend state operation prefixed with "frontend:".

[0137] In some embodiments, the partial update module can be used to: parse the incremental update data returned by the server; determine the update type and update scope; update the changed data items in the UI state model in response to a first-type update; update the changed component nodes in the UI component hierarchy in response to a second-type update; and refresh the changed areas in the UI interface based on the updated state data or UI component hierarchy. This partial update mechanism avoids the flickering problem caused by full-page redraws.

[0138] In some embodiments, the action routing module performs local modification of UI state data, including at least one of the following: In response to a first subtype action, setting the value of a specified path in the UI state data based on the path and value information in the action parameters; in response to a second subtype action, resetting the value of a specified path set in the UI state data based on the path set information in the action parameters; in response to a third subtype action, adding an element to a specified array path in the UI state data based on the array path and value information in the action parameters; and in response to a fourth subtype action, deleting a matching element from a specified array path in the UI state data based on the array path, matching key, and matching value information in the action parameters. These predefined local actions can cover most common UI state operation scenarios.

[0139] In some embodiments, the partial update module can be used to update changed component nodes in the UI structure by: parsing the second type of update data; determining the identifier and location of the changed component node; locating the corresponding original component node in the UI component hierarchy based on the component node's identifier; replacing the original component node with the new component node from the update data; and updating the reference relationships related to the component node in the UI component hierarchy. This fine-grained structure update mechanism can significantly improve the efficiency of UI updates.

[0140] It should be understood that Figure 9 and Figure 10 The various modules or units of system 900 and system 1000 shown can be compared with the reference. Figure 1 Method 100 and references described Figure 2The steps in method 200 described correspond to each other. Therefore, the operations, features, and advantages described above for methods 100 and 200 also apply to systems 900 and 1000 and their included modules and units. For the sake of brevity, some operations, features, and advantages will not be repeated here.

[0141] Although specific functions have been discussed with reference to specific modules above, it should be noted that the functions of the units discussed in this article can be divided into multiple units, and / or at least some functions of multiple units can be combined into a single unit.

[0142] According to another aspect of this disclosure, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores a computer program that, when executed by the at least one processor, implements the method described above.

[0143] According to another aspect of this disclosure, a computer-readable storage medium storing a computer program is also provided, wherein the computer program implements the above-described method when executed by a processor.

[0144] According to another aspect of this disclosure, a computer program product is also provided, comprising a computer program, wherein the computer program implements the above-described method when executed by a processor.

[0145] See Figure 11 The present invention describes a structural block diagram of an electronic device 1100 that can serve as a server or user terminal, etc., as an example of hardware devices applicable to various aspects of the present disclosure. The electronic device can be different types of computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0146] like Figure 11 As shown, the electronic device 1100 may include at least one processor 1101, working memory 1102, input unit 1104, display unit 1105, speaker 1106, storage unit 1107, communication unit 1108 and other output units 1109 that are capable of communicating with each other via system bus 1103.

[0147] Processor 1101 may be a single processing unit or multiple processing units, and all processing units may include single or multiple computing units or multiple cores. Processor 1101 may be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuits, and / or any device that manipulates signals based on operating instructions. Processor 1101 may be configured to acquire and execute computer-readable instructions stored in working memory 1102, storage unit 1107, or other computer-readable media, such as program code of operating system 1102a, program code of application program 1102b, etc.

[0148] Working memory 1102 and storage unit 1107 are examples of computer-readable storage media for storing instructions executed by processor 1101 to perform the various functions described above. Working memory 1102 may include both volatile and non-volatile memory (e.g., RAM, ROM, etc.). Furthermore, storage unit 1107 may include hard disk drives, solid-state drives, removable media including external and removable drives, memory cards, flash memory, floppy disks, optical disks (e.g., CDs, DVDs), storage arrays, network-attached storage, storage area networks, etc. Working memory 1102 and storage unit 1107 may be collectively referred to herein as memory or computer-readable storage media, and may be non-transitory media capable of storing computer-readable, processor-executable program instructions as computer program code, which may be executed by processor 1101 as a specific machine configured to perform the operations and functions described in the examples herein.

[0149] Input unit 1104 can be any type of device capable of inputting information to electronic device 1100. Input unit 1104 can receive input digital or character information and generate key signal input related to user settings and / or function control of electronic device, and can include, but is not limited to, a mouse, keyboard, touch screen, trackpad, trackball, joystick, microphone and / or remote control. Output unit can be any type of device capable of presenting information, and can include, but is not limited to, display unit 1105, speaker 1106 and other output units 1109. Other output units 1109 can include, but are not limited to, video / audio output terminals, vibrators and / or printers. Communication unit 1108 allows electronic device 1100 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and can include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers and / or chipsets, such as Bluetooth™ devices, 802.11 devices, WiFi devices, WiMax devices, cellular communication devices and / or the like.

[0150] The application program 1102b in the working memory 1102 can be loaded to execute the various methods and processes described above, for example... Figure 1 Boxes S110 to S150 in the middle Figure 2 Boxes S210 to S230, etc., are shown in the diagram. For example, in some embodiments, the AI-driven UI interaction method executed by a server or user terminal can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1107. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1100 via storage unit 1107 and / or communication unit 1108. When the computer program is loaded and executed by processor 1101, one or more steps of the AI-driven UI interaction method executed by the server or user terminal described above can be performed. Alternatively, in other embodiments, processor 1101 can be configured by any other suitable means (e.g., by means of firmware) to execute the AI-driven UI interaction method executed by the server or user terminal.

[0151] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0155] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0156] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0157] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0158] While embodiments or examples of this disclosure have been described with reference to the accompanying drawings, it should be understood that the methods, systems, and devices described above are merely exemplary embodiments or examples, and the scope of the invention is not limited by these embodiments or examples, but only by the granted claims and their equivalents. Various elements in the embodiments or examples may be omitted or replaced by their equivalents. Furthermore, the steps may be performed in a different order than that described in this disclosure. Further, various elements in the embodiments or examples may be combined in various ways. Importantly, as the technology evolves, many elements described herein can be replaced by equivalents that appear after this disclosure.

Claims

1. An AI-driven user interface (UI) generation method, executed by a server, the method comprising: In response to receiving a service request containing user natural language requirements and capability constraints, the service request is parsed to obtain a request context, which includes the semantic parsing results of the user's natural language requirements and capability constraints. Based on the semantic parsing results, the AI ​​agent is invoked; Based on the capability constraint information, a runtime constraint model is assembled, wherein the runtime constraint model is a model used to constrain the output range of the AI ​​agent; UI messages are generated by the AI ​​agent based on the runtime constraint model. Perform a first verification and a second verification on the UI message; Based on the verified UI message, a first data stream and a second data stream are obtained, and the first data stream and the second data stream are output through at least two parallel output channels. The first data stream is used for the front end to display natural language response content for the natural language requirement, and the second data stream is used for the front end to render the UI interface.

2. The AI-driven UI generation method according to claim 1 further includes: In response to receiving interactive action data, the interactive action data is parsed to obtain the action type and action parameters; Based on the parsed interaction action data, an incremental update message is generated; The incremental update message is output through at least two parallel output channels.

3. The AI-driven UI generation method according to claim 1, wherein, The step of assembling the runtime constraint model according to the request context includes: Load a preset first capability set, which includes general UI component definitions and general interactive action definitions; Based on the request context, load the corresponding second capability set, which includes dedicated UI component definitions and dedicated interaction action definitions that are adapted to the semantic parsing results and the capability constraint information; Merge the second set of capabilities with the first set of capabilities; Based on the capability constraint information, the merged capability set is trimmed; Based on the trimmed capability set, assemble the runtime constraint model.

4. The AI-driven UI generation method according to claim 1, wherein, The first and second checks performed on the UI message include: Perform the first verification on the UI message; If the first verification passes, a second verification is performed on the UI message; If the first check fails, an error message is generated and the current operation is terminated. If the second verification fails, an error message is generated and the current operation is terminated. If the second verification passes, a valid UI message is obtained.

5. The AI-driven UI generation method according to claim 1, wherein, The UI message based on successful verification outputs a first data stream and a second data stream through at least two parallel output channels, including: The first data stream is output through the first output channel; The second output channel outputs a second data stream, which contains the complete UI structure definition and state data. Control signals are output concurrently with the first data stream and the second data stream. These control signals are used to manage message reception, processing, and termination operations.

6. The AI-driven UI generation method according to claim 3, wherein, The step of trimming the merged capability set based on the capability constraint information includes: Extract the component constraints and action constraints from the capability constraint information; Based on the component constraints, remove component capabilities from the merged capability set that do not meet the conditions. Based on the action constraints, remove the interaction actions that do not meet the conditions from the merged capability set; The above two removal steps complete the trimming of the merged capability set in order to detect whether there are unknown components or unknown actions in the trimmed capability set; In response to the detection result indicating the presence of an unknown component or unknown action, an error message is generated and the current operation is terminated.

7. The AI-driven UI generation method according to claim 4, wherein, The first verification performed on the UI message includes at least one of the following: Verify the standardization of UI message formats; Verify the integrity of UI message fields; Verify the correctness of the UI message data type.

8. The AI-driven UI generation method according to claim 4, wherein, The second verification performed on the UI message includes at least one of the following: Verify the validity of the unique identifier of the UI instance contained in the UI message; Verify the validity of state transitions in UI messages; Verify the existence of all components referenced in the UI message; Verify the integrity of the hierarchical references of components in the UI messages.

9. The AI-driven UI generation method according to claim 5, wherein, The control signal output concurrently with the first data stream and the second data stream includes at least one of the following: Output a first control signal, which is used to indicate that message transmission is complete; Output a second control signal, which is used to indicate that message transmission has been interrupted; Output a third control signal, which is used to indicate that message transmission has been cancelled; Output a fourth control signal, which is used to indicate that an error has occurred during message transmission.

10. The AI-driven UI generation method according to claim 2, wherein, The step of generating an incremental update message based on the interaction action data includes: The interaction action data is parsed to determine the action type and action parameters; Based on the action type and action parameters, an incremental update message is generated; The incremental update message includes at least one of a first type of update message and a second type of update message. The first type of update message is used to update UI state data, and the second type of update message is used to update the UI component hierarchy structure.

11. An AI-driven UI interaction method, executed by a user terminal, the method comprising: The receiving server processes the data through an AI agent and outputs a first data stream and a second data stream. The first data stream is used to display the natural language response content, and the second data stream is used to render the UI interface. The creation and status of UI instances are maintained based on the unique identifier of the UI instance, and the UI instance is created based on the second data stream; The UI interface is rendered based on the second data stream, and the rendering result of the UI interface corresponds to the state of the UI instance.

12. The AI-driven UI interaction method according to claim 11 further includes displaying natural language response content corresponding to the first data stream.

13. The AI-driven UI interaction method according to claim 11, further comprising: Detect user interactions on the rendered UI. The traffic splitting process is performed according to the action type of the interactive action, and the traffic splitting process is used to distinguish between interactive actions that need to be processed locally and interactive actions that need to be sent back to the server for processing. In response to the interaction being sent back to the server, receive incremental update data generated and returned by the server based on the interaction; Perform a UI update based on the incremental update data.

14. The AI-driven UI interaction method according to claim 11, wherein, Maintaining the creation and status of UI instances based on their unique identifiers includes: In response to the initial request to generate the UI interface, it receives a globally unique UI instance identifier issued by the server; Bind the unique identifier of the UI instance to the newly created UI instance; Store the unique identifier of the UI instance and the state information of the UI instance; In response to receiving a unique UI instance identifier in subsequent interactions, locate the corresponding UI instance; Based on the state information already stored for the UI instance, perform an update operation on the UI instance to maintain its state.

15. The AI-driven UI interaction method according to claim 11, wherein, The rendering of the UI interface based on the second data stream includes: Parse the second data stream to obtain UI structure information and state data; Construct a UI component hierarchy structure based on the UI structure information; Based on the component type, map each node in the UI component hierarchy to the corresponding rendering component; The binding of rendering components to state data and the binding of rendering components to interactive actions are performed through the component interface; Render the UI component hierarchy to generate a visual UI interface.

16. The AI-driven UI interaction method according to claim 11, wherein, The process of receiving the first data stream and the second data stream output by the server after processing by the AI ​​agent further includes preprocessing the received data streams, including: Based on the unique identifier of the message, the received second data stream is deduplicated; In response to receiving update data for a UI instance that has not been created, the update data is stored in the pending message set; In response to the completion of the creation of the corresponding UI instance, update data in the pending message set is processed in the order of storage.

17. The AI-driven UI interaction method according to claim 13, wherein, The process of performing traffic splitting based on action type includes: Extract the type identifier of the interactive action; Extract the action parameters of the interactive action, wherein the action parameters are associated with the execution of the interactive action; Based on the type identifier, determine the category to which the action belongs; In response to the judgment result being a first type of action, the UI state data is modified locally according to the action parameters; Refresh the UI based on the modified status data; In response to the judgment result being a second type of action, the interaction action and associated UI state data are sent back to the server for processing.

18. The AI-driven UI interaction method according to claim 13, wherein, The step of performing UI updates based on the incremental update data includes: Parse the incremental update data returned by the server; The update type and update range are determined based on the incremental update data; In response to an update of type 1, update the data items in the UI state model that have changed; In response to an update of type 2, update the component nodes in the UI component hierarchy that have changed; Refresh the areas of the UI that have changed based on the updated state data or UI component hierarchy.

19. The AI-driven UI interaction method according to claim 17, wherein, The response to the judgment result being a first type of action involves modifying the UI state data locally, including at least one of the following: In response to the judgment result being the first subtype action, the value of the specified path in the UI state data is set according to the path information and numerical information in the action parameters; In response to the judgment result being a second subtype action, the value of the specified path set in the UI state data is reset according to the path set information in the action parameters; In response to the judgment result being a third subtype action, an element is added to the specified array path in the UI state data according to the array path information and value information in the action parameters; In response to the judgment result being a fourth subtype action, the matching element is deleted from the specified array path in the UI state data based on the array path information, matching key information, and matching value information in the action parameters.

20. The AI-driven UI interaction method according to claim 18, wherein, The response to an update of type 2, updating component nodes in the UI component hierarchy that have changed, includes: Analyze the second type of updated data; Determine the identifier and location of the component node that has changed; Based on the identifier of the component node, locate the corresponding original component node in the UI component hierarchy; Replace the original component node with the new component node from the updated data; Update the reference relationships related to the component node in the UI component hierarchy.

21. An artificial intelligence (AI) driven user interface (UI) generation system, deployed on a server, the system comprising: The request parsing module is configured to, in response to receiving a service request containing user natural language requirements and capability constraints, parse the service request to obtain a request context, the request context containing the semantic parsing results of the user's natural language requirements and capability constraints. The agent invocation module is configured to invoke the AI ​​agent based on the semantic parsing results; The constraint model assembly module is configured to assemble a runtime constraint model based on the capability constraint information, wherein the runtime constraint model is a model used to constrain the output range of the AI ​​agent; The UI message generation module is configured to generate UI messages through the AI ​​agent based on the runtime constraint model. The message verification module is configured to perform a first verification and a second verification on the UI message; The multi-channel output module is configured to split a first data stream and a second data stream based on a valid UI message, and output the first data stream and the second data stream through at least two parallel output channels. The first data stream is used for the front end to display natural language response content for the natural language requirement, and the second data stream is used for the front end to render the UI interface.

22. The AI-driven UI generation system according to claim 21 further includes an incremental update module, configured as follows: In response to receiving interactive action data, the interactive action data is parsed to obtain the action type and action parameters; Based on the interaction action data, an incremental update message is generated; The incremental update message is output through at least two parallel output channels.

23. An artificial intelligence (AI) driven user interface (UI) interaction system, deployed on a user terminal, the system comprising: The data receiving module is configured to receive a first data stream and a second data stream output by the server after processing by the AI ​​agent. The first data stream is used to display natural language response content, and the second data stream is used to render the UI interface. The UI state management module is configured to maintain the creation and state of UI instances based on the unique identifier of the UI instance, wherein the UI instance is created based on the second data stream; The UI rendering module is configured to render the UI interface based on the second data stream, and the rendering result of the UI interface corresponds to the state of the UI instance.

24. The AI-driven UI interaction system according to claim 23 further includes a text display module configured to display natural language response content corresponding to the first data stream.

25. The AI-driven UI interaction system according to claim 23, further comprising: The interaction detection module is configured to detect user interactions on the rendered UI interface. The action routing module is configured to perform routing processing based on the action type of the interactive action. The routing processing is used to distinguish between interactive actions that need to be processed locally and interactive actions that need to be sent back to the server for processing. The update receiving module is configured to receive incremental update data generated and returned by the server based on the interaction action in response to the interaction action being sent back to the server. The partial update module is configured to perform UI updates based on the incremental update data.

26. An electronic device comprising: At least one processor; as well as A memory that is communicatively connected to the at least one processor; in The memory stores a computer program that, when executed by the at least one processor, implements the method according to any one of claims 1-20.

27. A non-transitory computer-readable storage medium storing a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-20.

28. A computer program product comprising a computer program, wherein, The computer program, when executed by a processor, implements the method according to any one of claims 1-20.

Images