A multi-modal perception function interface intelligent generation method and system based on TPRI-DataVue
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XIAN TPRI POWER PLANT INFORMATION TECHNOLOGY CO LTD
- Filing Date
- 2026-03-05
- Publication Date
- 2026-06-12
AI Technical Summary
The TPRI-DataVue platform suffers from high learning costs, low development efficiency, and insufficient intelligence. In particular, it has low accuracy and significant illusions when generating DVSL code, resulting in low efficiency in interface generation.
By employing a multi-agent system (MAS) combined with retrieval-enhanced generation (RAG) technology and prompting engineering, high-precision generation of multimodal inputs into DVSL is achieved through demand analysis agents, layout planning agents, and code generation agents. This includes intent recognition, layout planning, and component code generation. Furthermore, the system's intelligence level is improved through knowledge base evolution.
It significantly lowers the barrier to entry for platform use, improves development efficiency and intelligence, and achieves end-to-end automated generation from multimodal input to highly available interfaces, reducing manual drag-and-drop configuration time and supporting the rapid construction of complex enterprise-level interfaces.
Smart Images
Figure CN122195556A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the fields of computer software technology and artificial intelligence technology, and relates to a method and system for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue. Background Technology
[0002] The TPRI-DataVue low-code development platform aims to enable rapid front-end interface construction through intuitive operations such as visual drag-and-drop, component configuration, and custom elements. Its interface is defined by a proprietary domain-specific language (DVSL) based on JSON format. However, in practical applications of TPRI-DataVue, the following common problems still exist: 1) High learning cost: New users need to spend a significant amount of time familiarizing themselves with the platform's component system and configuration, and custom element operations require a certain level of programming and scripting knowledge, making it difficult to independently complete complex interface development; 2) Limited development efficiency: Experienced users still need to perform a large amount of repetitive drag-and-drop, parameter configuration, and layout adjustments when building complex business models, limiting work productivity; 3) Insufficient intelligence: General large models (LLMs) lack expertise in proprietary and syntactically strict DVSL, resulting in severe "illusion" problems when directly generating DVSL, leading to rendering engine parsing failures, extremely low code usability, and inability to be directly integrated into platform applications. Therefore, there is an urgent need for an intelligent solution that can deeply understand DVSL, has high fault tolerance and continuous evolution capabilities, and can automate the construction of functional interfaces through multimodal interaction, so as to reduce the threshold for platform use, improve user development efficiency and work productivity, and enhance the intelligence level of the platform. Summary of the Invention
[0003] To address the problems existing in the prior art, the present invention aims to provide a method and system for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue. By constructing a multi-agent system (MAS) and introducing retrieval-augmented generation (RAG) and prompt engineering techniques, the interface generation task is decomposed into sub-tasks handled by specialized agents. Each specialized agent has the ability to autonomously call tools and iteratively optimize, solving the problems of low accuracy and severe illusions in general LLMs in DVSL generation tasks. This achieves end-to-end high-precision generation from multimodal information to DVSL, thereby assisting TPRI-DataVue platform users in interface design and improving platform usability and development efficiency.
[0004] This invention is achieved through the following technical solution: A method for intelligently generating multimodal awareness functional interfaces based on TPRI-DataVue includes the following steps: Receive multimodal input data from users, the multimodal input data including natural language description information and / or interface image information; The demand analysis agent performs intent recognition and entity extraction on the multimodal input data to generate a structured component list, which includes component types and attribute descriptions. The layout planning agent generates initial layout parameters for each component based on the canvas size and the component list, and performs conflict detection and iterative optimization on the initial layout parameters to obtain a layout scheme. The code generation agent retrieves similar component templates from the knowledge base based on the corresponding component type and attribute description for each component, and generates component code snippets that conform to the DVSL specification by combining syntactic constraints. The generated code snippets are then subjected to syntactic verification and iterative correction to obtain the component code. The layout schemes and component code of each component are merged to generate a complete UI-level DVSL file, which is then transferred to the TPRI-DataVue rendering engine for UI rendering.
[0005] Preferably, the intent recognition and entity extraction performed by the demand analysis agent includes: When the multimodal input data contains interface images, the images are first parsed using a large multimodal model to extract implicit functional requirements. Using mind chain technology to drive a large language model to perform semantic analysis of functional requirements, identify component types and attribute descriptions; The metadata verification tool is invoked to check whether the identified component type exists in the platform component library. If the verification fails or there is ambiguity, a clarification question is generated and interacted with the user until the confidence level meets the preset threshold or the maximum number of retries is reached.
[0006] Preferably, the layout planning agent performs conflict detection and iterative optimization, including: Call the boundary detection tool to check if the component exceeds the canvas area; if it does, adjust the layout parameters. Use a collision detection tool to check if there is overlap between components; if there is overlap, adjust the layout parameters. Repeat the above process until there are no layout conflicts or the maximum number of iterations is reached, and then output the final layout.
[0007] Preferably, the code generation agent uses retrieval-enhanced generation technology to generate component code fragments, including: For each component, construct component semantic query information, and match the Top-K similar component templates and corresponding JSON Schema syntax constraints in the knowledge base through vector retrieval; The retrieved templates, grammatical constraints, and component attribute descriptions are combined into structured prompt words. These structured prompt words are then input into a large language model to generate code snippets for multiple components. The generated code snippet is formatted using a syntax validation tool. If the validation fails, the error message is sent back to the large language model for regeneration, until the validation passes or the maximum number of retries is reached.
[0008] Preferably, it also includes a knowledge base evolution step: When a user modifies and saves the generated interface in the TPRI-DataVue interface designer, the modified DVSL code snippet and its corresponding original functional requirements are automatically captured and constructed as a data pair of instruction-correction code. The data pairs are stored in the knowledge base as new templates, and priority retrieval weights are assigned to high-quality templates to achieve continuous self-evolution of the generation effect; Preferably, the knowledge base evolution step further includes quality assessment of new templates, the indicators of which include user feedback evaluation, template call frequency and template usage success rate, and the retrieval weight of the templates is dynamically adjusted based on the evaluation results.
[0009] A multimodal awareness functional interface intelligent generation system based on TPRI-DataVue includes: The user interaction module is used to receive multimodal input from users and provide human-computer dialogue and visual interaction interfaces; The multi-agent module is used to instantiate and schedule the requirements analysis agent, the layout planning agent, and the code generation agent, and to manage message passing, task distribution, and state synchronization between agents. The tool service module provides a variety of tools, including at least metadata verification tools, boundary detection tools, collision detection tools, and syntax verification tools, for the intelligent agent to call autonomously. The retrieval enhancement module stores a knowledge base containing component templates and JSON Schema syntax constraints, and provides vector retrieval support for code generation agents; The integrated rendering module is used to communicate with the TPRI-DataVue platform, transmit the generated DVSL file to the platform's rendering engine, synchronize the canvas state in real time, and receive verification feedback from the platform and user modification data.
[0010] Preferably, the user interaction module also supports streaming output of the inference process and intermediate results of the multi-agent system, and has a built-in user feedback interface that allows users to copy, retry, visually edit, apply, and like / dislike the generated DVSL results. The tool service module encapsulates each tool using the Model Context Protocol standard, allowing the agent to call them as needed during task execution.
[0011] The multi-agent module is built on a state machine mechanism to ensure that the task flow from requirement understanding to code generation is coherent and logically consistent.
[0012] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a multimodal perception functional interface intelligent generation method based on TPRI-DataVue.
[0013] A computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements the steps of the intelligent generation method for a multimodal perception functional interface based on TPRI-DataVue.
[0014] Compared with the prior art, the present invention has the following beneficial technical effects: This invention discloses a multimodal awareness functional interface intelligent generation method based on TPRI-DataVue: A requirements analysis agent performs intent recognition and entity extraction on the user's multimodal input to generate a structured component list; a layout planning agent generates a conflict-free layout scheme based on the canvas size and component list; a code generation agent, for each component, uses retrieval-enhanced generation technology to search for similar component templates from the knowledge base and generates component code snippets conforming to the DVSL specification, while simultaneously using tool calls for iterative verification and correction; the layout scheme and component code are merged to generate a complete interface-level DVSL file, which is then transmitted to the TPRI-DataVue rendering engine for interface rendering. This invention effectively solves the illusion and formatting error problems when general LLMs generate private domain-specific languages (such as DVSL) through a multi-agent collaboration mechanism, RAG technology, strong JSON Schema constraints, and iterative verification via tool calls, ensuring that the generated code fully conforms to the platform's rendering standards. Furthermore, this invention compresses the previously time-consuming manual drag-and-drop configuration work to the second level, greatly liberating developer productivity and supporting the rapid construction of multi-component, complex enterprise-level interfaces, effectively improving the development efficiency of the TPRI-DataVue platform. Through the organic combination of multi-agent collaboration, retrieval-enhanced generation, and tool call mechanisms, this invention effectively solves the illusion problem in the generation of private domain-specific languages from general large models, achieving end-to-end automated generation from multimodal input to a highly available interface, significantly lowering the platform's usage threshold and improving development efficiency and intelligence. Attached Figure Description
[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 The present invention provides a step-by-step flowchart of a multimodal perception functional interface intelligent generation method based on TPRI-DataVue.
[0017] Figure 2 This is a schematic diagram of the output list of the intent recognition and entity extraction steps of this invention.
[0018] Figure 3 This is a schematic diagram of the output list of the dynamic layout steps of the interface of the present invention.
[0019] Figure 4This invention provides a modular structure for a multimodal perception functional interface intelligent generation system based on TPRI-DataVue. Detailed Implementation
[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.
[0021] The first aspect of this invention provides a method for intelligently generating a multimodal awareness functional interface based on TPRI-DataVue, comprising: Step 1: Intent Recognition and Entity Extraction This process is executed by the Requirement Analysis Agent. The agent receives user-input natural language interface construction requirements or uploaded functional interface prototypes, screenshots, sketches, etc. It first generates implicit functional requirements for the relevant interface images using Multimodal Large Language Models (MLLMs). Then, it uses Chain of Thought (CoT) technology to drive general-purpose LLMs to perform semantic analysis on the functional requirements, identifying and extracting the component types and associated attribute descriptions supported by TPRI-DataVue. The agent automatically calls a pre-built "metadata verification tool" to check if the identified components exist in the platform's component library. If verification fails or the identification results are ambiguous or incomplete, the agent automatically generates clarification questions and sends them back to the user for retrying, until the confidence level meets a preset threshold or the maximum number of retries is reached.
[0022] The demand analysis agent is the first-level specialized processing unit deployed in a multi-agent system, responsible for transforming unstructured user input into structured, machine-understandable data. It can simultaneously process natural language text and interface images (including prototypes, screenshots, and hand-drawn sketches); it has a built-in multimodal large model (MLLMs) for image parsing and utilizes CoT (CoT) technology to drive a general large language model for semantic analysis and entity extraction; it can autonomously invoke a "metadata verification tool" to check if identified components exist in the platform's component library; when the identification results are ambiguous or missing, it can proactively generate clarification questions to interact with the user until the result confidence level meets a preset threshold.
[0023] A multi-agent system (MAS) is a distributed computing system formed by the interaction of multiple autonomous or semi-autonomous intelligent agents in an environment. It can solve complex problems that are difficult for a single intelligent agent to handle. An agent is a proxy capable of perceiving its environment and taking actions to achieve a specific goal. It can be software, hardware, or a system, possessing autonomy, adaptability, and interactive capabilities. Intelligent agents perceive changes in the environment (e.g., through sensors or data input), make judgments and decisions based on their learned knowledge and algorithms, and then execute actions to influence the environment or achieve predetermined goals. Intelligent agents are widely used in the field of artificial intelligence.
[0024] Step 2: Dynamic layout of the interface This process is executed by the Layout Planning Agent. This agent obtains the canvas size parameters and the component list parsed in step 1, and uses suggestive engineering techniques to automatically plan and generate the initial coordinates (x, y) and dimensions (w, h) of each component using general LLMs. The agent automatically calls the "boundary detection tool" and "collision detection tool" for detection. If overlapping or exceeding the canvas is detected between components, the specific conflict information is used as feedback to drive the model to replan the layout. This process iterates until a conflict-free layout is generated or the maximum number of retries is reached.
[0025] The layout planning agent is the second-level specialized processing unit deployed in a multi-agent system. It is responsible for transforming the component list output from the requirements analysis phase into a geometric layout scheme that conforms to canvas constraints. It receives the component list and current canvas size parameters from the requirements analysis agent, and automatically generates the initial position coordinates (x, y) and dimensions (w, h) of each component based on prompting engineering techniques driven by a large language model. It can sequentially call the "boundary detection tool" to check if components exceed the canvas limits and the "collision detection tool" to check for overlaps between components. When boundary violations or overlaps are detected, the specific conflict information is used as feedback to drive the model to replan. It supports a "generate-detect-correct" cyclic execution mechanism until a conflict-free layout scheme is generated or the preset maximum number of retries is reached, ensuring that the output layout parameters can be directly used for subsequent code generation.
[0026] Step 3: Component Parallel Retrieval Enhanced Generation This process is executed by a Code Generation Agent. For each component parsed in step 1, the agent uses RAG technology to match and retrieve data from a pre-built platform asset knowledge base based on the component's type and attribute description. It combines the retrieved Top-K high-quality component templates, their corresponding JSON Schema syntax constraints, and attribute descriptions as contextual information to construct high-quality structured prompts. These prompts are then input into a large language model to generate DVSL-compliant component JSON code snippets in parallel. The agent automatically calls a syntax validation tool; if the generated code snippet fails validation, it regenerates it with the specific error message as feedback until it is completely correct or the maximum number of retries is reached. The generation process for all components is executed in parallel.
[0027] The code generation agent is the third-level specialized processing unit deployed in a multi-agent system. It is responsible for transforming the component descriptions output during the requirements analysis phase into component code snippets conforming to DVSL syntax specifications. For each component, semantic query information is constructed based on its type and attribute descriptions. Through Retrieval Augmented Generation (RAG) technology, similar component templates, corresponding JSON Schema syntax constraints, and attribute descriptions are matched in a pre-built platform asset knowledge base. The search results are combined into structured prompts and input into the large language model. It supports launching code generation tasks for multiple components simultaneously, transforming the originally sequential generation process into parallel execution, significantly improving the overall system processing efficiency. After code generation is complete, it autonomously calls a "syntax validation tool" to perform format validation on the generated component code snippets. When a syntax error is detected, the specific error path and error type information are extracted and fed back to the large language model for regeneration, until the validation is completely passed or the maximum number of retries is reached.
[0028] Step 4: Component Merging and Interface Integration Create a new layout component, inject the layout parameters generated in step 2 and the component DVSL code snippet generated in step 3 into the layout container of the layout component to synthesize a complete UI-level DVSL file; the TPRI-DataVue rendering engine parses and validates the synthesized JSON data, and automatically draws and displays the final configured front-end interface on the canvas of the interface designer.
[0029] Step 5: User Feedback and Knowledge Base Evolution When a user modifies and saves the interface obtained in step 4 in the TPRI-DataVue interface designer, the system automatically captures the modified DVSL code snippet and its corresponding original functional requirements, forming a high-quality "instruction-DVSL" data pair; this data pair is stored in the knowledge base as a new template; in subsequent searches, high-quality templates that have been manually corrected are retrieved first, so as to achieve continuous self-evolution of the generated effect.
[0030] A second aspect of this invention provides a multimodal awareness functional interface intelligent generation system based on TPRI-DataVue, comprising: User interaction module: Provides a human-computer dialogue and image upload interface, supporting streaming output. The module has a built-in user feedback interface, allowing users to copy, retry, visually edit, apply, and provide feedback (likes / dislikes) on the generated DVSL results.
[0031] Multi-agent module: As the core control unit of the system, it is responsible for instantiating and scheduling the requirements analysis agent, the layout planning agent, and the code generation agent, and managing message passing, task distribution, and state synchronization between agents.
[0032] Tool Service Module: This module encapsulates various tools using the Model Context Protocol (MCP) standard, allowing agents to call them autonomously. These include, but are not limited to, metadata verification tools, boundary detection tools, collision detection tools, and syntax verification tools.
[0033] The retrieval enhancement module configures a high-performance vector database to store and index the standard JSON Schema syntax constraints and high-quality templates of all DVSL components on the platform, providing contextual retrieval support for code generation agents.
[0034] Integrated rendering module: Responsible for data communication with the TPRI-DataVue platform, transmitting the final synthesized DVSL code to the platform's rendering engine, synchronizing the canvas state in real time, and receiving verification feedback from the platform and user modification data.
[0035] The technical solution of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] Example 1 like Figure 1 As shown, the first aspect of the present invention provides a specific implementation of a method for intelligently generating a multimodal perception functional interface based on TPRI-DataVue, comprising: Step 1: Intent Recognition and Entity Extraction This step aims to transform unstructured user requirements into a structured list of component metadata.
[0037] 1) The system first receives the interface construction requirement information input by the user through the user interaction module. The requirement information may be in text form or image form.
[0038] Functional description information in text form: Users input natural language descriptions, such as "Please generate an overview interface of nuclear power plant power generation, including a top title bar, a left-side power generation trend line chart, a middle real-time data dashboard, and a right-side alarm list"; Image format: User-uploaded interface prototype design drawings, screenshots of existing system interfaces, or hand-drawn sketches.
[0039] After receiving input information, the system first preprocesses the input data. For image input, the system guides Multimodal Large Language Models (MLLMs) to perform visual feature analysis on the image using preset prompts to extract the interface layout structure, control distribution, and interface function prompts. For example, MLLMs can identify visual elements such as "title bar," "line chart," "dashboard," and "list" in the image and convert them into corresponding function descriptions.
[0040] 2) The system calls a requirement analysis agent, which has built-in prompt templates based on Chain of Thought (CoT) technology to drive a general large language model to perform deep semantic analysis of functional requirements.
[0041] The requirements analysis agent parses the functional requirements of the natural language interface. During parsing, it first identifies the functional area division information, then converts the functional areas into TPRI-DataVue component types according to the platform's component mapping rules. Next, it segments the original requirements into names and descriptions corresponding to the component types. Finally, the agent outputs a list of component entities containing component names, component types, and component descriptions (e.g., if the user inputs "Generate a device operation status overview interface, including a top title bar, a left-side device distribution pie chart, a middle real-time data dashboard, and a right-side alarm list," the agent's output list would look like this). Figure 2 (As shown).
[0042] 3) After parsing, the requirements analysis agent automatically invokes the "metadata verification tool" to verify the identified component types. This verification tool is implemented based on the TPRI-DataVue platform component metadata repository, which records the names, categories, attribute definitions, and other information of all available components on the platform. It is used to determine whether a component type exists. When a component type is detected as not existing (e.g., the platform identifies "Flash Player," which is not supported by the platform), the agent automatically generates a clarification prompt (e.g., "The platform does not currently support Flash components. Should we replace it with a video player component?"), and returns confirmation or supplementary requirements information to the user through the user interaction module. The agent re-executes the semantic parsing and verification process based on the user's supplementary information until the component set meets the preset confidence requirements or reaches the maximum number of interactions (e.g., 3 times).
[0043] Step 2: Dynamic layout of the interface This step aims to automatically generate a component layout scheme based on the component metadata list and interface canvas parameters.
[0044] 1) The system first obtains the current canvas size parameters of the TPRI-DataVue interface designer, which include the canvas width and canvas height (e.g., 1920×1080). The layout information of each component is defined as a layout object containing the component position and component size. The layout object includes at least the coordinates of the top left corner of the component and the component's width and height information.
[0045] 2) The layout planning agent automatically generates the initial layout parameters for all components based on the component type and component description (e.g., ...). Figure 3As shown in the image, after the layout planning agent starts, it first generates the initial layout parameters for all components based on the component type and description, using prompting engineering techniques to drive the large language model. The design of the prompt text is crucial, for example: "You are a professional UI layout planner. The current canvas size is 1920×1080. You need to layout the following components: 1. Header, approximately 80px high; 2. Line Chart, displaying trends; 3. Gauge, highlighted; 4. ListView, displaying multiple records. Please generate reasonable initial coordinates and sizes for these components to ensure a visually appealing layout that conforms to conventional reading habits."
[0046] 3) After generating the initial layout, the layout planning agent enters an iterative optimization loop, calling the "boundary detection tool" to check whether components exceed the canvas area. The boundary detection tool determines whether a component has exceeded the boundary by comparing its coordinates and size parameters with the canvas boundary parameters. If a component is detected to have exceeded the boundary, the system feeds back the boundary information to the agent, which then adjusts the component's position or size parameters.
[0047] 4) Subsequently, the layout planning agent invokes the "collision detection tool" to detect whether there are overlapping areas between components. The collision detection tool determines whether components overlap by comparing whether their rectangular areas intersect. When component overlap is detected, the system feeds back the conflicting component identifier and conflict area information to the agent, which then eliminates the conflict by recalculating the component position parameters or adjusting the component arrangement order.
[0048] 5) The above layout detection and optimization process can be executed cyclically. After each round of optimization, the agent calls the detection tool again for evaluation until a layout scheme that meets the conditions of "no boundary crossing" and "no overlap" is generated, or the preset maximum number of optimizations (e.g., 5 times) is reached. If the conflict cannot be resolved after reaching the maximum number of optimizations, the system will return the current optimal solution and prompt the user to manually fine-tune it until a layout scheme that meets the conditions of no boundary crossing and no overlap is generated, or the maximum number of optimizations is reached.
[0049] Step 3: Component Parallel Retrieval Enhanced Generation This step aims to automatically generate component code snippets that conform to the DVSL syntax specification based on the component description information.
[0050] 1) The system has a pre-built platform asset knowledge base, which stores standard templates for TPRI-DataVue platform components, component attribute definitions, and corresponding JSON Schema syntax constraints. For each component, the code generation agent first constructs semantic query information for the component and then searches for similar component templates in the knowledge base using vector retrieval technology.
[0051] Standard templates: High-quality, manually validated DVSL component code examples selected from historical projects; JSON Schema syntax constraints: Official syntax definitions for each component type, including required attributes, optional attributes, data types, format requirements, etc. Component property definition information: Natural language explanations of each component property, making it easier for the model to understand its purpose.
[0052] 2) The code generation agent selects several high-quality templates as candidate templates based on similarity ranking results. Then, it combines the candidate templates, syntax constraint information, and component attribute descriptions into structured prompts, which are then input into the LLMs for code generation. To improve system efficiency, the agent divides the code generation tasks for multiple components into multiple parallel tasks.
[0053] 3) After code generation is complete, the code generation agent automatically calls the "syntax validation tool" to perform format validation on the generated result. The syntax validation tool uses the JSON Schema parsing mechanism to determine whether the component code conforms to the DVSL specification. When a syntax error is detected, the agent extracts the error path information and error type information, and feeds it back to the LLMs model for regeneration until the validation passes or the maximum number of retries is reached.
[0054] Taking the power generation trend line chart module as an example, the specific execution process is as follows: Constructing a query vector: The agent combines the component's type (LineChart) and attribute description (showing the trend of power generation changes over the past 24 hours, with time on the X-axis and power generation on the Y-axis) into a semantic query string.
[0055] Vector retrieval: Convert the query string into a vector, perform a similarity search in the vector database, and retrieve the top-K (e.g., K=3) most similar component templates and their associated JSON schema and attribute descriptions.
[0056] Constructing the prompt word: Combine the retrieved candidate templates, JSON Schema constraints, and the original component description into a structured prompt word. The prompt word explicitly requires the model to strictly adhere to the JSON Schema syntax, referring to the following template, and generate a DVSL code snippet for a LineChart component that can display the power generation trend over the past 24 hours, with the X-axis representing time and the Y-axis representing power generation.
[0057] Parallel LLM Invocation: The system simultaneously sends prompts from multiple components to the large language model, requesting the parallel generation of code snippets.
[0058] Syntax validation: After code generation is complete, the agent immediately invokes the "Syntax Validation Tool" in the tool service module. This validation tool is based on a JSON Schema parser and performs format and content validation on the generated JSON code. For example, it checks whether the type field is LineChart, whether the xAxis and yAxis fields exist and are correctly formatted, and whether the data field is an array, etc.
[0059] Iterative Correction: If validation fails, the syntax checker returns specific error messages (such as a missing required field xAxis.data, or "the data field should be an array, but a string was received"). The agent uses these error messages as feedback and sends them, along with the original prompt, back to the LLM, requesting regeneration. This process repeats until validation passes completely or the maximum number of retries (e.g., 3 times) is reached.
[0060] Output: After successful validation, each component generates an independent, syntactically correct DVSL code snippet.
[0061] Because the generation tasks of multiple components are executed in parallel, the overall processing time is much shorter than that required for serial generation.
[0062] Step 4: Component Merging and Interface Integration This step aims to integrate layout information with component code snippets to generate a complete UI-level DVSL configuration file.
[0063] 1) The system first constructs a unified layout container component. This layout container is used to hold all component layout parameters and component code nodes. The system writes the component position and size parameters generated by the layout planning agent into the layout container nodes, and simultaneously injects the component code snippets generated by the code generation agent into the layout container child nodes.
[0064] In TPRI-DataVue, components such as Container or GridLayout are typically used as the root node to hold other child components. The system writes the layout parameters (x, y, w, h of each component) generated in step 2 into the properties of the layout container, such as configuring the gridPos property of each child item in GridLayout.
[0065] Simultaneously, the system injects the component code snippets generated in step 3 as child nodes into the `children` array of the layout container. During injection, it is necessary to ensure that each child node corresponds one-to-one with the layout parameters, usually through component ID or key association.
[0066] The synthesized UI-level DVSL file is a fully structured JSON object. Its top-level structure contains necessary metadata (such as version and theme), and its main body consists of the layout container and its child components.
[0067] 2) After the system completes the DVSL compositing, the compositing result is transmitted to the TPRI-DataVue platform rendering engine through the integrated rendering module. Upon receiving the DVSL configuration file, the rendering engine sequentially performs data loading, structure verification, and component instantiation, and automatically draws the front-end interface on the interface designer canvas. When the rendering engine detects structural anomalies or syntax errors, the system automatically records the error information and feeds it back to the multi-agent module for correction.
[0068] Data loading involves parsing a JSON string into an in-memory object. Structure check: Perform a global syntax and structure check again to ensure there are no logical errors; Component instantiation: Based on the parsing results, various components are dynamically created and drawn on the canvas of the interface designer to form a visual front-end interface.
[0069] Step 5: User Feedback and Knowledge Base Evolution This step utilizes user feedback on modifications made in the designer to enhance the system's knowledge and evolve its capabilities.
[0070] 1) When the user adjusts and saves the automatically generated interface in the interface designer, the system automatically captures the modified DVSL code snippet and its corresponding original functional requirement description (i.e., the text or image description of the user input that triggered this generation), and constructs the captured data into instruction and correction code data pairs, and stores them in the platform knowledge base as new component template data.
[0071] 2) The system further evaluates the quality of the new template data. Evaluation metrics include user feedback, template call frequency, and template success rate. Based on the evaluation results, the system assigns priority retrieval weights to the template data. When a template's score exceeds a preset threshold, the system stores the template in the priority index area of the knowledge base. During subsequent component retrieval processes, the system prioritizes high-scoring templates, thereby continuously optimizing the interface generation effect.
[0072] User feedback and evaluation: If a user actively clicks the "like" button when making changes, that data will receive a high score; Template call frequency: If the template is called frequently by other generation tasks in the future, it indicates that its quality is high; Template success rate: The percentage of code generated based on this template that passes validation during rendering.
[0073] like Figure 4As shown, a second aspect of the present invention provides a specific implementation of a multimodal perception functional interface intelligent generation system based on TPRI-DataVue, comprising: The user interaction module, serving as the system's overall front-end entry point, provides a human-computer dialogue and visual interactive interface. This module not only supports user input of natural language commands (e.g., "Please generate an overview interface for nuclear power plant power generation"), but also integrates multimodal parsing capabilities, allowing users to upload functional prototype diagrams, UI screenshots, hand-drawn sketches, and other data. During data processing, this module supports streaming transmission, enabling real-time display of the multi-agent system's reasoning process, intermediate planning steps, and final generated results on the interface, thereby enhancing the transparency of the user experience. Furthermore, this module has a built-in closed-loop feedback interface, allowing users to directly and visually edit the generated interface and capturing user modifications.
[0074] Multi-Agent Module: As the core control unit of the system, it is built based on a state machine mechanism and is responsible for instantiating and scheduling three dedicated agents: a requirements analysis agent, a layout planning agent, and a code generation agent. During operation, this module is responsible for managing the message passing context between agents, controlling the task distribution order, and synchronizing the execution status of each agent in real time to ensure the continuity and logical consistency of the task flow from requirements understanding to code generation.
[0075] Tool Service Module: As the underlying support for system capabilities, it encapsulates various atomic tools based on the MCP standard, allowing the aforementioned specialized intelligent agents to autonomously invoke them when performing tasks. Key tools include: a "metadata verification tool" for querying the existence and attribute definitions of components in the TPRI-DataVue platform component library; "boundary detection tools" and "collision detection tools" for calculating the geometric positional relationships of components during the layout planning phase and preventing overlap and boundary overflow; and a "syntax verification tool" for checking the syntax compliance of DVSL code during the code generation phase.
[0076] The retrieval enhancement module, serving as the system's knowledge base, deploys a high-performance vector database to store and index component development documentation for the TPRI-DataVue platform, standard JSON Schema syntax constraint definitions, and selected high-quality DVSL interface or component templates. When the code generation agent is working, this module provides relevant contextual information through vector retrieval technology, thereby reducing the illusionary risk of LLMs and ensuring the standardization of generated DVSL code.
[0077] Integrated Rendering Module: Serving as the support for system data flow, this module is primarily responsible for data communication between the system and the external TPRI-DataVue platform. It transmits the DVSL code ultimately synthesized by the multi-agent module to the TPRI-DataVue platform's rendering engine for parsing and rendering, and can synchronize the latest canvas state in real time. Simultaneously, this module is also responsible for receiving rendering verification feedback information returned from the platform side, as well as modification data generated by the user on the platform side, and transmitting this information back to the multi-agent module or user interaction module, forming a complete development loop.
[0078] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can implement the operation of a multimodal perception functional interface intelligent generation method based on TPRI-DataVue.
[0079] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory). This computer-readable storage medium is a memory device within a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium of the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space containing the terminal's operating system. Furthermore, this storage space also contains one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the TPRI-DataVue-based multimodal perception functional interface intelligent generation method described in the above embodiments.
[0080] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0081] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0082] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0083] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.
[0085] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Those skilled in the art can readily implement the present invention based on the accompanying drawings and the above description. However, any modifications, alterations, or variations made by those skilled in the art without departing from the scope of the present invention, utilizing the disclosed technical content, are equivalent embodiments of the present invention. Furthermore, any modifications, alterations, or variations made to the above embodiments based on the essential technology of the present invention are still within the protection scope of the present invention.
Claims
1. A method for intelligently generating multimodal perceptual functional interfaces based on TPRI-DataVue, characterized in that, Includes the following steps: Receive multimodal input data from users, the multimodal input data including natural language description information and / or interface image information; The demand analysis agent performs intent recognition and entity extraction on the multimodal input data to generate a structured component list, which includes component types and attribute descriptions. The layout planning agent generates initial layout parameters for each component based on the canvas size and the component list, and performs conflict detection and iterative optimization on the initial layout parameters to obtain a layout scheme. The code generation agent retrieves similar component templates from the knowledge base based on the corresponding component type and attribute description for each component, and generates component code snippets that conform to the DVSL specification by combining syntactic constraints. The generated code snippets are then subjected to syntactic verification and iterative correction to obtain the component code. The layout schemes and component code of each component are merged to generate a complete UI-level DVSL file, which is then transferred to the TPRI-DataVue rendering engine for UI rendering.
2. The method for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue according to claim 1, characterized in that, The demand analysis agent performs intent recognition and entity extraction, including: When the multimodal input data contains interface images, the images are first parsed using a large multimodal model to extract implicit functional requirements. Using mind chain technology to drive a large language model to perform semantic analysis of functional requirements, identify component types and attribute descriptions; The metadata verification tool is invoked to check whether the identified component type exists in the platform component library. If the verification fails or there is ambiguity, a clarification question is generated and interacted with the user until the confidence level meets the preset threshold or the maximum number of retries is reached.
3. The method for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue according to claim 1, characterized in that, The layout planning agent performs conflict detection and iterative optimization, including: Call the boundary detection tool to check if the component exceeds the canvas area; if it does, adjust the layout parameters. Use a collision detection tool to check if there is overlap between components; if there is overlap, adjust the layout parameters. Repeat the above process until there are no layout conflicts or the maximum number of iterations is reached, and then output the final layout.
4. The method for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue according to claim 1, characterized in that, The code generation agent uses retrieval-enhanced generation technology to generate component code fragments, including: For each component, construct component semantic query information, and match the Top-K similar component templates and corresponding JSON Schema syntax constraints in the knowledge base through vector retrieval; The retrieved templates, grammatical constraints, and component attribute descriptions are combined into structured prompt words. These structured prompt words are then input into a large language model to generate code snippets for multiple components. The generated code snippet is formatted using a syntax validation tool. If the validation fails, the error message is sent back to the large language model for regeneration, until the validation passes or the maximum number of retries is reached.
5. The method for intelligent generation of multimodal perception functional interfaces based on TPRI-DataVue according to claim 1, characterized in that, It also includes the knowledge base evolution steps: When a user modifies and saves the generated interface in the TPRI-DataVue interface designer, the modified DVSL code snippet and its corresponding original functional requirements are automatically captured and constructed as a data pair of instruction-correction code. The data pairs are stored in the knowledge base as new templates, and high-quality templates are assigned priority retrieval weights to achieve continuous self-evolution of the generated effects.
6. The method for intelligent generation of multimodal perceptual functional interfaces based on TPRI-DataVue according to claim 1, characterized in that, The knowledge base evolution steps also include quality assessment of new templates. The quality assessment indicators include user feedback evaluation, template call frequency, and template usage success rate. The retrieval weight of the templates is dynamically adjusted based on the assessment results.
7. A multimodal perception-based intelligent interface generation system based on TPRI-DataVue, characterized in that, include: The user interaction module is used to receive multimodal input from users and provide human-computer dialogue and visual interaction interfaces; The multi-agent module is used to instantiate and schedule the requirements analysis agent, the layout planning agent, and the code generation agent, and to manage message passing, task distribution, and state synchronization between agents. The tool service module provides a variety of tools, including at least metadata verification tools, boundary detection tools, collision detection tools, and syntax verification tools, for the intelligent agent to call autonomously. The retrieval enhancement module stores a knowledge base containing component templates and JSON Schema syntax constraints, and provides vector retrieval support for code generation agents; The integrated rendering module is used to communicate with the TPRI-DataVue platform, transmit the generated DVSL file to the platform's rendering engine, synchronize the canvas state in real time, and receive verification feedback from the platform and user modification data.
8. The intelligent generation system for a multimodal perception functional interface based on TPRI-DataVue according to claim 7, characterized in that, The user interaction module also supports streaming output of the inference process and intermediate results of the multi-agent system, and has a built-in user feedback interface that allows users to copy, retry, visually edit, apply, and like / dislike the generated DVSL results. The tool service module encapsulates each tool using the Model Context Protocol standard, allowing the agent to call them as needed during task execution. The multi-agent module is built on a state machine mechanism to ensure that the task flow from requirement understanding to code generation is coherent and logically consistent.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the intelligent generation method for a multimodal perception functional interface based on TPRI-DataVue as described in any one of claims 1-7.
10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the intelligent generation method for a multimodal perception functional interface based on TPRI-DataVue as described in any one of claims 1-7.