An agent-driven generative AI design workflow reuse system and method

Abstract

Embodiments of the present application provide an agent-driven generative AI design workflow reuse system and method. The method uses a large language model-driven agent to generate structured tool invocation instructions based on the user's design requirements and the context engineering, so that the tool nodes and tool input data that match the user's design requirements can be displayed on the canvas. The user can also adjust the tool nodes directly on the canvas. Through multiple rounds of interaction between the agent and the user, a workflow that matches the user's intent is obtained. The above process does not require the user to understand each tool node. The workflow reuse is completed in the human-computer collaboration process, reducing the difficulty of use.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a generative AI design workflow reuse system and method driven by intelligent agents. Background Technology

[0002] With the rapid development of generative artificial intelligence technology, node graph-based visual programming interfaces have become a core tool for building, saving, and reusing design workflows in the field of digital creative design. A design workflow is essentially a structured representation of procedural design knowledge, capable of recording tool call sequences, parameter settings, and data flow during the design process, possessing high traceability and reproducibility.

[0003] Chinese patent application CN111813391A discloses a workflow generation method, apparatus, computer device, and storage medium. The method includes: displaying a workflow design interface, which includes a selection control for at least one atomic task; displaying task patterns for at least two atomic tasks in response to a selection operation on the selection control for the at least one atomic task; obtaining the hierarchical relationship between the at least two atomic tasks in response to a connection operation on the task patterns of the at least two atomic tasks; obtaining task execution parameters for the at least two atomic tasks; and generating a target workflow based on the hierarchical relationship between the at least two atomic tasks and the task execution parameters of the at least two atomic tasks. Users can design automatically executed workflows through a visual design process without requiring developers to write program code, saving code development time and improving the application efficiency of workflows.

[0004] Patent application CN116303589A discloses a workflow construction method, apparatus, device, and readable storage medium. This application can determine the target template corresponding to the workflow creation operation in a template table based on the user-input workflow creation operation, and determine each process node associated with the target template in a node table. After determining the sequence identifier of each process node and configuring node review information for each process node, a workflow instance is quickly constructed from the process nodes configured with node review information according to their sequence identifiers. This eliminates the need for repeatedly designing similar or identical workflows, improving workflow construction efficiency, and changes to the workflow will not affect the original workflow.

[0005] However, the reuse mechanism of the aforementioned patent application design workflow has significant bottlenecks. Mainstream platforms such as ComfyUI store workflows in structured data formats such as JSON. User reuse is usually limited to overall import and direct execution, which is a mechanical "data-execution" reuse. When users need to adjust the workflow according to specific needs, they must deeply understand the technical details, parameter semantics, and topological logical relationships between nodes. This process places a huge cognitive load on designers without a technical background, seriously hindering the effective flow and creative reuse of design knowledge. At the same time, existing systems lack the ability to perceive the dynamic nature of the design process, cannot understand the designer's high-level creative intentions, and cannot establish a progressive and collaborative dialogue mechanism between the designer and the system. The designer's creative process is often interrupted by technical details, making smooth human-computer collaboration difficult to achieve.

[0006] In recent years, large-scale language models have demonstrated tremendous potential in code understanding, multimodal reasoning, and task planning, providing a technological foundation for building intelligent design agents. However, how to deeply integrate intelligent agents driven by large language models into the reuse process of design workflows, enabling them not only to parse static workflow structures but also to perceive dynamic design states and assist designers in knowledge reorganization and innovation in a progressive and collaborative manner, remains a key technical challenge that urgently needs to be overcome in this field. Therefore, there is an urgent need for a new design knowledge reuse method and system that integrates agent cognition, process awareness, and progressive interaction to achieve a paradigm shift from "data reuse" to "knowledge collaboration." Summary of the Invention

[0007] This invention provides an agent-driven generative AI design workflow reuse method. This method aims to transform the reuse process of the design workflow into an intent-driven, process-aware human-computer collaborative activity by introducing an agent driven by a large language model, thereby reducing the difficulty of use.

[0008] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0009] This invention utilizes a large language model-driven agent to generate structured tool invocation instructions based on user design requirements and context engineering. This allows the canvas to intuitively display tool nodes and tool input data that closely match the user's design needs. Users can also directly adjust tool nodes on the canvas. Through multiple rounds of interaction between the agent and the user, a workflow that closely matches the user's intentions is obtained. The above process does not require the user to understand each tool node. Workflow reuse is achieved in the human-computer collaboration process, reducing the difficulty of use. Attached Figure Description

[0010] Figure 1 A flowchart illustrating an agent-driven generative AI design workflow reuse method provided in a specific embodiment of the present invention;

[0011] Figure 2 This is a block diagram of an agent-driven generative AI design workflow reuse system provided in a specific embodiment of the present invention. Detailed Implementation

[0012] This invention provides a method for reusing agent-driven generative AI design workflows, such as... Figure 1 As shown, it includes:

[0013] S1. The intelligent agent provided in the specific embodiment of the present invention provides suggestions for building workflows to users through a large language model based on the design requirements received from users, i.e., designers, and context engineering. At the same time, it issues structured tool invocation instructions, which include tool names, parameters, and tool input data. The context engineering includes user-intelligent agent interaction records, design templates that can represent user design intentions retrieved based on user semantic information, the current workflow recorded in the design template pattern, and a set of tools that can be invoked.

[0014] The Planner provided in this embodiment is a dedicated agent built on a large language model. This agent continuously receives user intent from the dialogue interface, the workflow status of the current canvas's design template format, and relevant reference design templates retrieved from the template library. Based on this multi-source information, the agent performs process awareness and incremental planning: that is, it understands the execution status of the design process and the generated intermediate results, and intelligently selects the most reasonable operation next, rather than outputting a complete but potentially inflexible process all at once.

[0015] In a specific embodiment of this invention, raw, execution-oriented design workflow data is transformed into a knowledge carrier—a design template—for agent understanding through context engineering strategies such as few-shots and thought chains. This design template is specifically designed to enhance the understanding of workflow topology and functional semantics by large language models, serving as a bridge connecting raw workflow data and agent planning.

[0016] The design templates provided in specific embodiments of this invention employ a hybrid representation that integrates code clarity, structural parsing ease, and natural language semantics. Each template consists of multiple tool entries, each corresponding to a node in the original workflow, and includes four key fields: node name, function definition, call information, and function description.

[0017] The node name provided in the specific embodiments of the present invention is a unique identifier for the tool, and nodes with the same function but different topological locations are distinguished and named accordingly.

[0018] The tool definition provided in the specific embodiments of the present invention is as follows: the input / output parameter types are declared in the form of class function signatures, such as IMAGE = ImageGeneration(model: str, prompt: str).

[0019] The specific embodiments of this invention provide invocation information that describes the specific invocation method of a node in the current workflow. Its key innovation lies in the fact that the input parameter values ​​explicitly mark the source node name, thereby explicitly encoding the data flow and topological dependencies between nodes.

[0020] The tool description provided in the specific embodiments of the present invention uses natural language to briefly describe the function and role of the tool, providing semantic context for intelligent agents.

[0021] The method for converting raw workflow data into a design template provided in a specific embodiment of the present invention includes:

[0022] Using a large language model, the nodes of the original workflow data are identified through a thought chain analysis method. Nodes with the same name but different locations and connection methods are assigned unique IDs to obtain the corresponding topological relationships. Based on the node names, the corresponding functional descriptions are obtained from the tool knowledge base through retrieval enhancement generation technology. Standardized design templates are obtained by using the large language model according to preset format specifications, as well as node names, function definitions, topological relationships, and functional descriptions. Multiple original workflow data are transformed into design templates to build a design template library.

[0023] The prompts used to generate this design template are constructed in the form of a thought chain, dividing the template generation process into the following sub-questions to form the entire thought chain:

[0024] (1) Input the raw workflow data (such as JSON), a typical example being the JSON workflow file exported from the project in ComfyUI.

[0025] (2) The first sub-problem in the thinking chain requires LLM to identify all nodes in the workflow based on specific fields of the original workflow data, record the list of node names, and assign unique IDs to nodes with the same name that have different topological roles (i.e., have different placement positions in the flowchart of the canvas and have the same name of method cards with the same name for connection logic before and after) as the basis for the number of nodes and the topological relationship in the subsequent design template construction steps.

[0026] (3) The second sub-problem in the thought chain uses the list of node names obtained in the previous step to search the tool knowledge base containing information on the nodes of each method in ComfyUI. The tool knowledge base is built using Elastic Search and uses an open-source node database to store the node names, function definitions, and method descriptions of almost all publicly available methods in ComfyUI. In this step, the agent uses retrieval enhancement generation to sequentially search for the method node that is most similar to the target method through vector matching to obtain accurate node function definitions and functional descriptions.

[0027] (4) The third step of the thinking chain is to synthesize the design template: LLM rereads the original workflow data according to the design template format example provided by the Few-shot strategy, and combines the node names, function definitions and functional descriptions obtained in the previous steps to deduce the topological relationship of the nodes, and finally synthesizes the standardized design template.

[0028] (5) The final step of the thought chain introduces a reflection mechanism, requiring the agent to reread the generated design template and the output of the preceding thought chain sub-problems, confirm the correctness of the generation node by node, and perform consistency verification and correction on the generated template.

[0029] The current workflow formed on the canvas provided in the specific embodiments of the present invention is recorded in design template mode, and the above-mentioned conversion is not required.

[0030] The specific embodiments of the present invention provide design templates that can represent user design intentions, retrieved based on user semantic information, including:

[0031] A simplified design template is formed by extracting a skeleton summary from each design template. Multiple simplified design templates are used to construct a simplified design template library. The skeleton summary includes the input type, output type, and a list of tool names. The user's natural language in the interaction record is converted into a query in the simplified design template format using a large language model. The query in the simplified design template format and the multiple simplified design templates are converted into vectors respectively. The vector of the query in the simplified design template format is used to perform an approximate nearest neighbor vector search with the vectors of the multiple simplified design templates to obtain Top-K design templates related to the user's design intent. The Top-K related design templates and the simplified design template library are filtered by the large language model to select design templates that meet the customer's needs and require the fewest supporting tools, thereby retrieving design templates that can represent the user's design intent.

[0032] In one specific embodiment, this invention uses a large language model to filter Top-K related design templates and a simplified design template library to select design templates that meet customer needs and require minimal supporting tools. The retrieved design templates represent the user's design intent. An example of the prompt words used in this large language model is:

[0033] Your current task is to select the "design template" that best suits the user's needs from the list below and return its template ID (serial number). Please reply with the ID directly.

[0034] Selection Criteria

[0035] 1. The input data type and output data type of the design template must be completely consistent with the user's requirements.

[0036] 2. The design template must at least meet the user's requirements for the type of design task (e.g., text-to-image generation, video generation, etc.).

[0037] 3. If multiple design templates still meet the above conditions, select the template that best matches the user's needs.

[0038] 4. If multiple design templates still meet the above conditions, select the template with the fewest "methods requiring external system support".

[0039]

Operating Instructions

[0040] 1. Please carefully read the list of "Design Templates + Simplified Design Templates" provided below.

[0041] 2. Based on the above criteria, compare each template in the list with the user's requirements.

[0042] 3. Select the best matching template and return its ID.

[0043] 4. Please confirm: Is the selected template truly the one that best meets the user's needs in the list? If yes, output the template ID directly; if not, please adjust the selection and output the final selected template ID.

[0044] S2. In a specific embodiment of the present invention, the intelligent agent sends the structured tool invocation instruction to the canvas, enabling the canvas to retrieve tools from the template and asset library and configure parameters, and / or tool input data, thereby forming tool nodes on the canvas. This allows the user to adjust the position, connection relationship, or tool parameters of the tool nodes on the canvas, thereby forming the current workflow constructed by the tool nodes on the canvas. The intelligent agent then provides the intelligent agent with design requirements including determining the current tool node and providing the next tool node or replacing the tool node. Through multiple rounds of interaction between the intelligent agent and the user, a final workflow constructed by multiple tool nodes is formed on the canvas.

[0045] A specific embodiment of the present invention also provides a second path: template-driven intelligent assisted modification. Designers select approximate templates from the template library, which can be directly imported into the canvas for manual editing, or submitted to the intelligent agent as a collaborative benchmark. In collaborative mode, designers guide the modification direction through dialogue, and the intelligent agent guides them to replace nodes, adjust parameters, and optimize structures, achieving precise iteration of the workflow.

[0046] The agent-driven generative AI design workflow reuse method provided in this invention generates "design templates" through a specialized workflow parsing and enhancement process. It utilizes embedded models and vector retrieval to achieve semantic matching between design intent and the template library, and constructs a multi-source context integration mechanism to enable the planning agent to make inference decisions by comprehensively considering canvas state, dialogue history, and tool information. By treating the agent as a collaborative partner in the design process, this invention significantly reduces the technical threshold and cognitive burden of workflow reuse, enhances the flexibility and creativity of design exploration, and provides an efficient and intelligent human-machine collaborative solution for generative AI-assisted design.

[0047] On the other hand, the present invention also provides an agent-driven generative AI design workflow reuse system, such as Figure 2 As shown, it includes intelligent agents, canvases and templates, and an asset library.

[0048] The intelligent agent provided in the specific embodiments of the present invention is used to provide suggestions for building workflows to users through a large language model based on the design requirements received from users and context engineering, and at the same time issue structured tool invocation instructions. The context engineering includes the interaction records between users and intelligent agents, templates that can represent the user's design intentions retrieved based on the user's semantic information, the current workflow, and a set of tools that can be invoked.

[0049] The canvas provided in this specific embodiment of the invention retrieves tools and configures parameters from templates and asset libraries based on the received structured tool call instructions, forming tool nodes on the canvas. This allows the user to adjust the position, connection relationship, or tool parameters of the tool nodes on the canvas, thereby forming the current workflow constructed by the tool nodes on the canvas. The design requirements, including determining the current tool node and providing the next tool node or replacing the tool node, are then provided to the intelligent agent. Through multiple rounds of interaction between the intelligent agent and the user, the final workflow constructed by multiple tool nodes is formed on the canvas.

[0050] Specifically, the canvas serves as a space for workflow visualization and construction. It allows users to freely drag and drop tool nodes, connect pipelines, and adjust parameters to edit workflows. After the user agrees in a dialogue, the intelligent agent can automatically add methods and asset cards based on the generated design scheme. Asset cards serve as inputs for the tools, such as images or text on images, while method cards serve as tools to build workflows. The user then executes the cards and indicates the next step, forming a progressive cycle of "suggestion-confirmation-execution-replanning".

[0051] The template and asset library provided in this specific embodiment of the invention consists of two parts: first, metadata for all generative AI tools (corresponding workflow nodes) supported by the system, including tool names, functional descriptions, and input / output parameter formats; and second, a knowledge base storing a large number of "design templates." Design templates are structured and semantically enhanced representations of the original workflow (such as ComfyUI JSON files), facilitating the agent's understanding of its topological logic and functional goals.

[0052] The agent-driven generative AI design workflow reuse system provided in the specific embodiments of the present invention also includes a visual interactive interface and a system integration module.

[0053] This module is responsible for implementing a user-friendly interface and facilitating the data flow between the front-end and back-end.

[0054] Front-end implementation: A component-based user interface is developed using the React framework, including a dialog panel, library browser, and dual-mode canvas. A professional graphics library (such as AntV G6) is used to render the workflow node graph, supporting free editing on canvas A and intelligent guided visualization on canvas B. The agent provides visual prompts on canvas B using highlighting, animation, or preset positions to reduce the user's cognitive load.

[0055] Front-end and back-end communication: Design a dialogue interface for transmitting user messages and receiving structured planning suggestions from the agent. Design a state synchronization interface to ensure that any changes to the canvas are promptly reported to the back-end agent engine. Design a template retrieval interface to process user intents and return relevant template information.

[0056] Through the synergy of the above modules, the specific embodiments of the present invention construct a complete technical system from design knowledge representation, intelligent understanding, dynamic retrieval to human-machine progressive collaboration, providing a powerful intelligent auxiliary framework for creative design work in the era of generative AI.

Claims

1. A method for reusing generative AI design workflows driven by intelligent agents, characterized in that, The agent provides workflow construction suggestions to the user based on the received design requirements and context engineering from the user through a large language model, and issues structured tool invocation instructions. The context engineering includes the user's interaction records with the agent, the design template that can represent the user's design intent retrieved based on the user's semantic information, the current workflow recorded in the design template pattern, and the set of tools that can be invoked. The agent sends the structured tool invocation command to the canvas, enabling the canvas to retrieve tools from the template and asset library and configure parameters, and / or tool input data, thereby forming tool nodes on the canvas. This allows the user to adjust the position, connection relationship, or tool parameters of the tool nodes on the canvas, thus forming the current workflow built by the tool nodes on the canvas. The agent then provides the agent with design requirements, including determining the current tool node and providing the next tool node or replacing the tool node. Through multiple rounds of interaction between the agent and the user, a final workflow built by multiple tool nodes is formed on the canvas. A method for converting raw workflow data into design templates, wherein the templates and asset library include multiple sets of raw workflow data, including: Using a large language model, the nodes of the original workflow data are identified through the chain of thought analysis method. Nodes with the same name but different locations and connection methods are assigned unique IDs to obtain the corresponding topological relationships. The corresponding functional descriptions are obtained from the tool's knowledge base based on the names of each node through retrieval enhancement generation technology. By using a large language model to obtain standardized design templates according to preset format specifications, as well as node names, function definitions, topological relationships, and functional descriptions, multiple original workflow data are transformed into design templates to build a design template library.

2. The agent-driven generative AI design workflow reuse method according to claim 1, characterized in that, The design template includes node names, function definitions, call information, and function descriptions.

3. The agent-driven generative AI design workflow reuse method according to claim 1, characterized in that, The current workflow formed on the canvas is recorded in design template mode.

4. The agent-driven generative AI design workflow reuse method according to claim 1, characterized in that, Design templates retrieved based on user semantic information that represent the user's design intent include: A simplified design template is formed by extracting a skeleton summary from each design template. Multiple simplified design templates are used to build a simplified design template library. The skeleton summary includes the input type, output type, and a list of tool names. The user's natural language in the interaction record is converted into a query in a simplified design template format using a large language model; The query for the simplified design template format and multiple simplified design templates are converted into vectors respectively. The vector of the query for the simplified design template format is then subjected to an approximate nearest neighbor vector search with the vectors of multiple simplified design templates to obtain the Top-K design templates that are relevant to the user's design intent. The Top-K related design templates and simplified design template library are filtered through a large language model to select design templates that meet customer needs and require minimal supporting tools, thereby retrieving design templates that can represent the user's design intent.

5. The agent-driven generative AI design workflow reuse method according to claim 1, characterized in that, The intelligent agent is also used to directly import the design templates selected by the user from the design template library into the canvas, and form the workflow corresponding to the design template on the canvas.

6. The agent-driven generative AI design workflow reuse method according to claim 1, characterized in that, The structured tool invocation command includes the tool name, parameters, and tool input data.

7. A generative AI design workflow reuse system driven by an intelligent agent, characterized in that, include: An intelligent agent is used to provide workflow construction suggestions to the user based on the received design requirements and context engineering from the user through a large language model, and at the same time issue structured tool invocation instructions. The context engineering includes the user's interaction records with the intelligent agent, a template that can represent the user's design intent retrieved based on the user's semantic information, the current workflow, and a set of tools that can be invoked. A method for converting raw workflow data into design templates, wherein the templates and asset library include multiple sets of raw workflow data, including: Using a large language model, the nodes of the original workflow data are identified through the chain of thought analysis method. Nodes with the same name but different locations and connection methods are assigned unique IDs to obtain the corresponding topological relationships. Based on the node names, the corresponding functional descriptions are obtained from the tool knowledge base through retrieval enhancement generation technology; standardized design templates are obtained through a large language model according to preset format specifications, as well as node names, function definitions, topological relationships and functional descriptions, and multiple original workflow data are transformed into design templates to build a design template library; The canvas, based on the received structured tool call instructions, retrieves tools from templates and asset libraries and configures parameters, forming tool nodes on the canvas. This allows the user to adjust the position, connection relationship, or tool parameters of the tool nodes on the canvas, thereby forming the current workflow constructed by the tool nodes on the canvas. The user then provides the agent with design requirements, including determining the current tool node and providing the next or replacement tool node. Through multiple rounds of interaction between the agent and the user, the final workflow constructed by multiple tool nodes is formed on the canvas.

Images