Intelligent canvas interaction system and method based on AIAgent

Abstract

The application discloses an intelligent canvas interaction system and method based on AIAgent. The system comprises a front-end canvas module, a ChatWebSocket channel, a BridgeWebSocket channel, an AIAgent decision module, a tool registration and routing module, and a front-end tool executor. The system assigns an execution position identifier to each tool in the tool set and defines a routing decision function. The canvas state is synchronized using an incremental superposition mechanism, and the layer set, element set, and configuration parameters are updated according to a formula after the first tool call. The system supports the canvas_state tool for AI to obtain the current canvas state in real time to assist in parameter decision-making, and the canvas_preview tool for performing a screenshot on the canvas and returning a base64 encoded image for AI visual verification. The method comprises the steps of instruction receiving, semantic analysis, tool routing, cross-end execution, result summarization, and response generation. The application realizes natural language driven multi-step canvas operation, effectively reducing the user threshold.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction and artificial intelligence application technology, and more particularly to an intelligent canvas interaction system and method based on AIAgent. Background Technology

[0002] With the rapid growth in demand for graphic design and digital creation, canvas editing software has become an important tool for content production. However, existing canvas systems generally rely on manual operations such as mouse drag-and-drop and menu selection as their core interaction paradigm. Users need to master multiple professional skills, including layer management, coordinate positioning, and resource retrieval, to complete a full creative process. This highly manual interaction mode not only has a steep learning curve and cumbersome operation steps, but also poses a significant barrier to entry for non-professional users, severely hindering the popularization and efficiency improvement of canvas tools.

[0003] To reduce operational complexity, existing technologies attempt to combine large language models with graphics editing tools. However, such solutions generally face the following core problems: On the one hand, core canvas operations (such as layer rendering, element positioning, screenshot preview, etc.) must be executed in the browser front-end environment, while AI inference and tool scheduling usually run on the server side. The lack of a unified dynamic routing mechanism between the two leads to a disconnect between front-end and back-end execution boundaries, making it impossible for AI to transparently schedule cross-platform tools. On the other hand, existing solutions fail to provide AI with real-time canvas state awareness. When planning multi-step tool calls, AI cannot obtain contextual data such as the current layer coordinates and size, resulting in inaccurate parameter calculations and the inability to self-verify operation results, making it difficult to support complex multi-round iterative interaction scenarios. Summary of the Invention

[0004] The purpose of this invention is to provide an AIAgent-based intelligent canvas interaction system and method to solve the problems existing in the prior art.

[0005] This invention provides an AIAgent-based intelligent canvas interaction system, comprising:

[0006] The front-end canvas module receives natural language text commands input by the user and presents the canvas interface;

[0007] The ChatWebSocket channel connects the front-end canvas module and the back-end AIAgent decision module to transmit user command messages and natural language response text generated by the AIAgent.

[0008] Tool registration and routing module, complete set of storage tools and for each tool Assign execution location identifier ,in This indicates that the tool is executed on a Python server. This indicates that the tool needs to be forwarded to the browser frontend for execution;

[0009] The AIAgent decision-making module performs semantic analysis on received user commands, based on the complete toolkit. The planning tool invokes a sequence of calls, based on the routing decision function. Routing each tool in the tool call sequence one by one;

[0010] The BridgeWebSocket channel connects the AIAgent decision-making module and the front-end tool executor, enabling bidirectional data transmission between the back-end and front-end. The tool invocation request and its execution result;

[0011] The front-end tool executor receives tool call requests forwarded via the BridgeWebSocket channel, distributes the requests to the corresponding canvas operation processing functions according to the pre-registered tool processor mapping table, and sends the execution results back to the AIAgent decision module via the BridgeWebSocket channel.

[0012] Through the above technical solutions: the dual-channel decoupled architecture improves the system's scalability and concurrent processing capabilities; the tool routing decision mechanism eliminates the front-end and back-end execution boundaries, enabling transparent scheduling of cross-platform tools; and natural language-driven end-to-end interaction significantly reduces the threshold for canvas operation.

[0013] Furthermore, the routing decision function Defined as: ,when At that time, the AIAgent decision-making module directly calls the tool on the Python server. The processing function and synchronously obtain the return value; when At that time, the AIAgent decision module will use the tool name. , calling parameters and unique request identifier Encapsulated as a JSON message, it is asynchronously forwarded to the front-end tool executor via a BridgeWebSocket channel. After the front-end tool executor completes the canvas operation, it returns the original... Execution status field and results data It is encapsulated as a JSON message and sent back via the BridgeWebSocket channel.

[0014] Furthermore, the front-end tool executor updates the canvas state using an incremental state overlay mechanism after each canvas operation:

[0015] No. Snapshot of the canvas state before the tool call is executed. From layer collection Element set and canvas configuration parameter vector It consists of three parts; the first After this tool is invoked, the canvas state is updated according to the following formula: ,in This is the state increment generated by this call. For layer change collection, This is a set of changes in the position and size of elements. A vector for changing configuration parameters such as canvas width and height. This is an overlay operation that merges the increments into the current state.

[0016] Furthermore, the complete set of tools Includes execution location identifier Tools; when the AIAgent decision-making module plans in the tool invocation sequence When using a tool, the front-end tool executor will display the current canvas state. Layer collection in Element set and canvas configuration parameter vector The data is serialized into JSON format and returned to the AIAgent decision-making module via the BridgeWebSocket channel. The AIAgent decision-making module uses this JSON data to calculate parameters and make decisions for subsequent tool calls.

[0017] Furthermore, the complete set of tools Includes execution location identifier of Tools; when the AIAgent decision-making module plans in the tool invocation sequence When using the tool, the front-end tool executor performs a screenshot operation on the current canvas, encodes the screenshot data into a base64 format PNG image, and then encapsulates it into... The field value is "image". Fields contain Image-type messages containing "image / png" and base64 encoded data are returned to the AIAgent decision module via the BridgeWebSocket channel. The AIAgent decision module analyzes the screenshot content using visual understanding capabilities to confirm whether the operation result meets the user's instructions.

[0018] The AIAgent-based intelligent canvas interaction method includes the following steps:

[0019] S001, the user inputs natural language text commands through the front-end interactive interface, and the front-end encapsulates the commands into Chat message format and transmits them to the back-end AIAgent through the ChatWebSocket channel;

[0020] S002, the backend AIAgent performs semantic parsing on the received user commands, identifies the user's intent, and bases its decisions on the full set of available tools. Plan a complete sequence of tool calls;

[0021] S003, AIAgent sequentially retrieves the next tool to be invoked from the tool invocation sequence. The query tool retrieves its execution location identifier from the registry. Substitute into the routing decision function Calculate the route target;

[0022] S004, when At that time, AIAgent directly calls the tool on the Python server. The processing function takes in the parameters generated during the planning phase, synchronously obtains the execution result, and writes the result into the tool call chain context;

[0023] S005, when At that time, AIAgent will , and unique identifier The message is encapsulated in JSON format and sent asynchronously to the front-end tool executor via the BridgeWebSocket channel.

[0024] S006, After the front-end tool executor receives the Bridge message, it parses it. Based on the pre-registered tool processor mapping table, the field dispatches the call request to the corresponding canvas operation processing function to complete the actual canvas operation;

[0025] S007, the front-end tool executor will original Execution status field and results data Encapsulate it as a JSON message and send it back to the backend AIAgent via the BridgeWebSocket channel;

[0026] S008, after receiving the execution result of the front-end tool, AIAgent incorporates it into the tool call chain context and determines whether the tool call sequence has been completely executed. If not, it returns to step S003 to continue processing the next tool. If it has been completed, it proceeds to step S009.

[0027] S009, AIAgent generates natural language response text based on the execution result of the complete tool call chain, and sends it to the front end via the ChatWebSocket channel. The front end renders and displays the response text in the dialog interface.

[0028] Further, in step S002, when the user instruction is to create an image poster, the tool call sequence planned by AIAgent is as follows: ;

[0029] Tool execution location identifier AIAgent calls this tool on the Python server side, passing in... and The parameters perform a database retrieval, extracting the target from the returned resource list. And write it to the tool call chain context;

[0030] Tool execution location identifier AIAgent will contain and The parameter call request is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor calls the canvas size setting processing function to adjust the canvas width and height to the specified pixel values.

[0031] Tool execution location identifier AIAgent will contain , coordinates and The coordinate request is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor then... Load the corresponding image resources, create a new layer at the specified coordinates and complete image rendering. The canvas state is determined according to the formula. Update, in which It contains all the attribute data for the newly added layer.

[0032] Furthermore, in step S002, when the user instruction involves adjusting the position or size of an existing layer on the canvas, the tool call sequence planned by AIAgent is as follows: ;

[0033] S010, AIAgent plans in the tool invocation sequence When using the tool, the front-end tool executor will display the complete state of the current canvas. The layer collection, element collection, and canvas configuration parameters are serialized into JSON format and returned via a BridgeWebSocket channel. The returned data includes the layer's... Current coordinates and current width With height AIAgent parses the above JSON data according to the formula. Calculate the target state increment Determine the updated coordinates and dimensions;

[0034] Tool execution location identifier AIAgent will contain The request for the target coordinates and dimensions is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor calls the element update processing function to update the specified layer coordinates and dimensions to the target values. The canvas state is determined according to the formula. Complete synchronization, among which Includes all change data for coordinates and dimensions;

[0035] S011, AIAgent plans in the tool invocation sequence When the tool is in use, the front-end tool executor performs a screenshot operation on the current canvas, encodes the screenshot data into a base64 format PNG image, encapsulates it into an image type message, and returns it through the BridgeWebSocket channel. AIAgent analyzes the screenshot using its visual understanding capabilities to confirm whether the layer position and size are consistent with the user's instructions.

[0036] Specifically, in step S005, AIAgent is used for each... The operation generates a unique request identifier. , and , Together they are encapsulated into a JSON message and sent to the front-end tool executor via the BridgeWebSocket channel; in step S007, the front-end tool executor will... Follow and The data is packaged and sent back together, and AIAgent follows accordingly. After matching the returned result with the corresponding tool call request, it is included in the tool call chain context, and then the completion judgment of step S008 is performed.

[0037] The beneficial effects of this invention are:

[0038] 1. This invention completely decouples the communication links between the user dialogue layer and the tool execution layer by introducing independent ChatWebSocket and BridgeWebSocket channels. The Chat channel is dedicated to the sending and receiving of natural language commands and the downlink push of response text, while the Bridge channel is dedicated to the bidirectional asynchronous transmission of tool call requests and execution results. The two do not interfere with each other, enabling the system to continuously respond to user interactions while processing multi-step tool call chains, significantly improving concurrency processing capabilities and architectural scalability.

[0039] 2. This invention provides a complete set of tools. Each tool in the process is assigned an execution location identifier. And define the routing decision function. This allows AIAgent to plan tool call sequences without needing to be aware of the underlying execution location of individual tools; the system automatically... The tool can be directly called on the Python server side, The tools are asynchronously forwarded to the browser front-end executor via the BridgeWebSocket channel, realizing unified scheduling and transparent routing of server-side logic tools and browser-side canvas operation tools, eliminating architectural barriers to cross-platform tool integration.

[0040] 3. This invention enables users to drive the system to complete the entire automated process of semantic parsing, tool sequence planning, cross-terminal routing execution, and result aggregation simply by using natural language text commands. AIAgent maps user intent into a structured tool call sequence and routes and executes them one by one. Finally, it provides feedback on the operation results to the user in natural language response text. There is no need for users to manually operate the canvas interface, which significantly reduces the usage threshold of complex multi-step creation processes. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a diagram illustrating the architecture of the AIAgent-based intelligent canvas interaction system of the present invention.

[0043] Figure 2 This is a flowchart of the AIAgent-based intelligent canvas interaction method of the present invention;

[0044] Figure 3 This is a flowchart illustrating the process of generating a cartoon cat poster, as described in Embodiment 1 of the present invention.

[0045] Figure 4 This is a flowchart illustrating the adjustment of layer position and size, which is a second embodiment of the present invention. Detailed Implementation

[0046] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0047] It should be noted that the use of terms such as "an embodiment," "an embodiment," "an exemplary embodiment," and "some embodiments" in the specification indicates that the described embodiment may include a specific feature, structure, or characteristic, but not every embodiment necessarily includes that specific feature, structure, or characteristic. Furthermore, when a specific feature, structure, or characteristic is described in connection with an embodiment, implementing such a feature, structure, or characteristic in conjunction with other embodiments (whether explicitly described or not) should be within the knowledge of those skilled in the art.

[0048] Generally, terms can be understood at least partly from their use in context. For example, depending at least partly on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in a singular sense, or a combination of features, structures, or characteristics in a plural sense. Additionally, the term "based on" can be understood not necessarily to convey an exclusive set of factors, but rather, alternatively, depending at least partly on the context, to allow for the presence of other factors that are not necessarily explicitly described.

[0049] See Figures 1 to 4 As shown

[0050] The AIAgent-based intelligent canvas interaction system of this invention consists of a front-end canvas module, a ChatWebSocket channel, a BridgeWebSocket channel, an AIAgent decision module, a tool registration and routing module, and a front-end tool executor. Each module works together to complete the entire chain of processing from natural language commands to canvas operations.

[0051] Tool routing decision is the core mechanism of this invention. Let the complete set of registered tools in the system be . Each tool An execution location identifier is assigned during the registration phase. ,in This indicates that the tool is executed directly on the Python server. This indicates that the tool needs to be forwarded to the browser frontend for execution via a BridgeWebSocket channel. AIAgent supports any tool. The routing decision function is defined as follows:

[0052]

[0053] in This indicates that the tool is called directly on the server side. The processing function and synchronously obtain the return value. Indicates the tool Name, parameters and unique request identifier After being encapsulated as a JSON message, it is asynchronously forwarded to the front-end executor via the BridgeWebSocket channel.

[0054] Multi-step synchronization of canvas states adopts an incremental state overlay mechanism, let the first step be... A snapshot of the canvas state before the tool call was executed. It consists of a collection of layers. Element set and canvas configuration parameter vector It consists of three parts, then the first The formula for updating the canvas state after this tool call is completed is:

[0055]

[0056] in For the first The state increment generated by this tool call. This is the set of layer changes caused by this call. This is a set of changes in the position and size of elements. A vector for changing configuration parameters such as canvas width and height. This indicates an overlay operation that merges the increments into the current state.

[0057] The detailed steps of the overall system workflow are as follows.

[0058] S001 Users input natural language text commands through the front-end interactive interface. The front-end encapsulates the commands into a standard Chat message format and transmits them to the back-end AIAgent via the ChatWebSocket channel.

[0059] After receiving user commands, the S002 backend AIAgent performs semantic parsing to identify user intent and then uses the available toolset. Plan a complete sequence of tool calls.

[0060] S003AIAgent sequentially retrieves the next tool to be invoked from the tool invocation sequence. The query tool retrieves its execution location identifier from the registry. Substitute into the routing decision function Calculate the route target.

[0061] S004 At that time, AIAgent directly calls the tool on the Python server. The processing function takes in the parameters generated during the planning phase, synchronously obtains the execution result, and writes the result into the context record of the current tool call chain.

[0062] S005 At that time, AIAgent will name the tool. , calling parameters and unique identifier The message is encapsulated in JSON format and sent asynchronously to the front-end tool executor via the BridgeWebSocket channel.

[0063] After receiving the Bridge message, the S006 front-end tool executor parses it. Based on the pre-registered tool processor mapping table, the field dispatches the call request to the corresponding canvas operation processing function to complete the actual canvas operation.

[0064] After the S007 front-end tool executor completes the canvas operation, it will... Execution status field and results data It is encapsulated as a JSON message and sent back to the backend AIAgent via the BridgeWebSocket channel.

[0065] After receiving the execution result of the front-end tool, S008AIAgent incorporates the result into the tool call chain context and determines whether the entire call sequence has been executed. If not, it returns to S003 to continue processing the next tool; if it has been completed, it enters the response generation stage.

[0066] S009AIAgent generates natural language response text based on the execution results of the complete tool call chain, and sends it to the front end via the ChatWebSocket channel. The front end then renders and displays the response text in the dialog interface.

[0067] S010 When the AIAgent schedules in the tool invocation sequence When using the tool, the front-end tool executor will display the complete state of the current canvas. After being serialized into JSON format, the data is returned. AIAgent uses this status data to calculate parameters and make decisions for subsequent tool calls.

[0068] S011 When the AIAgent schedules in the tool invocation sequence When the tool is in use, the front-end tool executor performs a screenshot operation on the current canvas, encodes the screenshot data into a base64 format PNG image, encapsulates it into an image type message and returns it. AIAgent analyzes the screenshot content through visual understanding capabilities to confirm whether the operation result meets the user's instructions.

[0069] Example 1: Generating a Cartoon Cat Poster Based on Natural Language Commands

[0070] This embodiment describes the detailed execution process of the system completing resource retrieval, canvas settings, and layer addition after the user inputs the command "create a cartoon cat poster".

[0071] When an S101 user enters the text command "Create a cartoon cat poster" in the front-end dialog interface, the front-end encapsulates the command as a Chat message and sends it to the back-end AIAgent via the ChatWebSocket channel.

[0072] S102AIAgent performs semantic analysis on the commands, identifying three types of operations that need to be completed sequentially: resource retrieval, canvas size setting, and layer addition. The planned tool call sequence is as follows: .

[0073] S103AIAgent retrieves the first tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function This tool is invoked on the Python server side, passing in parameters. For image type, For the search term "cartoon, cat", the server performs a database search and returns a list of resources. AIAgent then extracts the target resource identifier from the list. And write the value to the tool call chain context.

[0074] S104AIAgent retrieves the second tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function ,Will For "set_view", Include and and unique identifier It is encapsulated as a JSON message and sent to the front-end tool executor via the BridgeWebSocket channel.

[0075] After receiving the Bridge message, the S105 front-end tool executor parses it. The field locates the canvas size setting function, adjusting both the canvas width and height to 1024 pixels. The canvas state is then set according to the formula. Update, in which Includes width and height change data; after the operation is complete, it will... and original It is encapsulated as a JSON message and returned to AIAgent via the BridgeWebSocket channel.

[0076] S106AIAgent receives The tool successfully returned a result; retrieve the third tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function ,Will For "add_layer", Include , , and unique identifier It is encapsulated as a JSON message and sent to the front-end tool executor via the BridgeWebSocket channel.

[0077] After receiving the Bridge message, the S107 front-end tool executor parses it. The field is located to the layer to add a processing function, based on... Load the corresponding cartoon cat image resource, in the canvas coordinates Create a new layer and complete image rendering. The canvas state is set according to the formula. Update, in which It contains all the attribute data of the newly added layer, and the rendering result is displayed in real time on the front-end canvas.

[0078] The S108 front-end tool executor will The execution result of the operation is returned to AIAgent via the BridgeWebSocket channel. AIAgent determines that the entire sequence of tool calls has been executed and generates a natural language response: "A cartoon cat layer has been added to the center of the canvas, and the canvas size has been set to 1024×1024 pixels." This response is then sent to the front-end dialog interface via the ChatWebSocket channel for display.

[0079] Example 2: Layer position and size adjustment based on canvas state awareness

[0080] This embodiment describes the detailed execution process of a state-aware multi-step operation after the user inputs the command "move the layer in the canvas to the top left corner and shrink it to half its original size". The system completes the operation by obtaining the current canvas state, updating elements, and verifying the visual screenshot.

[0081] S201 When a user enters the text command "move the layer in the canvas to the top left corner and shrink it to half its original size" in the front-end dialog interface, the front-end encapsulates the command as a Chat message and sends it to the back-end AIAgent via the ChatWebSocket channel.

[0082] S202 AIAgent performs semantic parsing of the commands, recognizing that it needs to first read the current canvas state to obtain the layer's position and size baseline values, then perform element updates, and finally perform visual verification through screenshots. The planning tool call sequence is as follows: .

[0083] S203 AIAgent retrieves the first tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function The call request is sent to the front-end tool executor via the BridgeWebSocket channel.

[0084] After receiving the request, the S204 front-end tool executor reads the current canvas state data and sets the layer collection. Element set With canvas configuration parameter vector The data is serialized into JSON format and returned to AIAgent via a BridgeWebSocket channel. The returned data includes the layer identifier. Current coordinates Current width Current height .

[0085] S205 AIAgent Analysis The returned JSON data is used to read the layers. Given the current position coordinates and dimensions, calculate the target state increment based on the incremental state update formula. Determine the target coordinates as The target size is The above calculation results are written into the tool call chain context.

[0086] S206 AIAgent retrieves the second tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function ,Will For "update_element", Include and unique identifier It is encapsulated as a JSON message and sent to the front-end tool executor via the BridgeWebSocket channel.

[0087] After receiving this message, the S207 front-end tool executor parses it. The field is located to the element update processing function, and The layer coordinates are updated to And scale the layer's width and height to 100 pixels, with the canvas state determined by the formula. Complete synchronization, among which It contains all the changed data of coordinates and dimensions, and the front-end canvas renders the updated layer status in real time.

[0088] The S208 front-end tool executor will The execution result of the operation is returned to AIAgent via the BridgeWebSocket channel. After receiving the successful result, AIAgent retrieves the third tool in the call sequence. The execution location identifier can be obtained by querying the registry. Execute according to the routing decision function The screenshot request is sent to the front-end tool executor via the BridgeWebSocket channel.

[0089] After receiving a screenshot request, the S209 front-end tool executor calls the canvas screenshot processing function to capture the complete rendering result of the current canvas as a PNG image, encodes the image data into a base64 string, and encapsulates it into a... The field value is "image". Fields contain The message containing "image / png" and base64 encoded data is returned to AIAgent via the BridgeWebSocket channel.

[0090] S210 AIAgent receives The returned base64 image data is analyzed using visual understanding capabilities to confirm that the layer is located in the upper left corner of the canvas and its size has been reduced to half of its original value. The results show that the operation is completely consistent with the user's instructions, and the entire tool call sequence has been executed.

[0091] The S211 AIAgent natural language response has been moved to the top left corner of the canvas, and the coordinates have been adjusted. The image size has been reduced to 100×100 pixels, and the visual screenshot has been confirmed as completed. This image is then sent to the front-end chat interface for display via the ChatWebSocket channel.

[0092] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0093] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An AI Agent-based intelligent canvas interaction system, characterized in that, include: The front-end canvas module receives natural language text commands input by the user and presents the canvas interface; The ChatWebSocket channel connects the front-end canvas module and the back-end AIAgent decision module to transmit user command messages and natural language response text generated by the AIAgent. Tool registration and routing module, complete set of storage tools and for each tool Assign execution location identifier ,in This indicates that the tool is executed on a Python server. This indicates that the tool needs to be forwarded to the browser frontend for execution; The AIAgent decision-making module performs semantic analysis on received user commands, based on the complete toolkit. The planning tool invokes a sequence of calls, based on the routing decision function. Routing each tool in the tool call sequence one by one; The BridgeWebSocket channel connects the AIAgent decision-making module with the front-end tool executor, enabling bidirectional transmission between the back-end and front-end. The tool call request and its execution result; The front-end tool executor receives tool call requests forwarded via the BridgeWebSocket channel, distributes the requests to the corresponding canvas operation processing functions according to the pre-registered tool processor mapping table, and sends the execution results back to the AIAgent decision module via the BridgeWebSocket channel.

2. The system according to claim 1, characterized in that, The routing decision function Defined as: ,when At that time, the AIAgent decision-making module directly calls the tool on the Python server side. The processing function and synchronously obtain the return value; when At that time, the AIAgent decision module will use the tool name. , calling parameters and unique request identifier Encapsulated as a JSON message, it is asynchronously forwarded to the front-end tool executor via the BridgeWebSocket channel. After completing the canvas operation, the front-end tool executor returns the original... Execution status field and results data The message is encapsulated as a JSON message and sent back via the BridgeWebSocket channel.

3. The system according to claim 1, characterized in that, The front-end tool executor updates the canvas state using an incremental state overlay mechanism after each canvas operation is completed. No. Snapshot of the canvas state before the tool call is executed. From layer collection Element set and canvas configuration parameter vector It consists of three parts; the first After this tool is invoked, the canvas state is updated according to the following formula: ,in This is the state increment generated by this call. For layer change collection, This is a set of changes in the position and size of elements. A vector for changing configuration parameters such as canvas width and height. This is an overlay operation that merges the increments into the current state.

4. The system according to claim 1, characterized in that, The complete set of tools Includes execution location identifier Tools; when the AIAgent decision module plans in the tool invocation sequence When using the tool, the front-end tool executor will display the current canvas state. Layer collection in Element set and canvas configuration parameter vector The data is serialized into JSON format and returned to the AIAgent decision module via the BridgeWebSocket channel. The AIAgent decision module then uses this JSON data to calculate parameters and make decisions for subsequent tool calls.

5. The system according to claim 1, characterized in that, The complete set of tools Includes execution location identifier of Tools; when the AIAgent decision module plans in the tool invocation sequence When using the tool, the front-end tool executor performs a screenshot operation on the current canvas, encodes the screenshot data into a base64 format PNG image, and then encapsulates it into... The field value is "image". Fields contain Image type messages containing "image / png" and base64 encoded data are returned to the AIAgent decision module via the BridgeWebSocket channel. The AIAgent decision module analyzes the screenshot content using visual understanding capabilities to confirm whether the operation result meets the user's instruction requirements.

6. An AIAgent-based intelligent canvas interaction method, characterized in that, Includes the following steps: S001, the user inputs natural language text commands through the front-end interactive interface, and the front-end encapsulates the commands into Chat message format and transmits them to the back-end AIAgent through the ChatWebSocket channel; S002, the backend AIAgent performs semantic parsing on the received user commands, identifies the user's intent, and bases its decisions on the full set of available tools. Plan a complete sequence of tool calls; S003, AIAgent sequentially retrieves the next tool to be invoked from the tool invocation sequence. The query tool retrieves its execution location identifier from the registry. Substitute into the routing decision function Calculate the route target; S004, when At that time, AIAgent directly calls the tool on the Python server. The processing function takes in the parameters generated during the planning phase, synchronously obtains the execution result, and writes the result into the tool call chain context; S005, when At that time, AIAgent will , and unique identifier The message is encapsulated in JSON format and sent asynchronously to the front-end tool executor via the BridgeWebSocket channel. S006, After the front-end tool executor receives the Bridge message, it parses it. Based on the pre-registered tool processor mapping table, the field dispatches the call request to the corresponding canvas operation processing function to complete the actual canvas operation; S007, the front-end tool executor will the original Execution status field and results data Encapsulate it as a JSON message and send it back to the backend AIAgent via the BridgeWebSocket channel; S008, after receiving the execution result of the front-end tool, AIAgent incorporates it into the tool call chain context and determines whether the tool call sequence has been completely executed. If not, it returns to step S003 to continue processing the next tool. If it has been completed, it proceeds to step S009. S009, AIAgent generates natural language response text based on the execution result of the complete tool call chain, and sends it to the front end via the ChatWebSocket channel. The front end renders and displays the response text in the dialog interface.

7. The method according to claim 6, characterized in that, In step S002, when the user instruction is to create an image poster, the tool call sequence planned by AIAgent is as follows: ; The Tool execution location identifier AIAgent calls this tool on the Python server side, passing in... and The parameters perform a database retrieval, extracting the target from the returned resource list. And write it to the tool call chain context; The Tool execution location identifier AIAgent will contain and The parameter call request is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor calls the canvas size setting processing function to adjust the canvas width and height to the specified pixel values. The Tool execution location identifier AIAgent will contain , coordinates and The coordinate request is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor then... Load the corresponding image resources, create a new layer at the specified coordinates and complete image rendering. The canvas state is determined according to the formula. Update, in which It contains all the attribute data for the newly added layer.

8. The method according to claim 6, characterized in that, In step S002, when the user instruction involves adjusting the position or size of an existing layer on the canvas, the tool call sequence planned by AIAgent is as follows: ; S010, AIAgent plans in the tool invocation sequence When using the tool, the front-end tool executor will display the complete state of the current canvas. The layer collection, element collection, and canvas configuration parameters are serialized into JSON format and returned via a BridgeWebSocket channel. The returned data includes the layer's... Current coordinates and current width With height AIAgent parses the above JSON data according to the formula. Calculate the target state increment Determine the updated coordinates and dimensions; The Tool execution location identifier AIAgent will contain The request for the target coordinates and dimensions is forwarded to the front-end tool executor via the BridgeWebSocket channel. The front-end tool executor calls the element update processing function to update the specified layer coordinates and dimensions to the target values. The canvas state is determined according to the formula. Complete synchronization, among which Includes all change data for coordinates and dimensions; S011, AIAgent plans in the tool invocation sequence When the tool is in use, the front-end tool executor performs a screenshot operation on the current canvas, encodes the screenshot data into a base64 format PNG image, encapsulates it into an image type message, and returns it through the BridgeWebSocket channel. AIAgent analyzes the screenshot using its visual understanding capabilities to confirm whether the layer position and size are consistent with the user's instructions.

9. The method according to claim 6, characterized in that, In step S005, the AIAgent is for each... The operation generates a unique request identifier. , and , Together they are encapsulated into a JSON message and sent to the front-end tool executor via the BridgeWebSocket channel; in step S007, the front-end tool executor will... Follow and The data is packaged and sent back together, and AIAgent follows accordingly. After matching the returned result with the corresponding tool call request, it is included in the tool call chain context, and then the completion judgment of step S008 is performed.

Images