Method and device for automatic editing of graphic design canvas based on large language model

Abstract

This application relates to a method and related apparatus for automatic editing of a graphical design canvas based on a large language model. The method includes: receiving a user's natural language message; appending the current structured state of the canvas to the natural language message to obtain an enhanced user message; submitting the enhanced user message and dialogue history to the large language model on the interface side; streaming the response content and performing presentation layer stripping; performing multi-stage extraction on the response content to obtain a canvas operation script; generating a unique call number for the canvas operation script and sending it to the engine side via a message channel; dynamically constructing a restricted asynchronous function based on a whitelist of tools on the engine side and executing the script within it, then sending the script execution result back to the interface side; matching and waiting for callbacks according to the unique call number on the interface side, appending the script execution result to the dialogue history, and driving the next round of inference. This application has the advantages of stably forming a closed loop for canvas editing under a dual-threaded isolated architecture, while balancing expressiveness and security.

Description

Technical Field

[0001] This application relates to the field of graphical interfaces, and in particular to a method and apparatus for automatic editing of a graphical design canvas based on a large language model. Background Technology

[0002] Graphical interface design software uses a canvas, layer tree, and property panel organization, allowing users to manually place and adjust design elements. With the advent of large language models possessing natural language understanding and code generation capabilities, the industry has begun to explore integrating them into design tools, enabling dialogue-driven generation and modification of design elements.

[0003] Due to considerations of security isolation and layered architecture, graphical user interface (GUI) design software plugins typically employ a dual-threaded structure: a UI thread for handling chat interfaces and model calls, and a main thread with full canvas document manipulation permissions. These two threads can only communicate via asynchronous messages. The UI thread often runs as a restricted webpage on a specific origin, constrained by the same-origin policy, making it difficult to directly initiate network requests to any endpoint. While the main thread can manipulate the canvas document, it cannot display the chat interface or independently perform model calls. Integrating a large language model onto this misaligned infrastructure, enabling the "UI calling the model, main thread animates the canvas" actions into a complete task flow, constitutes the primary challenge for the intelligence of design tools.

[0004] Two approaches have been developed to address this issue. The first approach uses function calls based on a fixed list of tools. Several discrete interfaces are pre-registered on the design tool side and submitted to the model with structured descriptions. The model outputs structured tool calls in the dialogue, which are then parsed and executed sequentially by the main thread. This approach preserves safety boundaries, but the set of interfaces available to the model is insufficient to cover the parameter combinations and conditional branches required in actual design scenarios. Each new complex operation requires simultaneous expansion of the interface definition and adaptation layer, resulting in high iteration costs. When encountering complex layouts with multi-step dependencies, the model often needs to repeatedly iterate between multiple rounds of calls, making it difficult to maintain consistency in intermediate states and reducing task success rates. The second approach directly evaluates the model's output dynamically on the main thread. While this achieves near-native code expressiveness, it cannot constrain the range of interfaces accessible to the model and makes it difficult to perform layered capture of build-time and runtime exceptions. Once the model generates out-of-bounds scripts, the main thread's document operation permissions become difficult to control.

[0005] The two approaches described above involve a trade-off between expressiveness and security. The cross-domain constraints and state propagation challenges arising from dual-thread isolation have not been systematically addressed, preventing a stable closed loop from forming between model invocation, script execution, state propagation, and re-inference. Furthermore, when tasks span multiple rounds, the termination conditions, upper bound constraints, and tool extension methods for the planning, execution, and feedback behaviors lack a unified orchestration framework. This frequently leads to the model entering invalid planning loops or producing scripts disconnected from the current canvas state. Summary of the Invention

[0006] In order to stably form a model-driven canvas editing closed loop under a dual-threaded isolated architecture, while taking into account both expressiveness and security, this application provides a method and related system, computer equipment and computer-readable storage medium for automatic editing of graphic design canvas based on a large language model.

[0007] Firstly, this application provides an automatic editing method for a graphic design canvas based on a large language model, which adopts the following technical solution:

[0008] An automatic editing method for graphical design canvas based on a large language model, applied to a dual-threaded plug-in system including both the interface and engine sides, includes the following steps:

[0009] S1. Receive the user's natural language message, attach the current canvas's structured state to the natural language message to obtain an enhanced user message. The structured state includes the current page name, the attribute information of the selected layer, and a list of identifiers for the top-level nodes of the page.

[0010] S2. On the interface side, the enhanced user messages and dialogue history are submitted to the large language model. The response content of the large language model is received in a streaming manner, and the presentation layer is stripped from the response content. The stripped text is then written into the dialogue bubble.

[0011] S3. On the interface side, perform multi-stage extraction on the response content of the large language model to obtain the canvas operation script, and determine the current round type based on the planning signaling carried in the response content. The round type includes execution round, planning round, termination round and abort round.

[0012] S4. When the round type is execution round, generate a unique call number for the canvas operation script, and send the canvas operation script and the unique call number to the engine side via the message channel.

[0013] S5. Receive the canvas operation script and unique call number on the engine side, dynamically construct a restricted asynchronous function based on the preset whitelist tool list, and inject the whitelist tool list as a parameter into the restricted asynchronous function. Execute the canvas operation script in the restricted asynchronous function to operate the canvas document, obtain the script execution result, and send the script execution result and unique call number back to the interface side via the message channel.

[0014] S6. On the interface side, match the callback by the unique call number, append the script execution result to the dialogue history in the form of a user message, and return to S2 to execute the next round of reasoning until the round type is terminated or aborted, thus ending the current session.

[0015] By adopting the above technical solutions, user natural language messages, canvas structured state, and dialogue history are merged into enhanced user messages on the interface side, enabling the large language model to have the current canvas document context when generating canvas operation scripts. The sending, execution, and result feedback of scripts are precisely paired through message channels and unique call numbers, aligning asynchronous waiting on the interface side with execution results on the engine side. By constructing restricted asynchronous functions using a whitelist of tools as parameters, the script can express complex canvas operation combinations in code form, but can only access the injected tools. Under the dual-thread isolation architecture, a stable closed loop consisting of four stages—model invocation, script execution, result feedback to dialogue history, and re-inference—is formed, achieving a balance between expressiveness and security boundaries.

[0016] Optionally, the message channel can carry the following three types of messages with type constraints:

[0017] Script execution messages and script execution result messages are used to carry the canvas operation script sent from the interface to the engine and the script execution result returned from the engine. Script execution messages and script execution result messages are paired with each other by a unique call number.

[0018] Canvas state messages are used to carry structured state from the engine side to the interface side. Canvas state messages are actively pushed from the engine side to the interface side in two situations: selection change event and completion of canvas operation script execution.

[0019] Network proxy messages and network proxy response messages are used to carry network request forwarding initiated by the interface to the engine side and network responses returned by the engine side.

[0020] S4 also includes maintaining a callback pairing table on the interface side, recording the unique call number and the callback in the callback pairing table; S5 also includes actively pushing a canvas state message on the engine side to synchronize the structured state after the script execution result is returned.

[0021] By adopting the above technical solution, the three types of messages with type constraints, along with the pairing of unique call numbers and the forced state synchronization after script execution, ensure that concurrent requests and results are not misaligned, and that the canvas context on the interface side always corresponds to the latest document state.

[0022] Optionally, a network proxy process may be included before S2. The network proxy process includes:

[0023] In response to the network request initiated by the interface side to the large language model being intercepted by the limited runtime environment of the interface side, the network request is forwarded to the engine side via a network proxy message;

[0024] The engine side initiates network requests on behalf of the user with its own execution permissions, receives the response returned by the large language model, and sends the response back to the interface side via a network proxy.

[0025] By adopting the above technical solution, the network proxy process initiates requests that are blocked by the same-origin policy on the interface side on behalf of the engine side with the running permissions, so that model service calls can still be completed in a restricted UI environment and the closed loop of model calls is not interrupted due to cross-domain restrictions.

[0026] Optionally, S4 also includes performing an initial format check on the canvas operation script by the interface side before sending the canvas operation script to the engine side;

[0027] S5 includes sub-steps S51-S54:

[0028] S51. Perform a secondary format check on the received canvas operation script. The check rules for the secondary format check are the same as those for the initial format check.

[0029] S52. In response to the passing of the secondary format validation, a restricted asynchronous function is dynamically constructed. The formal parameter of the restricted asynchronous function is the name of each tool in the whitelist tool list, and the function body of the restricted asynchronous function is the canvas operation script, so that the canvas operation script can access the tools in the whitelist tool list only.

[0030] S53. The execution process of the canvas operation script is wrapped in an inner exception capture structure, and the construction process of the restricted asynchronous function is wrapped in an outer exception capture structure. The inner exception capture structure is used to capture the syntax errors and runtime exceptions of the canvas operation script itself, and the outer exception capture structure is used to capture the construction exceptions of the restricted asynchronous function. Both exception capture structures convert exceptions into structured failure results.

[0031] S54. Send the script execution result and unique call number back to the interface side, and push the structured status once after the script execution result is sent back.

[0032] By adopting the above technical solutions, consistent format validation at both ends prevents scripts from being tampered with in the message channel, parameter-injected restricted asynchronous functions limit the accessible interfaces to a whitelist, and dual-layer exception handling ensures that exceptions in both the construction and execution phases can be returned in a structured manner.

[0033] Optionally, the presentation layer stripping in S2 and the multi-stage extraction in S3 process the same response content in the following ways:

[0034] The presentation layer strips the response content during the streaming reception process in real time. The stripped objects include completed canvas operation code blocks, task planning signaling blocks, step start and step end signaling blocks, and content wrapped by inference tags inside the model. The tail of unclosed code blocks is truncated so that the text presented in the dialog bubble does not contain the stripped objects.

[0035] The process involves extracting the complete response content after the convection reception is completed in multiple stages, attempting each of the following four stages sequentially and ending upon success in any one of them:

[0036] In the first stage, the canvas operation script is matched according to the standard code fence containing the agreed canvas markers;

[0037] In the second phase, in response to the failure of the first phase, the canvas operation script is matched according to the loose format of the code fence header containing the canvas keyword;

[0038] In the third phase, in response to the second phase miss, the canvas operation script is matched according to the format that the first line of the code fence content contains the agreed canvas mark.

[0039] In the fourth stage, in response to a missed third stage, the canvas operation script is matched according to the paragraph format that is no code fence and begins with the convention canvas marker.

[0040] Multi-stage extraction also includes performing standardized cleaning on the hit canvas operation scripts. Standardized cleaning includes removing the first line identifier prefix, removing leading and trailing whitespace, retaining the first occurrence of repeated script blocks, and judging scripts that exceed the preset length limit as extraction failures.

[0041] By adopting the above technical solutions, the presentation layer stripping maintains the readability of the dialog bubble in real time, the multi-stage extraction provides a fallback-compatible matching path for model responses of different formats, and the standardized cleaning ensures that script extraction maintains a stable hit rate despite format differences.

[0042] Optionally, S2 to S6 are arranged in a directed state diagram on the interface side. The directed state diagram includes nodes for calling the large language model, executing tools, parsing and responding, executing scripts, and providing feedback formatting.

[0043] The large language model node handles S2; the parsing response node handles the multi-stage extraction and round type determination in S3; the script execution node handles the interface-side processes in S4 and S5; the feedback formatting node handles the process of appending the script execution result to the dialogue history as a user message in S6; and the tool execution node is used to execute the tool call request in parallel when the response content of the large language model contains a tool call request, and append the return content of the tool call request to the dialogue history.

[0044] The directed state graph sets the upper limit for the number of planning rounds and the upper limit for the total number of steps. When the cumulative number of times the round type is planning round reaches the upper limit for the number of planning rounds, it will forcibly jump to the termination round. When the cumulative number of node jumps in the current session reaches the upper limit for the total number of steps, it will forcibly end the current session.

[0045] By adopting the above technical solution, the five-node directed state graph automatically connects various behaviors according to a fixed route, the upper limit of the planning rounds prevents idle running without execution, and the upper limit of the overall steps prevents the session from running indefinitely, making multi-round tasks terminated and observable.

[0046] Optionally, the following may also be included before S2:

[0047] Maintain multiple skill texts indexed by name, each skill text carrying domain specifications oriented towards specific design tasks;

[0048] In the system instructions submitted to the large language model, only the task completion signaling, the format conventions of the canvas operation code block, and the operation pre-confirmation conventions are retained, and the domain specifications are not directly written into the system instructions.

[0049] Declare skill loading tools in the tool declaration list submitted to the large language model. The tool declaration list is independent of the whitelist tool list. The tool declaration list is used for the large language model to perceive and initiate tool call requests, while the whitelist tool list is used for restricted asynchronous functions to access when executing canvas operation scripts.

[0050] In response to the large language model's output of a tool call request for the skill loading tool after multi-stage extraction in S3, the interface side retrieves the corresponding skill text by name in the tool call request and appends the content of the skill text to the dialogue history.

[0051] By adopting the above technical solution, domain specifications are loaded on demand in the form of named skill text rather than written into system instructions all at once, which controls the baseline length of system instructions, and the model can actively retrieve relevant specifications according to the current task, saving context window consumption.

[0052] Optionally, the enhanced user messages in S1 adopt a multimodal format, which is obtained by the interface side by combining the user's natural language messages, the locally cached structured state, and the locally cached bitmap data;

[0053] Structured state and bitmap data are cached locally on the interface side through the following mechanisms:

[0054] Before S1 is executed, the engine side continuously monitors selection area change events. In response to selection area change events, the engine side pushes the latest structured state to the interface side and the interface side caches it locally.

[0055] In response to a selection change event, the engine initiates an asynchronous image export for a preset number of selected layers and assigns a monotonically increasing selection version number to this asynchronous image export. The output of the asynchronous image export is the bitmap data of the preset number of selected layers.

[0056] When the asynchronous image export is complete and the selection version number is the latest version, the bitmap data is pushed to the interface and cached locally on the interface; when the asynchronous image export is complete and the selection version number has been incremented by subsequent selection change events, the current bitmap data is discarded.

[0057] By adopting the above technical solutions, user messages are enhanced to carry bitmap data in a multimodal form, enabling models that support visual understanding to directly perceive the appearance of the selected layer; the selection version number mechanism discards outdated export results, ensuring that the bitmap and structured state always correspond to the latest selection.

[0058] Optionally, in S5, the mechanism for sandboxing the execution of canvas operation scripts can be implemented by dynamically constructing restricted asynchronous functions, or by using the WebAssembly runtime, a restricted interpreter, or a host-specific script virtual machine. All of these methods constrain the scope of interfaces that the script can access by injecting whitelisted tool list parameters.

[0059] By adopting the above technical solutions, the sandbox implementation can be flexibly selected in different host environments, and the core constraint mechanism of whitelist parameter injection will not fail due to changes in the specific implementation.

[0060] Optionally, in addition to postMessage, the underlying implementation of the message channel can use MessageChannel port pairs or a host vendor's private bridging layer, and the selected underlying implementation can also maintain a pairing mechanism with unique call numbers.

[0061] By adopting the above technical solution, the underlying communication mechanism can be switched according to the host platform conditions, and the pairing relationship of the unique call number is retained under different communication methods, and the closed-loop stability is not affected by the underlying protocol.

[0062] Optionally, the multi-round orchestration from S2 to S6 can be implemented using an explicit finite state machine, a workflow engine, or a handwritten asynchronous loop, except for the directed state graph. The upper limit of the number of planned rounds and the upper limit of the total number of steps are retained in the selected orchestration method.

[0063] By adopting the above technical solutions, the specific implementation of the orchestration framework can be adjusted according to engineering conditions. Both types of upper limit mechanisms are retained as necessary hard constraints in different orchestration methods to prevent multi-round task loss of control.

[0064] Optionally, in addition to HTTP requests, the protocols forwarded by the network proxy process can be extended to WebSocket and gRPC long connection protocols. The engine side can maintain the long connection and route the server messages back to the interface side according to the request number.

[0065] By adopting the above technical solution, the network proxy is extended to the long connection protocol. The streaming response under the long connection is sent back to the interface side according to the request number. Model services that support long connection interfaces can also be accessed.

[0066] Optionally, in addition to regular expression matching, multi-stage extraction in S3 can be implemented using Markdown abstract syntax tree parsing or XML encapsulation block parsing, preserving the fallback order of the four stages under the selected parsing method.

[0067] By adopting the above technical solutions, script block recognition is not limited to regular expression matching. Parsing based on syntax trees can handle nested formats, and the four-stage fallback order remains unchanged to be compatible with the output of multiple models.

[0068] Optionally, the structured state appended to the natural language message in S1 may be carried in the form of text, or in the form of vector structured serialized data and differential summary of two adjacent canvas snapshots.

[0069] By adopting the above technical solutions, the form of carrying structured states can be selected according to the model input capability, and the differential summarization form significantly reduces the additional length of the context in each round in large document scenarios.

[0070] Optionally, the whitelist of tools includes canvas document operation interfaces, shortcuts to the current page, color conversion helper functions, node information serialization functions, canvas state snapshot functions, font loading functions, and default color palette arrays. The font loading function automatically downgrades to a backup font when loading fails.

[0071] By adopting the above technical solution, the whitelist tool list covers common auxiliary capabilities such as canvas operation, color and font, and with the font degradation mechanism, the scripts generated by the model can still be rendered stably in different host environments.

[0072] Optionally, the specific form of the task completion signal is as follows: the model's response during task execution must include canvas operation markers, and the model's response at the end of the task must not include any canvas operation markers, with a plain text summary serving as the basis for determining the termination round.

[0073] By adopting the above technical solution, the determination of the termination round does not rely on additional structured fields, but only on whether the model response still contains operation tags, so that the determination logic is naturally aligned with the model output habits.

[0074] Optionally, the pre-operation confirmation convention is declared in the system instructions submitted to the large language model: the model should confirm the target container before performing canvas operations, and should not arbitrarily insert layers into the entire page without clearly defining the target container.

[0075] By adopting the above technical solution, the pre-operation confirmation agreement is written into the system instruction in the form of hard constraints, so that the model will not blindly insert layers into the root container when facing multi-region pages, thus reducing the cost of misoperation.

[0076] Optionally, the preset maximum length is 500,000 characters. Scripts exceeding 500,000 characters will be judged as failing to extract during the first or second format check.

[0077] By adopting the above technical solution, the specific upper limit value of the length ensures that extremely large-scale abnormal output will not block the sandbox construction on the engine side, while reserving sufficient expression space for legal and complex scripts.

[0078] Optionally, the maximum number of planning rounds is 3. If the cumulative number of consecutive planning rounds reaches 3 and no execution round has occurred, the system will forcibly jump to the termination round.

[0079] By adopting the above technical solution, the specific value of the planning round is reserved for typical task decomposition, and at the same time, it serves as a hard threshold to prevent the model from getting stuck in a loop of continuous refinement without execution.

[0080] Optionally, the total number of steps is capped at 100. If the cumulative number of node jumps in this session reaches 100, the session will be forcibly terminated.

[0081] By adopting the above technical solution, the specific value of the overall step limit is above the number of jumps required for a conventional complex design task, which reserves margin for normal sessions while preventing the system from running indefinitely in abnormal situations.

[0082] Optionally, the preset number of selected layers is set to 3, meaning that the engine initiates asynchronous image export for the first 3 selected layers.

[0083] By adopting the above technical solution, the specific value of the preset quantity takes into account both the amount of multimodal information and the export bandwidth, so that the model can perceive the appearance of the main selected layer without excessively consuming the dialogue context.

[0084] Optionally, the canvas tag is defined as js:canvas, meaning that the canvas operation script uses js:canvas as the identifier for the code fence or content line.

[0085] By adopting the above technical solution, the specific values ​​of the canvas operation markers are compatible with the code fence syntax of most large language models, and the four stages of multi-stage extraction cover the output format differences of different models.

[0086] Optionally, the field values ​​for the structured state can be represented as follows: The current page name is "Homepage Design", the selected layers include a Frame type layer named "Navigation Bar" with a width of 1440 and a height of 60, and a Text type layer named "Title Text" with a width of 320 and a height of 48. The top-level nodes of the page include "Header Area", "Content Area" and "Footer Area".

[0087] By adopting the above technical solution, the field value format of the structured state allows the model to grasp the document partitions through the page name and the top-level node name, and to determine the object to be operated on by the type and size of the selected layer.

[0088] Secondly, the computer device provided in this application adopts the following technical solution:

[0089] A computer device comprising:

[0090] One or more processors;

[0091] Memory;

[0092] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to:

[0093] Perform the above-described automatic editing method for graphic design canvas based on a large language model.

[0094] Thirdly, this application provides a computer-readable storage medium that adopts the following technical solution:

[0095] A computer-readable storage medium storing a computer program capable of being loaded by a processor and executed as described above regarding the automatic editing method for a graphical design canvas based on a large language model.

[0096] The storage medium stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the following:

[0097] The above-mentioned method for automatic editing of graphic design canvases based on large language models.

[0098] In summary, this application includes at least one of the following beneficial technical effects:

[0099] 1. Under the dual-thread isolation architecture, model calls, script execution and result return are aligned through message channels and unique call numbers. Restricted asynchronous functions are constructed by injecting parameters from a whitelist of tools, enabling the canvas editing closed loop to achieve near-native code expressiveness within a safe boundary.

[0100] 2. The combination of three types of message channels, consistent format verification on both ends, two-layer exception capture, five-node state diagram orchestration and double upper limit mechanism ensures that concurrent requests are not misaligned, exceptions can be returned, and multi-round tasks can be terminated and observed; streaming display stripping and four-stage extraction further maintain the readability of dialog bubbles and the stability of script extraction.

[0101] 3. Skills on demand loading decouples domain specifications from system instructions to compress the context baseline; multimodal structured state and anti-competitive state version numbers enable the model to be aware of the current document state; and equivalent alternatives to sandboxing, communication, orchestration, and proxy protocols extend the feasibility of the method in different host environments. Attached Figure Description

[0102] Figure 1 A general flowchart of an automatic editing method for a graphic design canvas based on a large language model, according to some embodiments of this application, is shown.

[0103] Figure 2 A schematic diagram of the overall structure of an automatic editing system for a graphic design canvas based on a large language model, according to some embodiments of this application, is shown.

[0104] Figure 3 Directed states according to some embodiments of this application are shown. Figure 5 Node orchestration flowchart.

[0105] Figure 4 A flowchart illustrating the presentation stripping and four-stage script extraction of large language model response content according to some embodiments of this application is shown.

[0106] Figure 5 A timing diagram of engine-side sandbox execution according to some embodiments of this application is shown.

[0107] Figure 6 A schematic diagram of a computer device according to an embodiment of this application is shown. Detailed Implementation

[0108] The present application will be further described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the scope of the application.

[0109] This application provides an embodiment of an automatic editing method for a graphic design canvas based on a large language model, referring to... Figure 1The method includes S1-S6, which combines the user's natural language message with the current structured state of the canvas into an enhanced user message. The large language model generates a response with an embedded canvas operation script. The script is executed in the engine-side sandbox by a restricted asynchronous function to manipulate the canvas document. The execution result is fed back into the dialogue history to drive the next round of inference, so that the four links of model invocation, script execution, state return and re-inference form a stable closed loop under the dual-thread isolation architecture.

[0110] In this embodiment, the graphical design canvas refers to the two-dimensional workspace in graphical interface design software that carries design elements. Layers of various types, such as Frame, Text, images, and vector graphics, can be placed on the workspace, organized in a layer tree structure. The large language model refers to a generative model with natural language understanding and code generation capabilities, including both cloud-based models accessed via network APIs and locally deployed models. The dual-threaded plug-in system refers to a typical runtime environment provided by the host design software for third-party plug-ins. Specifically, the interface side carries user interaction controls such as chat windows and runs within embedded web pages or iframes, while the engine side carries canvas document operation APIs and runs in the host's main thread. The two sides communicate through an asynchronous message channel provided by the host.

[0111] The following explanation uses a baseline scenario that runs throughout the text as an example: a user enters "Generate navigation bar in header area" in the chat box. At this time, the "Homepage Design Draft" page on the canvas has selected a Frame layer named "Header Area", which is 1440 pixels wide and 80 pixels high. The top-level nodes of the page include "Header Area", "Content Area" and "Footer Area". The dual-thread architecture and message channel are introduced first, and then the method flow is gradually unfolded in the order of S1 to S6. Finally, system mapping and device implementation examples are given.

[0112] Figure 2 A schematic diagram of the overall structure of an automatic editing system for a graphic design canvas based on a large language model, according to some embodiments of this application, is shown. Figure 1 As shown, this method is applied to a dual-threaded plug-in system comprising a user interface and an engine. The user interface runs in a restricted environment with network access limitations, handling interaction with the large language model, maintaining dialogue history, script extraction, and multi-turn orchestration. The engine runs in the host thread with full canvas document operation permissions, handling sandboxed script execution and the creation, modification, and deletion of canvas documents. The user interface and engine communicate bidirectionally via a message channel.

[0113] In practice, the restricted runtime environment on the interface side typically manifests as an iframe-embedded webpage or a restricted origin assigned by the host to a plugin, such as `data: URL` or a host-built sandbox origin. This type of runtime environment is strongly constrained by the browser's same-origin policy, making it difficult to directly initiate fetch requests to any HTTPS endpoint. On the engine side, the main thread permissions are typically manifested as the document manipulation namespace exposed by the host. Interfaces under this namespace can directly read and write canvas node properties, create new layers, and adjust the layer tree structure. While the interface side can support rich interactive controls, it cannot access the canvas document; conversely, while the engine side can manipulate the canvas document, it lacks UI capabilities and cannot freely initiate network requests. This mismatch in capabilities between the two sides is a fundamental condition that this approach's architecture must address.

[0114] The message channel carries three types of messages with type constraints. The first type is script execution messages and script execution result messages, used to carry the canvas operation script sent from the interface to the engine, and the script execution result returned by the engine. These two types are paired according to a unique call number. The second type is canvas status messages, used to carry the structured state of the canvas transmitted from the engine to the interface. Canvas status messages are actively pushed from the engine to the interface during selection change events and when the canvas operation script execution is completed. The third type is network proxy messages and network proxy response messages, used to carry the forwarding of network requests initiated from the interface to the engine and the network responses returned by the engine.

[0115] The typical triggering scenarios for the three types of messages differ between the two threads. Script execution messages are proactively initiated by the UI side after completing the multi-stage extraction of the response content; this is a request-response communication, where each request and response is paired one-to-one using a unique call number. Canvas status messages have two triggering times: first, when the user clicks or switches the selected layer on the canvas, the engine side immediately pushes the latest page name, selected layer attributes, and top-level node list to the UI side; second, after each canvas operation script execution, the engine side pushes the message again, regardless of success or failure, ensuring that the canvas context on the UI side is aligned with the engine side after each round of operations; this is an engine-side proactive push communication. Network proxy messages are only triggered when the UI side's request for model services is intercepted by the restricted runtime environment, and do not occur in every session.

[0116] In some embodiments, the underlying implementation of the message channel, besides postMessage, uses MessageChannel port pairs or a host vendor-specific bridging layer. MessageChannel achieves bidirectional communication through a pre-established pair of ports, avoiding the performance overhead of broadcast message filtering compared to postMessage; the host vendor-specific bridging layer relies on the host-provided native communication interface, suitable for scenarios where the host exposes independent communication protocols to plugins. A unique call number pairing mechanism is also maintained in the selected underlying implementation, and this alternative approach also achieves the goal of aligning asynchronous waiting on the interface side with execution results on the engine side.

[0117] In some embodiments, the structured state appended to the natural language message is carried not only in text form but also in the form of vector structured serialized data or differential summaries of two adjacent canvas snapshots. Vector structured serialized data carries the page structure in JSON or equivalent formats, supporting direct traversal of layer nesting relationships by the model; differential summaries only carry fields that change between two states, suitable for large document scenarios. This alternative approach also achieves the purpose of passing the current canvas context to the model while significantly reducing the additional length of the context in each round.

[0118] Before S1 is executed, the following three types of pre-processing mechanisms continue to run on the interface side and the engine side to provide the necessary conditions for the combination of enhanced user messages in S1 and the subsequent invocation of the large language model in S2.

[0119] The first type of pre-processing mechanism: network proxy process. In response to the network request initiated by the interface side to the large language model being intercepted by the restricted runtime environment of the interface side, the network request is forwarded to the engine side via a network proxy message. The engine side then initiates the network request on behalf of the interface side with its runtime permissions, receives the response returned by the large language model, and sends the response back to the interface side via a network proxy response message.

[0120] Interception is typically determined by the return value of the `fetch` call on the user interface or a network error thrown by the browser. In practice, this manifests as `fetch` returning a status code of 0 or throwing an exception like "Blocked by CORS". After catching this exception, the user interface encapsulates the target URL, request method, request headers, and request body into a network proxy message, which is then sent by the engine. Because the engine's main thread is in the host's extended context, it is usually not restricted by the same-origin policy and can directly initiate requests to any HTTPS endpoint and send the response back as is. For example, when the user interface first attempts to call the model service, an interception is triggered, and the engine completes the call on its behalf. Subsequent calls to the same domain can be completed directly by the user interface if they are trusted by the host session cache.

[0121] In some embodiments, the protocols forwarded by the network proxy process are extended beyond HTTP requests to include WebSocket and gRPC long-connection protocols. The engine side maintains the long connection and routes server messages back to the interface side according to the request number. In a long-connection scenario, one "establish connection" request on the interface side corresponds to a long-connection handle maintained by the engine side. The streaming response pushed by the server is encapsulated into consecutive response messages by the engine side according to the request number and sent back to the interface side. This extension method also achieves the goal of breaking through the network limitations of the interface side, allowing large language models that provide services through long-connection interfaces to also access this method.

[0122] The second type of pre-processing mechanism: on-demand skill loading. The interface maintains multiple skill texts indexed by name, each carrying domain specifications specific to the design task. In the system instructions submitted to the large language model, only task completion signals, canvas operation code block format conventions, and pre-operation confirmation conventions are retained; domain specifications are not directly written into the system instructions. Skill loading tools are declared in the tool declaration list submitted to the large language model. This tool declaration list is independent of the whitelist tool list. The tool declaration list is used by the large language model to detect and initiate tool call requests, while the whitelist tool list is used by restricted asynchronous functions to access when executing canvas operation scripts. These two belong to two different registry types within the system: the former is submitted in JSON Schema format with each model request, while the latter resides as a JavaScript object in the engine-side sandbox construction process.

[0123] Typical names for skill texts include "Automatic Layout Parameter Rules," "Team Component Usage Guidelines," and "Default Color Palette Rules." Each skill text describes the constraints and best practices for the corresponding domain in plain text. In response to the parsing node identifying a tool call request for the skill loading tool within the response content, beyond the multi-stage extraction and round type determination in S3, the tool execution node on the interface side retrieves the corresponding skill text by the name in the tool call request and appends the skill text content to the dialogue history. For example, when a user requests to generate a navigation bar in the baseline scenario, the model can first initiate a tool call request for the "Automatic Layout Parameter Rules" skill. Based on this, the interface side retrieves the main text of the specification from the skill text index and appends it to the dialogue history. In the next round of inference, the model can then generate an operation script according to this specification.

[0124] Organizing domain specifications using an on-demand skill loading approach can significantly reduce the baseline length of system instructions. If all domain specifications were directly written into system instructions, even if a user initiates only a simple task without complex layouts, thousands or even tens of thousands of characters of specification text would remain in the system instructions, cluttering the dialog window. With on-demand loading, system instructions retain only essential hard constraints such as task completion signals, code block formatting conventions, and pre-operation confirmation conventions; specifications not touched in a single session do not consume context resources.

[0125] The pre-operation confirmation convention is declared as a hard constraint in the system instructions: before performing canvas operations, the model should first confirm the target container, and should not arbitrarily insert layers into the entire page without clearly defining the target container. For example, in the baseline scenario, the model should first confirm that the operation target is the "header area", rather than directly inserting a layer into the root container of the page, thereby avoiding the navigation bar being mistakenly placed in the footer area or on a separate page.

[0126] The third type of pre-processing mechanism: Selection monitoring and bitmap export. The engine continuously monitors selection change events. In response to these events, the engine pushes the latest structured state to the interface, which is then cached locally. Simultaneously, the engine initiates asynchronous image export for a predetermined number of selected layers, assigning a monotonically increasing selection version number to each. The output of the asynchronous image export consists of bitmap data for each of the predetermined number of selected layers. When the asynchronous image export is complete and the selection version number is the latest version, the bitmap data is pushed to the interface and cached locally. When the asynchronous image export is complete and the selection version number has been incremented by subsequent selection change events, the current bitmap data is discarded.

[0127] The comparison rule for the selection version number is as follows: At the start of asynchronous image export, the current version number is recorded as the "export number for this time." Upon completion of the export, this "export number for this time" is compared with the latest version number currently maintained by the engine. If they are equal, the export corresponds to the latest selection; otherwise, it is considered that the selected area has been overwritten by an updated selection. For example, if a user clicks on the "header area," triggering the version number to increment from 17 to 18, and the engine starts exporting the bitmap of the "header area," and before the export is complete, the user quickly clicks on the "content area," triggering the version number to increment to 19 and starting a new export. In this case, the export corresponding to version number 18, even if completed, will be discarded; only the bitmap of the "content area" corresponding to version number 19 will be pushed. This mechanism ensures that the bitmap data and structured state always correspond to the latest selection, preventing screenshots from being taken when the old image overwrites the new selection when the user quickly switches selections.

[0128] In this embodiment, the preset quantity is set to 3, meaning that the engine initiates asynchronous image export for the first 3 layers in the selected layers. This choice balances multimodal information content and export bandwidth: when the preset quantity is set to 1, the information content is insufficient for the model to perceive the multi-layer selected scene; if the value is too large, the export will occupy the rendering thread for a long time when the user selects a large number of layers.

[0129] S1. Receive the user's natural language message, append the current canvas's structured state to the natural language message, and obtain an enhanced user message. The structured state includes the current page name, the attribute information of the selected layer, and a list of identifiers for the page's top-level nodes.

[0130] In S1, enhanced user messages adopt a multimodal format, which is obtained by combining the user's natural language message, locally cached structured state, and locally cached bitmap data from the interface side. The structured state and bitmap data are the data continuously pushed to and cached by the aforementioned selection monitoring and bitmap export mechanism. The multimodal message delivery format is based on the message structure of mainstream model APIs, with the text portion carried by a `text` field and the bitmap portion carried by an `image` field with an accompanying MIME type identifier; both are combined into a single user message submission.

[0131] The field values ​​in the structured state can be presented in various forms. For example, when a user selects only a single layer, the number of selected layers in the structured state is 1, and the corresponding layer's attributes include name, type, width, and height. When the user selects multiple layers simultaneously, the number of selected layers in the structured state is 2 or more, with the attributes of each layer listed side-by-side. The types of each layer can be the same or different; a typical example is the simultaneous inclusion of Frame and Text type layers. The identifier list of the top-level nodes on the page contains the various non-nested blocks under the current page, such as the "header area," "content area," and "footer area."

[0132] In the baseline scenario, after the user enters "Generate navigation bar in header area" in the chat box, the text portion of the enhanced user message combined in S1 includes the user's original message "Generate navigation bar in header area" and an additional canvas status description paragraph: "Current canvas status, page: homepage design draft, currently selected layer: header area (Frame, width 1440, height 80), top-level nodes of the page: header area, content area, footer area". The bitmap portion includes a PNG thumbnail of the "header area".

[0133] When the selection version number corresponding to the locally cached bitmap data is inconsistent with the selection version number of the current structured state, or when the bitmap data has not yet been asynchronously exported, the combination strategy in S1 automatically downgrades to include only the text portion, and the bitmap data does not participate in the combination, thus avoiding submitting a bitmap that does not correspond to the current selection to the model. This downgrade typically occurs when the user initiates a natural language message immediately after the selection changes, at which point the asynchronous image export is still in progress or the export result of the new selection has not yet arrived at the interface.

[0134] When the model is a plain text model that does not support visual understanding, the combination strategy in S1 automatically downgrades to include only the text portion, with bitmap data not participating in the combination. This downgrade is completed on the interface side based on the capability declaration of the currently selected model, without affecting the appending of text to the structured state, nor with any subsequent steps from S2 to S6.

[0135] S2. On the interface side, the enhanced user messages and dialogue history are submitted to the large language model. The response content of the large language model is received in a streaming manner, and the presentation layer is stripped from the response content. The stripped text is then written into the dialogue bubble.

[0136] The dialogue history is maintained in the local state on the interface side. Each round of user input, model response, and tool call return result is added as an independent entry in chronological order. The message sequence submitted in S2 is the result of concatenating system commands, dialogue history, and the current enhanced user message in sequence. Streaming reception returns server-side push events or equivalent incremental protocols, and each received incremental block triggers a presentation layer stripping.

[0137] Figure 4 A flowchart illustrating the presentation stripping and four-stage script extraction process for large language model response content according to some embodiments of this application is shown. See also Figure 4 In the upper part, the presentation layer strips the response content during the convection reception process in real time. The stripped objects include the completed canvas operation code block, task planning signaling block, step start and step end signaling block, and the content wrapped by the inference label inside the model. The tail of the code block that has not yet been closed is truncated so that the text presented in the dialog bubble does not contain the above stripped objects.

[0138] The timing of the stripping action differs fundamentally from stripping after all responses have been collected. If stripping is performed only after all responses have been collected, in a streaming scenario, users will first see a large amount of raw code and internal inference tags scrolling through the screen, only switching to the cleaned text at the very last moment, resulting in a disjointed experience. This method immediately strips the currently accumulated response content each time an incremental block is received, and the stripping result is refreshed to the chat bubble in real time. Users see the cleaned text generated word by word, rather than a jump from dirty to clean.

[0139] The trigger condition for truncating the end of an unclosed code block is as follows: when the accumulated response content contains a code fence start character but no corresponding close character has appeared, the start character and all subsequent content are considered an incomplete code block and are not displayed in the dialog bubble. Once a subsequent incremental block brings a close character, the code segment is recognized as a completed canvas operation code block, removed entirely from the presentation layer, and the dialog bubble immediately fills in the explanatory text following that segment. For example, in the baseline scenario, when the model streams "Okay, I'll create a navigation bar in the header area\n```js:canvas\n / / create navigation bar\nconstnav=page.create", the dialog bubble only displays the sentence "Okay, I'll create a navigation bar in the header area," and the subsequent code portion never enters the dialog bubble until it is closed.

[0140] The stripping of the presentation layer directly impacts the refresh rate of the chat bubble. If the original model's response contains 500 words of explanatory text and 200 lines of script code, after stripping, only the explanatory text is displayed in the chat bubble. The refresh rate matches the generation rate of the explanatory text, and the user experience is comparable to a pure dialogue scenario without code generation.

[0141] S3. On the interface side, perform multi-stage extraction on the response content of the large language model to obtain the canvas operation script, and determine the current round type based on the planning signaling carried in the response content. Round types include execution round, planning round, termination round, and abort round.

[0142] See also Figure 4 The second half involves a multi-stage extraction of the complete response content after the convection reception is completed. This process attempts in four stages sequentially and terminates upon success in any one of them:

[0143] The first phase matches canvas manipulation scripts against standard code fences that contain the convention canvas markup. A typical hit example is a standard code fence that begins with ```js:canvas and ends with ```, which is the default output format for most models that follow Markdown specifications.

[0144] In the second phase, in response to a missed match in the first phase, canvas operation scripts are matched based on a loose format where the code fence header contains the keyword "canvas". A typical example of a successful match is a fence starting with "javascriptcanvas" or "jscanvas", a format that appears in some models where language identifiers and custom tags are separated by spaces.

[0145] In the third phase, in response to a missed match in the second phase, canvas operation scripts are matched based on the format where the first line of the code fence content contains the conventional canvas tag. A typical example of a match is a fence that starts with ```js and begins with ` / / js:canvas` or an equivalent comment. This format appears in some models where custom tags are written in the code content line rather than at the fence header.

[0146] In the fourth stage, in response to a missed match in the third stage, the canvas operation script is matched against paragraphs that have no code fences and begin with the conventional canvas marker. A typical example of a hit is a paragraph without any code fences that begins directly with js:canvas; this type of format appears in the very few model outputs that do not use code fences.

[0147] In this embodiment, the canvas tag is defined as js:canvas, meaning that the canvas operation script uses js:canvas as the identifier for the code fence or content line. The choice of js:canvas is compatible with the code fence syntax of most large language models, and the js prefix ensures that the tag is naturally recognized as JavaScript syntax highlighting during Markdown rendering, facilitating readability verification.

[0148] Taking a complete model response in the baseline scenario as an example, suppose the model returns "Okay, I'll create the navigation bar in the header area.\n```js:canvas\nconst header=page.findByName('Header Area');\nconst nav=header.createFrame({name: 'Navigation Bar', width: 1440, height: 60});\nav.setFill('#FFFFFF');\n```\nCreation complete." During the first stage of scanning this response, the standard code fence ```js:canvas` is directly hit, and the three lines of JavaScript code within the fence are extracted as canvas operation scripts. These are not executed in the second to fourth stages.

[0149] Multi-stage extraction also includes standardizing and cleaning the matched canvas operation scripts. Standardization and cleaning includes removing the first-line identifier prefix, removing leading and trailing whitespace, retaining the first occurrence of duplicate script blocks, and classifying scripts exceeding a preset length limit as extraction failures. The specific rules for retaining multiple duplicate script segments are as follows: when two identical script blocks exist in the same response content, only the first occurrence is retained, and subsequent duplicates are discarded to prevent the same operation from being executed twice due to streaming retries or formatting errors. In this embodiment, the preset length limit is 500,000 characters; scripts exceeding 500,000 characters are considered extraction failures during the extraction stage, and the reason for failure is recorded as a structured object and returned to the dialogue history.

[0150] The criteria for determining the round type are as follows: When the response content contains at least one canvas operation script extracted through multiple stages, it is determined to be an execution round, and the system is routed to S4 to enter the script sending process; when the response content only carries task planning signaling and no canvas operation script is extracted, it is determined to be a planning round, and the system is routed to the next round S2 for the model to continue refining the planning; when the response content contains neither canvas operation script nor task planning signaling, and the response content is a plain text summary, it is determined to be a termination round, the current session ends, and the termination text is displayed to the user; when the user clicks stop or receives an external termination signal, it is determined to be an aborted round, the session ends immediately, and no further actions of the current round are waited for.

[0151] In some embodiments, multi-stage extraction is implemented not only through regular expression matching but also through Markdown abstract syntax tree parsing or XML encapsulation block parsing, preserving the fallback order of the four stages under the selected parsing method. Abstract syntax tree parsing can accurately identify nesting levels in scenarios involving nested code blocks, such as when one code block references another, avoiding mismatches of inner fence closing characters by regular expressions. This alternative method also achieves the goal of accurately extracting canvas operation scripts from model responses.

[0152] S4. When the round type is execution round, generate a unique call number for the canvas operation script, and send the canvas operation script and the unique call number to the engine side via the message channel.

[0153] The strategy for generating unique call numbers can be either a monotonically increasing integer sequence number or a randomly generated UUID, as long as it is guaranteed that no two calls are repeated within the same session. This embodiment uses a monotonically increasing integer sequence number, with an initial value of 1, incrementing by 1 for each generated number. For example, in the baseline scenario, the script extracted in the first round is assigned number 1. If a new script is subsequently executed in the same session, it is assigned number 2, and so on.

[0154] In S4, the interface performs an initial format check on the canvas operation script before sending it to the engine. This initial format check confirms that the script's content is not empty and its length does not exceed a preset maximum. Scripts that fail this initial format check are directly deemed as failed and the result is returned; they are not sent to the engine. Failure to pass the initial format check typically occurs when multi-stage extraction encounters a seemingly valid but actually empty string or extremely long content. In such cases, the interface directly returns the failure result as a callback, avoiding invalid messages consuming message channel bandwidth.

[0155] S4 also includes maintaining a callback matching table on the interface side, recording the unique call number and the callback function to be waited for in the callback matching table, so that S6 can match the script execution result according to the unique call number. The callback matching table uses the unique call number as the key and the callback function as the value. Each time a script execution is initiated, a record is written, and each time a script execution result is received, the corresponding record is retrieved by number and cleared.

[0156] Taking a specific concurrent scenario as an example, suppose the same round of model responses contains two independent canvas operation scripts, assigned numbers 1 and 2 respectively. The two scripts are sent to the engine side almost simultaneously and each enters its own execution queue. Script 1 involves creating a large number of nodes, with an execution time of 200 milliseconds; script 2 only performs a single node attribute modification, with an execution time of 5 milliseconds. Without a matching table, the result of script 2, which returns first, might be mistakenly matched by the interface as the callback for script 1, causing the callback for script 1 to be permanently suspended. With a matching table, the interface precisely matches the corresponding callback according to the number carried in the message. The result of script 2 completes the callback for script 2 first, and the result of script 1 completes the callback for script 1 subsequently. The two callbacks are independent and do not interfere with each other.

[0157] S5. Receive the canvas operation script and unique call number on the engine side, dynamically construct a restricted asynchronous function based on the preset whitelist tool list, and inject the whitelist tool list as a parameter into the restricted asynchronous function. Execute the canvas operation script in the restricted asynchronous function to operate the canvas document, obtain the script execution result, and send the script execution result and unique call number back to the interface side via the message channel.

[0158] The engine, as the executor of the script, faces two parallel requirements. The first is security boundary control, meaning scripts should not be able to access arbitrary global interfaces in the host environment. If the model generates an out-of-bounds script, such as attempting to read localStorage or initiating local network requests, the execution layer must prevent such access during the build phase, rather than relying on runtime access control. The second is observability, meaning any exceptions occurring during execution must be propagated back to the interface in a structured form, allowing the interface to input clear reasons for failure into the model's next round of inference, rather than vague string errors. This method satisfies both requirements simultaneously through parameter-injected restricted asynchronous functions and two-layer exception handling.

[0159] The script's path from the message channel entry point to the sandbox and then to the documentation API is as follows: After receiving the script execution message, the message entry point hands it over to the execution scheduling module on the engine side; the execution scheduling module calls the restricted asynchronous function builder, using the names of each tool in the whitelist tool list as formal parameters and the script as the function body to construct a restricted asynchronous function; the constructed restricted asynchronous function is called and the corresponding tool object in the whitelist tool list is injected as an actual parameter. The script can then call the corresponding tool through the formal parameter name inside the function body, and then access the canvas documentation API through the tool; after execution, the output result or the captured exception is encapsulated into a structured result through a two-layer exception capture structure, and sent back to the interface side along with a unique call number through the message entry point.

[0160] S5 also includes a proactive push of a canvas state message by the engine side after the script execution result is returned to synchronize the structured state. This synchronization action is performed regardless of whether the script execution is successful or not, ensuring that the canvas context on the interface side is aligned with the actual canvas state on the engine side after each round of script execution. This forced synchronization directly benefits the model's decision-making in subsequent rounds: in addition to the execution result appended in S6, the model will also receive an updated canvas state injection in the next round S2, ensuring that the canvas state used by the model to generate the next script is completely consistent with the actual document, avoiding the generation of mismatched operations based on outdated states.

[0161] Figure 5 A timing diagram of engine-side sandbox execution according to some embodiments of this application is shown. See also Figure 5 Specifically, S5 includes sub-steps S51-S54.

[0162] S51. Perform a secondary format check on the received canvas operation script. The check rules for the secondary format check are the same as those for the primary format check, i.e., they are also used to confirm that the content of the canvas operation script is not empty and its length does not exceed the preset length limit. Although the secondary format check has the same rules as the primary check, it is executed on the engine side. Its purpose is to prevent the script from being tampered with or abnormally spliced ​​during message channel transmission. If the check rules on both sides are inconsistent, messages that each side considers legitimate may be judged as illegitimate on the other side, resulting in logical inconsistencies in the script execution result.

[0163] S52. In response to successful secondary format validation, a restricted asynchronous function is dynamically constructed. The formal parameters of the restricted asynchronous function are the names of each tool in the whitelist of tools, and the function body of the restricted asynchronous function is the canvas operation script, enabling the canvas operation script to access only the tools in the whitelist. Specifically, the construction process is equivalent to calling `newAsyncFunction('api', 'page', 'hexToRgb', 'serializeNode', 'snapshot', 'loadFont', 'palette', scriptBody)` and calling the constructed function with `(apiImpl, pageImpl, hexToRgbImpl, serializeNodeImpl, snapshotImpl, loadFontImpl, paletteImpl)` as actual parameters, where `scriptBody` is the content of the canvas operation script. Because the lexical scope when constructing `AsyncFunction` is the global scope rather than the current function scope, the script cannot access any local variables of the scheduling module through closures, nor can it access global interfaces not listed in the formal parameters.

[0164] In this embodiment, the whitelist tool list includes a canvas document operation interface, a shortcut reference to the current page, a color conversion helper function, a node information serialization function, a canvas state snapshot function, a font loading function, and a default color palette array. The font loading function automatically downgrades to a backup font if loading fails. The canvas document operation interface aggregates atomic operations such as creating layers, modifying attributes, and importing components in a namespace format. The shortcut reference to the current page points to the root node of the currently active page using the `page` variable, allowing the script to access common operation targets without needing to search through layers. The color conversion helper function supports converting hexadecimal color codes and Chinese / English color names into color objects acceptable to the canvas. The node information serialization function converts node objects into printable structured text, facilitating the recording of intermediate states during script execution. The canvas state snapshot function obtains a complete structured snapshot of the current page. The font loading function loads a specified font, automatically downgrading to a backup font (e.g., the system default sans-serif font) if the specified font is missing in the user's environment, preventing script execution failure due to font loss. The default color palette array carries a set of preset colors for the script to use as default values ​​when the user does not specify a color.

[0165] Parameter-injected restricted asynchronous functions differ fundamentally from dynamic evaluation in terms of access boundary control. Dynamic evaluation executes the script directly within the lexical scope of the scheduling module, allowing the script to access all local variables of the scheduling module and all interfaces in the global scope through closures. In contrast, parameter-injected restricted asynchronous functions explicitly declare access whitelists through formal parameters. The set of names accessible to the script is limited to utility names in the whitelist, and any host APIs not declared in the whitelist are no longer visible to the script.

[0166] S53. The execution process of the canvas operation script is wrapped by an inner exception handling structure, and the construction process of the restricted asynchronous function is wrapped by an outer exception handling structure. The inner exception handling structure is used to catch syntax errors and runtime exceptions in the canvas operation script itself. Typical triggers include undefined variable references in the script, invalid parameters for canvas document API calls, and rejected asynchronous Promises. The outer exception handling structure is used to catch construction exceptions of restricted asynchronous functions. Typical triggers include SyntaxError being thrown during the lexical analysis phase of the script, such as mismatched brackets, unclosed strings, or the AsyncFunction constructor being unavailable due to host environment limitations. Both exception handling structures convert exceptions into structured failure results. The structured failure results contain at least three fields: exception type identifier, exception message text, and a unique call number for this call, so that the interface can distinguish the cause of failure and determine the prompt content for the next round of the model.

[0167] S54. Send the script execution result and unique call number back to the interface side, and push the structured status once after the script execution result is sent back.

[0168] Taking the baseline scenario as an example, this demonstrates the intermediate data generated sequentially by S51-S54. S51 receives script number 1, which contains three lines of JavaScript code, and passes validation. S52 constructs a restricted asynchronous function. After injecting formal parameters, the script calls page.findByName to locate the "header area," then calls header.createFrame to create the navigation bar, and then calls nav.setFill to set the fill color. S53 does not catch any exceptions during execution. If an invalid color string is passed to setFill in the script, the inner exception handling structure encapsulates it as a failure result in the form of {code: 'RuntimeError', message: 'Invalidcolorstring', callId: 1}. S54 sends the success result or the aforementioned failure result along with number 1 back to the interface, and pushes another canvas status message containing information about the newly created "navigation bar" layer.

[0169] In some embodiments, the mechanism for sandboxing the execution of canvas operation scripts in S5, besides being implemented by dynamically constructing restricted asynchronous functions, is replaced by the WebAssembly runtime, a restricted interpreter, or a host-specific script virtual machine. All three methods constrain the scope of interfaces accessible to the script through whitelist tool list parameter injection. The WebAssembly runtime provides whitelisted tools to the Wasm module as import objects, allowing the module to call tools only through the import table, achieving isolation similar to restricted asynchronous functions. The restricted interpreter directly restricts the types of parsable syntax nodes during the interpretation phase and looks up the table for each variable access. The host-specific script virtual machine is suitable for scenarios where the host provides V8Isolate or an equivalent mechanism, achieving complete isolation from the main thread by creating independent Isolate instances. These three alternative methods also achieve the goal of executing scripts within a security boundary and accessing a controlled toolset.

[0170] S6. On the interface side, match the callback by the unique call number, append the script execution result to the dialogue history in the form of a user message, and return to S2 to execute the next round of reasoning until the round type is terminated or aborted, thus ending the current session.

[0171] In S6, after the interface receives the script execution result message, it retrieves the corresponding waiting callback from the waiting callback pairing table using the unique call number it carries as the key, completes the waiting, and clears the record from the pairing table. The textual description of the script execution result is then encapsulated as a user message and appended to the dialogue history, enabling the large language model to perceive the execution status of the previous script in the next round of inference in S2, and determine whether to continue initiating a new canvas operation, correct the previous operation, or declare the task complete.

[0172] The field mapping rules for user messages appended to the dialogue history are as follows: For successful messages, the main text is "Script execution result: Success," accompanied by a new node summary generated by the node information serialization function during execution; for failed messages, the main text is "Script execution result: Failure, Reason: {Abnormal message}," accompanied by an exception type identifier for the model to identify the failure category. In terms of encapsulation format, this message is written with the user role and is isomorphic to the user's original message, eliminating the need for the model to perform special format recognition for execution feedback.

[0173] Using a two-round interaction in a baseline scenario as an example, this demonstrates how the result of the first round affects the model's decision in the second round. In the first round, the user requests "generate a navigation bar in the header area." After the model generates a script to create a Frame, it is successfully executed by the engine. S6 adds "Script execution result: Success, new node: navigation bar (Frame, parent node: header area)". In the second round, the user asks "add three more menu items to the navigation bar." After seeing the successful feedback and the name of the new node in the dialogue history, the script generated in the second round will directly locate the node created in the first round using page.findByName('navigation bar'), without needing to search or create it again. If the first round fails and the feedback is "RuntimeError: Invalidcolor", the model will first correct the color parameters in the second round of inference before initiating a new script.

[0174] The specific form of the task completion signal is as follows: the model's response during task execution must contain canvas operation markers, and the model's response at the end of the task must not contain any canvas operation markers. A plain text summary is used as the criterion for determining the termination round. The identification standard for a plain text summary is that none of the four stages extracted by S3's multi-stage model are hit, and the response content also lacks task planning signals. In this case, S3 determines the round type as a termination round, S6 no longer drives the next round of inference, and the session ends after the model's final text is displayed to the user.

[0175] Figure 3 Directed states according to some embodiments of this application are shown. Figure 5 Node orchestration flowchart. (Example) Figure 3 As shown, S2 to S6 are arranged in a directed state diagram on the interface side. The directed state diagram includes five nodes: calling the large language model node, tool execution node, parsing response node, script execution node, and feedback formatting node.

[0176] The large language model node in S2 is responsible for submitting enhanced user messages and dialogue history to the large language model and receiving responses in a streaming manner. The incoming edges of this node are divided into four categories: the first category comes from the starting point of the entire state graph, triggered by the initial submission of the current user message; the second category comes from the feedback formatting node, triggered by re-initiating inference after receiving the result of the previous script execution; the third category comes from the tool execution node, triggered by re-initiating inference after the tool call request is executed and the returned content is appended to the dialogue history; and the fourth category comes from the back edge when the parsing response node determines it to be a planning round, triggered by re-initiating inference after the model generates planning text and receives supplementary prompts. The outgoing edges of this node point to the parsing response node.

[0177] In addition to handling multi-stage extraction and round type determination in S3, the parsing reply node is also responsible for identifying whether the reply content contains a request to call tools from the tool declaration list. The incoming edges of this node come from the calling large language model node, and the outgoing edges are divided into four paths based on the parsing results: if the reply content contains a tool call request, it points to the tool execution node; if it does not contain a tool call request and the round type is determined to be an execution round, it points to the execution script node; if it does not contain a tool call request and is determined to be a planning round, it back-edges to the calling large language model node. In this case, if the end of the current dialogue history is a model reply without any new user message, a "Please continue execution" prompt is automatically added to satisfy the interface constraint of some model services that the dialogue ends with a user message; if it does not contain a tool call request and is determined to be a terminated or aborted round, it points to the end of the state graph.

[0178] The script execution node handles the interface-side flow in S4 and S5, namely generating a unique call number, sending a script execution message, and waiting for and receiving the script execution result message. The node's incoming edges come from the parsing response node, and its outgoing edges point to the feedback formatting node.

[0179] The feedback formatting node handles the process in S6 of appending the script execution result to the dialogue history as a user message. The incoming edges of this node come from the script execution node, and the outgoing edges point to the node that calls the large language model, forming a complete multi-turn closed loop.

[0180] The tool execution node is used to execute tool call requests in parallel when the response content of the large language model contains a tool call request, and to append the response content of the tool call request to the dialogue history. The aforementioned on-demand loading of skills is implemented through this node. The incoming edges of this node come from the parsing response node, and the outgoing edges return to the calling large language model node.

[0181] Taking a baseline scenario as an example, the node transition trajectory of a complete session is plotted as follows: Starting point → Calling the large language model node (first round submission, the model initiates a skill loading tool call request) → Tool execution node (loading the "Automatic Layout Parameter Rules" skill) → Calling the large language model node (second round submission, the model generates planning text) → Parsing the response node (determined as a planning round) → Calling the large language model node (third round submission, the model generates a Frame creation script) → Parsing the response node (determined as an execution round) → Executing the script node → Feedback formatting node → Calling the large language model node (fourth round submission, the model generates a plain text summary) → Parsing the response node (determined as a termination round) → End point. This trajectory involves 5 node transitions back to the calling large language model node, 1 planning round, 1 execution round, and 1 termination round.

[0182] The directed state graph sets two types of upper limits: a planning round limit and an overall step limit. When the cumulative number of planning rounds reaches the planning round limit, a forced jump to the termination round is initiated. When the cumulative number of node jumps in the current session reaches the overall step limit, the current session is forcibly terminated. In this embodiment, the planning round limit is set to 3, and the overall step limit is set to 100. Setting the planning round limit to 3 reserves space for typical task decomposition while preventing the model from getting stuck in a loop of continuous refinement without execution; setting the overall step limit to 100 exceeds the number of jumps required for conventional complex design tasks, reserving margin for normal sessions while preventing the system from running indefinitely in abnormal situations.

[0183] In some embodiments, the multi-round orchestration of S2 to S6 is implemented not only with a directed state graph, but also with an explicit finite state machine, a workflow engine, or a handwritten asynchronous loop, while retaining the upper limit of the number of planning rounds and the upper limit of the overall steps in the selected orchestration method. The explicit finite state machine lists all legal node transitions through a state transition table; the workflow engine describes the dependencies between nodes using a declarative syntax; and the handwritten asynchronous loop strings together the actions of the five nodes using a while loop and a switch branch. These three alternative methods also achieve the goal of incorporating the five types of behaviors of multi-round reasoning into a unified orchestration and preventing infinite loops with two types of upper limits.

[0184] Corresponding to the above methods, this application also provides an automatic editing system for a graphical design canvas based on a large language model. This system includes a user interface module and an engine module, which communicate bidirectionally via a message channel.

[0185] The interface-side module is configured to receive natural language messages from users, append the current structured state of the canvas to the natural language messages to obtain enhanced user messages; submit the enhanced user messages and dialogue history to the large language model, stream the response content, and perform presentation layer stripping on the response content to write it into the dialogue bubble; perform multi-stage extraction on the response content to obtain the canvas operation script, and determine the current round type based on planning signaling; generate a unique call number for the canvas operation script, and send the canvas operation script and the unique call number to the engine-side module via the message channel, and maintain a waiting callback matching table locally; in response to receiving the script execution result, match the waiting callback in the waiting callback matching table according to the unique call number, and append the script execution result to the dialogue history in the form of a user message to drive the next round of inference.

[0186] The engine-side module is configured to receive the canvas operation script and unique call number via a message channel; dynamically construct a restricted asynchronous function based on a preset whitelist of tools and inject the whitelist of tools as a parameter into the restricted asynchronous function; execute the canvas operation script in the restricted asynchronous function to manipulate the canvas document and obtain the script execution result; send the script execution result and unique call number back to the interface-side module via a message channel, and actively push a structured state after the script execution result is sent back.

[0187] The mapping relationship between interface-side modules and method steps is as follows: S1 to S4 are handled by the interface-side module, S6 is handled by the interface-side module, and the sending and waiting process on the interface side in S5 is handled by the interface-side module. The mapping relationship between engine-side modules and method steps is as follows: the script sandboxing execution process in S5 is handled by the engine-side module, as are the aforementioned selection area monitoring and bitmap export, and the proxy network request initiation step in the network proxy process.

[0188] The UI-side modules are distributed within the restricted runtime environment allocated to the plugin by the host design software, such as an iframe or a restricted source webpage. The engine-side modules are distributed within the host's main thread or the main thread communication interface opened by the host for the plugin. Physically, the two modules belong to different runtime contexts of the same host process. Communication relies on asynchronous message channels provided by the host, does not cross process boundaries, and does not depend on external networks.

[0189] This application also provides a computer device. (See attached document.) Figure 6 The computer device includes one or more processors, memory, and one or more applications, the one or more applications being stored in memory and configured to be executed by one or more processors, and the one or more applications being configured to perform the above-described automatic editing method for a graphic design canvas based on a large language model.

[0190] Typical processors include central processing units (CPUs), graphics processing units (GPUs), and dedicated accelerators with general-purpose computing capabilities. CPUs handle control-flow-intensive tasks such as dialogue management, script extraction, and state diagram orchestration for the user interface module, while GPUs provide acceleration for bitmap export and image preprocessing. Typical memory types include random access memory (RAM), solid-state drives (SSDs), and combinations thereof. RAM holds dynamic data such as runtime dialogue history, callback waiting lists, and whitelisted tool objects, while SSDs hold persistent data such as the application itself, skill text indexes, and user configurations. One or more applications can be deployed as plug-ins embedded in the host design software, or as standalone desktop or web applications connecting to the canvas document via an interface exposed by the host.

[0191] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program that can be loaded and executed by a processor to implement the aforementioned automatic editing method for a graphic design canvas based on a large language model. The storage medium stores at least one instruction, at least one program segment, a code set, or an instruction set, which is loaded and executed by the processor to implement the aforementioned automatic editing method for a graphic design canvas based on a large language model.

[0192] The specific forms of storage media include optical discs, magnetic disks, solid-state drives, read-only memory, random access memory, flash memory cards, and media that are downloaded over a network and stored locally. Any physical or logical medium capable of carrying the aforementioned computer programs and being loaded by a processor falls within the scope of the computer-readable storage media described in this application.

[0193] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0194] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A large language model-based graphic design canvas automatic editing method applied to a double-thread plug-in system including an interface side and an engine side, characterized in that, Includes the following steps: S1. Receive the user's natural language message, attach the current canvas's structured state to the natural language message to obtain an enhanced user message, wherein the structured state includes the current page name, the attribute information of the selected layer, and a list of identifiers for the top-level nodes of the page; S2. On the interface side, submit the enhanced user message and the dialogue history to the large language model, stream the response content of the large language model, and perform presentation layer stripping on the response content, and write the stripped text into the dialogue bubble. S3. Perform multi-stage extraction on the response content of the large language model on the interface side to obtain the canvas operation script, and determine the current round type based on the planning signaling carried in the response content. The round type includes execution round, planning round, termination round and abort round. S4. When the round type is the execution round, a unique call number is generated for the canvas operation script, and the canvas operation script and the unique call number are sent to the engine side via the message channel; S5. Receive the canvas operation script and the unique call number on the engine side, dynamically construct a restricted asynchronous function based on a preset whitelist tool list and inject the whitelist tool list as a parameter into the restricted asynchronous function, execute the canvas operation script in the restricted asynchronous function to operate the canvas document, obtain the script execution result, and send the script execution result and the unique call number back to the interface side via the message channel; S6. On the interface side, match the callback according to the unique call number, append the script execution result to the dialogue history in the form of a user message, and return to S2 to execute the next round of reasoning until the round type is the terminated round or the aborted round to end the current session. 2.The method of claim 1, wherein, The message channel carries the following three types of messages with type constraints: The script execution message and the script execution result message are used to carry the canvas operation script sent from the interface to the engine and the script execution result returned by the engine. The script execution message and the script execution result message are paired with each other through the unique call number. The canvas status message is used to carry the structured state from the engine side to the interface side. The canvas status message is actively pushed from the engine side to the interface side in two situations: when the selection area changes and when the canvas operation script is completed. Network proxy messages and network proxy response messages are used to carry network request forwarding initiated by the interface side to the engine side and network response returned by the engine side. In step S4, the method further includes maintaining a waiting callback pairing table on the interface side, and recording the unique call number and the waiting callback in the waiting callback pairing table; in step S5, the method further includes actively pushing the canvas status message once by the engine side after the script execution result is returned to synchronize the structured status. 3.The method of claim 2, wherein, Prior to S2, a network proxy process is also included, which includes: In response to the network request initiated by the interface side to the large language model being intercepted by the restricted operating environment of the interface side, the network request is forwarded to the engine side via the network proxy message; The engine side initiates the network request on behalf of the user with its running permissions, receives the response returned by the large language model, and sends the response back to the interface side via the network proxy response message.

4. The automatic editing method of a graphic design canvas based on a large language model according to claim 1, wherein, The S4 further includes: performing an initial format check on the canvas operation script by the interface side before sending the canvas operation script to the engine side; S5 includes the following sub-steps: S51. Perform a secondary format check on the received canvas operation script, wherein the check rules for the secondary format check are the same as those for the initial format check; S52. In response to the passing of the secondary format verification, the restricted asynchronous function is dynamically constructed. The formal parameter of the restricted asynchronous function is the name of each tool in the whitelist tool list, and the function body of the restricted asynchronous function is the canvas operation script, so that the canvas operation script can access the tools in the whitelist tool list only. S53. The execution process of the canvas operation script is wrapped in an inner exception capture structure, and the construction process of the restricted asynchronous function is wrapped in an outer exception capture structure. The inner exception capture structure is used to capture the syntax errors and runtime exceptions of the canvas operation script itself, and the outer exception capture structure is used to capture the construction exceptions of the restricted asynchronous function. Both exception capture structures convert exceptions into structured failure results. S54. The script execution result and the unique call number are sent back to the interface side, and the structured state is pushed synchronously once after the script execution result is sent back. 5.The method of claim 1, wherein, The presentation layer stripping in S2 and the multi-stage extraction in S3 process the same response content in the following ways: The presentation layer strips the response content during the streaming reception process in real time. The stripped objects include completed canvas operation code blocks, task planning signaling blocks, step start and step end signaling blocks, and content wrapped by inference tags inside the model. The tail of unclosed code blocks is truncated so that the text presented in the dialog bubble does not contain the stripped objects. The multi-stage extraction of the complete response content after the convection reception is completed will attempt the following four stages sequentially and terminate upon success in any one of the stages: In the first stage, the canvas operation script is matched according to the standard code fence containing the agreed canvas markers; In the second stage, in response to the failure of the first stage, the canvas operation script is matched according to the loose format of the code fence header containing the canvas keyword; In the third stage, in response to the failure of the second stage, the canvas operation script is matched according to the format that the first line of the code fence content contains the agreed canvas mark; In the fourth stage, in response to a failure in the third stage, the canvas operation script is matched according to the no-code fence and the paragraph format starting with the agreed canvas marker; The multi-stage extraction also includes performing standardized cleaning on the hit canvas operation scripts. The standardized cleaning includes removing the first line identifier prefix, removing the first and last whitespace, retaining the first occurrence of repeated script blocks, and judging scripts that exceed the preset length limit as extraction failures. 6.The method of claim 1, wherein, S2 to S6 are arranged in a directed state diagram on the interface side. The directed state diagram includes nodes for calling large language models, tool execution, parsing and replying, executing scripts, and feedback formatting. The large language model calling node is responsible for S2; the parsing and response node is responsible for the multi-stage extraction and round type determination in S3; the script execution node is responsible for the process on the interface side in S4 and S5. The feedback formatting node is responsible for appending the script execution result to the dialogue history in the form of a user message in S6; the tool execution node is used to execute the tool call request in parallel when the response content of the large language model contains a tool call request and append the return content of the tool call request to the dialogue history; The directed state graph sets an upper limit for the number of planning rounds and an upper limit for the total number of steps. In response to the cumulative number of planning rounds of the round type reaching the upper limit for the number of planning rounds, it forcibly jumps to the termination round. If the cumulative number of node jumps in this session reaches the overall step limit, the session will be forcibly terminated. 7.The method of claim 1, wherein, Prior to S2, the method further includes: Maintain multiple skill texts indexed by name, each skill text carrying a domain specification oriented towards a specific design task; In the system instructions submitted to the large language model, only the task completion signaling, the canvas operation code block format convention, and the operation pre-confirmation convention are retained, and the domain specifications are not directly written into the system instructions. The tool loading tool is declared in the tool declaration list submitted to the large language model. The tool declaration list is independent of the whitelist tool list. The tool declaration list is used for the large language model to perceive and initiate tool call requests. The whitelist tool list is used for the restricted asynchronous function to access when executing the canvas operation script. In response to the large language model outputting a tool call request for the skill loading tool after the multi-stage extraction in S3, the interface side retrieves the corresponding skill text according to the name in the tool call request and appends the content of the skill text to the dialogue history.

8. The automatic editing method of a graphic design canvas based on a large language model according to claim 1, wherein, The enhanced user message in S1 adopts a multimodal format, which is obtained by the interface side by combining the user's natural language message, the locally cached structured state, and the locally cached bitmap data; The structured state and the bitmap data are cached locally on the interface side through the following mechanism: Before S1 is executed, the engine side continuously monitors selection area change events. In response to the selection area change events, the engine side pushes the latest structured state to the interface side and the interface side caches it locally. In response to the selection change event, the engine initiates asynchronous image export for a preset number of selected layers in the selected layers, and assigns a monotonically increasing selection version number to this asynchronous image export. The output of the asynchronous image export is the bitmap data of each of the preset number of selected layers. In response to the asynchronous image export being completed and the selection area version number being the latest version, the bitmap data is pushed to the interface side and cached locally on the interface side; in response to the asynchronous image export being completed and the selection area version number being incremented by subsequent selection area change events, the bitmap data is discarded.

9. A computer device, comprising: include: One or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, the one or more applications being configured to perform the automatic editing method for a graphic design canvas based on a large language model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described in any one of claims 1 to 8, which is a method for automatically editing a graphic design canvas based on a large language model.

Images