An interactive multi-agent collaborative three-dimensional model editing method and system

Abstract

The application discloses an interactive multi-agent cooperative three-dimensional model editing method and system, and the method comprises the following steps: a planning agent receives a natural language editing instruction, disassembles the instruction into ordered subtasks, and generates a structured instruction containing a candidate tool, an editing primitive, a target description, initial parameters and completion conditions. An action agent relies on the instruction to locate a target anchor point in four-part recursive positioning, matches three types of underlying editing primitives, namely, smearing, dragging and drawing, generates a screen track, retains a model snapshot before editing, and completes in-situ modification of a three-dimensional model through a software adaptation layer. A reflection agent compares before and after views, checks whether the editing meets the requirements and whether there is regional misediting, and if the editing meets the requirements, the next subtask is promoted, if the editing does not meet the requirements, the parameters are adjusted and the editing is retried, and if the retry reaches an upper limit, the snapshot is rolled back and user feedback is requested. The application provides reliable technical support for natural language interactive in-situ editing of a three-dimensional model.

Description

Technical Field

[0001] This invention belongs to the field of 3D model editing technology, and more specifically, relates to an interactive multi-agent collaborative 3D model editing method and system. Background Technology

[0002] 3D model editing is widely used in digital content production, industrial design, animation production, virtual reality, and additive manufacturing. For existing surface models such as triangular meshes, operators need to adjust the viewpoint, locate the target area, and configure various editing tools within professional 3D software to complete operations such as dragging, sculpting, and drawing. This work requires a high level of professional knowledge from the operator, and ordinary users cannot operate it independently.

[0003] Traditional geometric processing algorithms rely on curvature, topology, and energy functions to achieve mesh deformation and smoothing. They can only handle standardized constraint instructions and cannot interpret natural language editing instructions with visual semantics and local positioning requirements. Schemes that use generative models to reconstruct 3D assets or edit 2D graphics and then reconstruct 3D models are prone to losing occluded areas, fine geometric and topological information, and may modify areas of the model that do not need adjustment, making local in-situ editing impossible.

[0004] Solutions that rely on large language models to generate software scripts and API calls are only compatible with parametric and procedural models. When faced with general mesh models without structured scripts, they struggle to accurately map semantic requirements to local geometric parameters and cannot verify the editing visual effects in real time. Multimodal models that directly output mouse trajectories and pixel coordinates for automated operation of 3D software have a high dimension of motion space and are prone to issues such as target offset and accidental modification of irrelevant areas. Errors accumulate continuously in multi-step tasks.

[0005] Existing technical solutions all have obvious shortcomings. The industry lacks an in-situ 3D model editing solution that integrates natural language intent, real-time software visual status, and user feedback, and relies on the collaborative execution of multiple intelligent agents with clear division of labor, verifiable and rollback capabilities. Summary of the Invention

[0006] This invention aims to solve the problems of existing 3D editing technologies, such as difficulty in parsing natural language, easy mis-modification of irrelevant areas of the model, error accumulation, and lack of verification and rollback mechanisms. It provides a multi-agent collaborative solution that achieves verifiable and rollback-enabled interactive local editing of 3D models through instruction decomposition, visual positioning, in-situ editing, and quantitative reflection correction.

[0007] To address the aforementioned deficiencies or improvement needs of existing technologies, as a first aspect of this invention, the present invention provides an interactive multi-agent collaborative 3D model editing method, characterized by comprising: S1. The planning agent decomposes the received natural language editing instructions into multiple ordered sub-tasks, and generates structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. S2. The action agent determines the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive positioning strategy, and selects the corresponding low-dimensional editing primitive based on the editing primitive type; S3. The action agent generates a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and performs in-situ editing operations on the existing 3D model in the 3D software through the 3D software adaptation layer, and saves a checkpoint snapshot of the current model state before performing the in-situ editing operation. S4. The reflecting agent compares the edited view with the view before editing, and evaluates whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, it outputs the edited 3D model and proceeds to the next subtask. If it does not meet the conditions, it triggers a bounded correction strategy. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches the preset limit, it requests user intervention and feedback.

[0008] Furthermore, before generating the structured editing instructions, the planning agent in S1 also includes: Information related to the natural language editing instructions is retrieved from a reference information base, which includes a 3D software operation knowledge base, user profiles, and an editing history case library. The 3D software operation knowledge base stores the functional descriptions and parameter constraints of various tools in the 3D software. The user profiles store user preference settings. The editing history case library stores the sub-task sequences of historical editing tasks and their corresponding execution parameters and reflection evaluation results.

[0009] Furthermore, the four-part recursive positioning strategy in S2 specifically includes: The current view of the 3D software is divided into a 2×2 four-quadrant grid area. The action agent selects the grid area where the target is located according to the target description. The selected grid area is recursively subdivided and the selection operation is repeated until the pixel accuracy of the selected area meets the preset positioning threshold, so as to finally determine the anchor point position.

[0010] Furthermore, the low-dimensional editing primitives in S2 include smearing / covering primitives, directional dragging primitives, and trajectory drawing primitives; The smearing and covering primitives are suitable for tasks involving smoothing, uniform sculpting, texture overlay, or repeated operations within a region; the directional dragging primitives are suitable for elongation, compression, bending, posture adjustment, and local protrusions; and the trajectory drawing primitives are suitable for text, symbols, patterns, and high-frequency surface details.

[0011] Furthermore, the smearing / covering primitive, the directional dragging primitive, and the trajectory drawing primitive are connected in series to form a composite editor. The smearing / covering primitive is used to solve the area coverage problem, the directional dragging primitive is used to solve the local displacement problem, and the trajectory drawing primitive is used to solve the fine path problem. For operations that do not belong to the three types of primitives, the planning agent calls the existing discrete tools in the software to expand new primitives.

[0012] Furthermore, the generation of the screen space trajectory in S3 specifically includes: When the editing primitive type is a smear overlay primitive, the mask outline of the target area is obtained, and the mask outline is subjected to a layer-by-layer erosion operation to generate a layer-by-layer overlay trajectory from the outside to the inside as the screen space trajectory. When the editing primitive type is a directional drag primitive, the anchor point is taken as the starting point, the end point is determined according to the displacement vector in the initial parameters, and the starting point and the end point are connected to generate a straight drag trajectory as the screen space trajectory. When the editing primitive type is a trajectory drawing primitive, the skeleton of the target pattern is extracted to obtain a skeleton map, and the skeleton map is traversed to generate a continuous drawing path as the screen space trajectory.

[0013] Furthermore, the in-situ editing operation of the existing 3D model in the 3D software through the 3D software adaptation layer in S3 specifically includes: The 3D software adaptation layer converts the screen space trajectory into a sequence of native tool call commands of the 3D software. The sequence of tool call commands includes tool selection commands, parameter setting commands, and trajectory execution commands. Before executing the trajectory execution command, the 3D software adaptation layer automatically switches the editing mode of the 3D software according to the type of the candidate software tool, and restores the editing mode before execution after execution.

[0014] Furthermore, the bounded correction strategy in S4 specifically includes: When the editing result does not meet the completion condition, the reflective agent generates a correction suggestion containing a description of the failure reason. The action agent adjusts the initial parameters according to the correction suggestion and then re-executes S2 and S3, incrementing the retry count by one. When the number of retries reaches the preset limit, the system reverts to the model state corresponding to the most recent checkpoint snapshot, presents a comparison of the views before and after editing and a description of the reason for the failure to the user, and waits for the user to provide supplementary constraints or confirm acceptance of the current editing result through natural language before continuing to execute the next subtask.

[0015] As a second aspect of the present invention, an interactive multi-agent collaborative 3D model editing system is also provided, comprising: The task decomposition and instruction generation unit is used to plan the agent to decompose the received natural language editing instructions into multiple ordered sub-tasks, and generate structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. The target localization and primitive selection unit is used by the action agent to determine the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive localization strategy, and to select the corresponding low-dimensional editing primitive based on the editing primitive type. The trajectory generation and in-situ editing unit is used by the action agent to generate a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and to perform in-situ editing operations on the existing 3D model in 3D software through the 3D software adaptation layer, and to save a checkpoint snapshot of the current model state before performing the in-situ editing operation. The reflection evaluation and bounded correction unit is used by the reflective agent to compare the edited view with the view before editing, evaluate whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, the edited 3D model is output and the next subtask is entered. If it does not meet the conditions, a bounded correction strategy is triggered. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches a preset limit, user intervention and feedback are requested.

[0016] As a third aspect of the invention, a computer-readable storage medium is also provided, on which a computer program is stored, which is executed by a processor to provide an interactive multi-agent collaborative 3D model editing method as described in any one of the claims.

[0017] In summary, compared with the prior art, the above-described technical solutions conceived by this invention can achieve the following beneficial effects: 1. The interactive multi-agent collaborative 3D model editing method of the present invention decomposes natural language editing instructions by planning agents and generates standardized structured instructions. It improves the instruction content by relying on a reference information database containing software operation knowledge, user preferences and historical cases. It breaks down complex editing requirements into ordered sub-tasks, unifies and standardizes the tools, parameters and completion standards of various editing operations, eliminates the conversion barrier between natural language semantics and 3D software operation logic, and allows users to issue editing requirements without having professional modeling knowledge, thereby reducing the operation threshold of interactive 3D model editing.

[0018] 2. The interactive multi-agent collaborative 3D model editing method of the present invention uses an action agent to lock the target anchor point through a four-partition recursive positioning strategy, matches three types of fixed low-dimensional editing primitives to generate an adaptive screen space trajectory, and completes in-situ editing with the help of native tool commands of the 3D software adaptation layer conversion. Before editing, a snapshot of the model checkpoint is saved to completely preserve the original model state. The entire process follows a unified underlying operation logic, can adapt to any target area under the current software viewport, avoids the target misalignment and trajectory distortion problems caused by directly outputting pixel coordinates, and ensures that local editing operations are accurately applied to the specified model parts.

[0019] 3. The interactive multi-agent collaborative 3D model editing method of the present invention, through reflective agents executing hierarchical bounded correction strategies based on verification results, forms a closed-loop control system with parameter retry, snapshot rollback, and user intervention, gradually eliminating the cumulative errors generated by multi-step editing. After each sub-task is completed, effect verification is performed to avoid irrelevant model areas being mistakenly modified, thereby improving the controllability and stability of 3D in-situ editing results. Attached Figure Description

[0020] Figure 1 This is a flowchart of an interactive multi-agent collaborative 3D model editing method according to an embodiment of the present invention; Figure 2 This is a schematic diagram of the overall interactive multi-agent collaborative 3D model editing system according to an embodiment of the present invention; Figure 3 This is a schematic diagram of the interactive multi-agent 3D model editing method according to an embodiment of the present invention; Figure 4 This is a schematic diagram illustrating three types of low-dimensional editing primitives and their trajectory generation methods according to an embodiment of the present invention; Figure 5 This is a schematic diagram of the coarse-to-fine target anchor point determination process for four-part recursive positioning according to an embodiment of the present invention. Figure 6 This is a schematic diagram illustrating the viewpoint selection, localization, parameterization, and interactive execution process of the action intelligent agent according to an embodiment of the present invention. Figure 7 This is a schematic diagram illustrating the closed loop of reflection and correction, user interaction, and reference information database update in an embodiment of the present invention. Figure 8 This is a schematic diagram of the system units in an embodiment of the present invention. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0022] Example 1 Please refer to Figure 1 This embodiment 1 provides an interactive multi-agent collaborative 3D model editing method, including: S1. The planning agent decomposes the received natural language editing instructions into multiple ordered sub-tasks, and generates structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. S2. The action agent determines the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive positioning strategy, and selects the corresponding low-dimensional editing primitive based on the editing primitive type; S3. The action agent generates a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and performs in-situ editing operations on the existing 3D model in the 3D software through the 3D software adaptation layer, and saves a checkpoint snapshot of the current model state before performing the in-situ editing operation. S4. The reflecting agent compares the edited view with the view before editing, and evaluates whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, it outputs the edited 3D model and proceeds to the next subtask. If it does not meet the conditions, it triggers a bounded correction strategy. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches the preset limit, it requests user intervention and feedback.

[0023] This embodiment describes 3D model editing as a closed-loop process involving the user, planning agent, action agent, reflective agent, 3D editing software, and reference information database. This embodiment does not require retraining a dedicated model for each editing task. Instead, it utilizes a basic model with language understanding and visual perception capabilities to complete semantic parsing, view judgment, and result evaluation, while leaving the actual geometric modifications to the editing tools of the 3D software itself. Thus, the model's visual reasoning ability, the software's geometric processing ability, and the user's intent judgment each assume appropriate responsibilities.

[0024] like Figure 2As shown, the user inputs an existing 3D model and editing requirements through an interactive terminal. The planning agent generates sequential and dependent subtasks based on the current model state and reference information library, and converts each subtask into structured editing instructions. The action agent selects the viewpoint, determines the target area, selects editing primitives and tool parameters according to these instructions, and executes the interaction through the 3D software adaptation layer. The reflection agent compares the before-and-after views with the subtask objectives, deciding whether to proceed, whether local corrections are needed, or whether to roll back. The user can provide feedback at any stage, and successful operations, along with the task, action, result, and evaluation, are written to the experience storage module.

[0025] The inputs in this embodiment include at least: an existing 3D model to be edited, natural language instructions describing the desired changes, and the 3D software runtime environment. Optional inputs include user-annotated target parts, prohibited modification areas, style preferences, custom brushes, editing intensity range, maximum allowed execution time, and phased feedback. The 3D model can be a triangular mesh, a quadrilateral mesh, or other 3D surface representation that can be displayed in the software viewport and modified by interactive tools.

[0026] The method in this embodiment maintains task-level and subtask-level states during execution. The task-level state records the original instructions, model identifier, global constraints, subtask dependencies, current execution position, and user feedback history. The subtask-level state records the model snapshot before editing, selected viewpoint, target region, primitive type, tool parameters, trajectory, post-edit view, reflection score, text evaluation, and number of retries. A recoverable checkpoint is established before each execution to undo the modification if significant changes occur in non-target regions or if the user rejects the result.

[0027] The planning agent sends structured editing instructions to the action agent. These instructions serve as the data carrier in the natural language processing solution and are not limited to a specific serialization format; they can be represented by key-value fields, message objects, or database records. The necessary fields and their functions are shown in Table 1.

[0028] Table 1 Required fields for structured editing commands The output of this embodiment includes the edited 3D model, execution records of each subtask, before-and-after comparison views, and the final evaluation results. The model can be saved in a format supported by the 3D software; the execution records are used for review, reproduction, and subsequent case retrieval, but it is not required to expose the internal reasoning process of the language model.

[0029] Please refer to Figures 3 to 7 Specifically, this embodiment will further elaborate on the method used.

[0030] (1) Task decomposition and instruction generation Load the existing 3D model and establish initial checkpoints. Perform basic integrity checks on user instructions, and identify editing objects, expected changes, prohibited modification areas, and output requirements.

[0031] Then, based on user instructions, model category, and current software environment, the system retrieves task-related tool descriptions, user information, and historical success cases from the reference information database. Historical cases are sorted using both semantic similarity and operational conditions, returning a preset number of candidate cases; for one implementation method, the five cases with the highest similarity are selected. The search results are used only as a planning basis and do not directly replace the visual judgment of the current model.

[0032] The planning agent comprises a task decomposition unit and an instruction translation unit. The task decomposition unit answers "which local changes need to be completed sequentially," while the instruction translation unit answers "how the current local changes can be implemented using software tools and editing primitives." Both receive the current model view and user feedback, but the task decomposition unit primarily maintains subtask dependencies, while the instruction translation unit primarily maintains executable parameters. This division of labor allows execution failures to typically be handled within the current subtask, rather than requiring frequent rewriting of the entire plan.

[0033] The task decomposition unit of the planning agent breaks down complex requirements into ordered subtasks based on the current model state, user instructions, and retrieval results. For example, "bringing a person's arms inward and writing on their chest" is broken down into left arm adjustment, right arm adjustment, chest surface preparation, and text drawing. Each subtask clearly defines the target part, dependencies, and unmodifiable areas.

[0034] The instruction translation unit of the planning agent determines candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions for the current subtask. The translation results can be locally updated during execution based on views, location reliability, and reflective opinions, without requiring the entire task to be replanned every time parameters are adjusted. Based on the updated translation results, structured editing instructions containing candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions are generated for each subtask.

[0035] Meanwhile, the reference information base includes at least three sub-bases. The first is a 3D software knowledge base, storing official tool instructions, operating conditions, parameter meanings, and available interfaces; the second is a user-provided resource base, storing custom brushes, style specifications, prohibited areas, and preferences; and the third is a historical success case database, storing tasks, actions, and results that have been reviewed and evaluated. Software knowledge and user resources can be established before deployment, while the success case database is continuously updated during operation.

[0036] When a user adds constraints during execution, the interactive terminal converts the feedback into additional conditions for the current subtask. For example, if the user specifies "only move the ear tip, do not change the ear root," the planning agent adds the ear root to the prohibited modification area, the action agent reduces the positioning mask and pen radius, and the reflection agent focuses on comparing the ear root region during evaluation. If the feedback changes the overall goal, subsequent dependencies are regenerated starting from the earliest affected subtask.

[0037] (2) Target positioning and primitive selection The action agent is responsible for translating structured editing instructions into actual operations in 3D software. Its execution chain includes viewpoint selection, target localization, parameter configuration, trajectory generation, interactive execution, and state acquisition. Unlike directly having the language model output a large number of free coordinates, this embodiment summarizes common local editing into three categories of low-dimensional primitives, allowing the model to determine only a small number of semantically necessary parameters, with trajectory details generated by deterministic processing.

[0038] The action agent generates multiple candidate viewpoints and selects the viewpoint where the target area is visible, has minimal occlusion, and a projection scale suitable for operation. One implementation uses positive and negative coordinate axes to form six standard viewpoints. Subsequently, based on the nature of the subtask, it selects primitives such as smearing, directional dragging, or trajectory drawing, and matches them with smoothing, dragging, sculpting, texturing, or custom brushes from 3D software.

[0039] Among them, the smudge overlay primitive is suitable for tasks that require smoothing, uniform sculpting, texture overlay, or repeated operations within an area; the directional drag primitive is suitable for elongation, compression, bending, posture adjustment, and local protrusions; and the trajectory drawing primitive is suitable for text, symbols, patterns, and high-frequency surface details.

[0040] The three types of primitives mentioned above can be used individually or chained together to form composite edits. Smearing and covering solves area coverage problems, directional dragging solves local displacement problems, and trajectory drawing solves fine-grained path problems. For operations that do not belong to these three types of primitives, the planning agent can call upon existing discrete tools in the software to extend new primitives.

[0041] Multimodal models are generally more stable than directly outputting pixel coordinates when discretely selecting which region the target is located in. Based on this characteristic, this embodiment sets up a four-region recursive localization process: a 2x2 grid is overlaid on the current view, and four regions are assigned distinct numbers, textures, or other distinguishable identifiers; the visual model is queried about which region the target anchor point is located in; the selected region is cropped and re-divided; the above process is repeated until the width, height, or diagonal length of the sub-region does not exceed a preset localization scale. Finally, the sub-region center, salient points within the sub-region, or local segmentation results are used as anchor points.

[0042] The stopping condition for this process is determined by both the target scale and the viewport resolution. For large brush drags, it can stop earlier; for small parts such as ear tips and fingers, further subdivision is required. If two consecutive selections yield contradictory results, or if the target is near the boundary of an adjacent partition or partially occluded, the anchor point is not forcibly output; instead, the viewing angle is switched, the context area is expanded, or the user is asked to click for confirmation. Partition identifiers can use numbers, letters, colors, or textures; the patent drawings use numbers and wireframes, without limiting the visual encoding method of the actual interface.

[0043] For area-type targets, the action agent first generates a coarse bounding box using a multimodal model, then calls an image segmentation tool to obtain the target mask. For parts requiring single-point dragging, a four-part recursive localization method is used to gradually shrink the target area until the sub-region scale is no larger than a preset localization threshold. Then, the center of the sub-region or the position with the highest confidence level is used as the anchor point. Low-confidence results require switching the viewpoint, re-segmentation, or requesting user confirmation.

[0044] (3) Trajectory generation and in-situ editing The 3D software adaptation layer sits between the action agent and the specific software, shielding the differences in window layout, tool names, and input interfaces between different software. The adaptation layer maintains the software window position, viewport boundaries, current mode, selected objects, active tools, and coordinate transformation relationships, and provides the action agent with unified capabilities for viewpoint switching, tool selection, parameter settings, trajectory execution, screenshotting, undoing, and model saving.

[0045] The action agent generates a screen-space trajectory based on primitives and localization results, which is then converted into the current window and viewport coordinates by a software adaptation layer. The adaptation layer is responsible for selecting tools, setting parameters, calibrating viewport boundaries, and generating input events such as press, hold, move, and release. A model snapshot is saved before execution, and the same viewpoint and necessary supplementary viewpoints are acquired after execution.

[0046] When the editing primitive type is a smear / cover primitive, the action agent first obtains the target bounding box in the current view. Using the bounding box as a cue, the image segmentation model generates a binary mask. Then, the outer contour is extracted, and the mask is eroded layer by layer according to the brush radius. Trajectory points are sampled along each contour layer and connected in an outward-to-inward order to form a continuous trajectory with uniform coverage that avoids crossing the mask boundaries as much as possible. The language model primarily determines the tool, radius, intensity, and target region, and does not directly generate dense mouse points.

[0047] When the editing primitive type is a directional drag primitive, its necessary parameters are the target anchor point and the displacement vector. The anchor point is obtained through four-part recursive positioning, keypoint detection, or user confirmation; the displacement direction and magnitude are determined according to user commands, the current viewpoint, and the target scale. The software adaptation layer converts this vector into viewport displacement and performs press, hold, move, and release operations. For bilaterally symmetrical targets, anchor points on both sides can be located separately, and two sub-tasks can be generated according to symmetry or mirror constraints. Taking the anchor point position as the starting point, the endpoint is determined according to the displacement vector in the initial parameters, and connecting the starting point and the endpoint generates a straight drag trajectory as the screen space trajectory.

[0048] When the editing primitive type is a trajectory drawing primitive, for text, the target text is rasterized into a binary image, the central skeleton is extracted, the intersection points and endpoints are used to construct a topological graph, and the graph edges are traversed in a manner close to handwriting. For any pattern, a user image or a high-contrast reference image can be obtained based on the text description, and then the main outline is extracted. The resulting path is scaled, rotated, and transformed before being mapped to the target area and executed by the corresponding sculpting or texturing tools. The skeleton of the target pattern is extracted to obtain a skeleton image, and graph traversal is performed on the skeleton image to generate a continuous drawing path as the screen space trajectory.

[0049] The 3D software adaptation layer converts the screen space trajectory into a sequence of native tool call commands of the 3D software. The sequence of tool call commands includes tool selection commands, parameter setting commands, and trajectory execution commands. Before executing the trajectory execution command, the 3D software adaptation layer automatically switches the editing mode of the 3D software according to the type of the candidate software tool, and restores the editing mode before execution after execution.

[0050] Before execution, the adaptation layer checks whether the software is in the expected object and editing mode, whether the target viewport is visible, and whether the tools are available. If the checks fail, the trajectory is not sent; instead, the window is restored, the object is reselected, or a report is sent to the user. During execution, the trajectory start and end times and tool parameters are recorded. After execution, a model snapshot and screenshot are saved. If the viewport size or window position changes, the transformation between screen coordinates and viewport coordinates is recalibrated before continuing the operation.

[0051] One implementation runs on Windows 11 and Blender 4.5 LTS, simulating mouse actions via screen input events, using the software's native sculpting tools to process geometry, and generating target masks from the segmented model. Specific implementations can utilize PyAutoGUI, operating system automation interfaces, software plugins, remote control protocols, or the 3D software's own application programming interface (API). Regardless of the interface used, action execution is constrained by structured editing instructions, target regions, and reflection evaluation.

[0052] (4) Reflection, evaluation and bounded revision The reflexive agent evaluates each executed subtask. Its inputs include the pre-edit view, post-edit view, subtask objective, prohibited modification area, and execution context. The evaluation includes at least: whether the target area has undergone the expected change, whether the direction and magnitude of the change conform to the instruction, whether there is obvious damage to the surface, whether non-target areas are preserved, and whether the current result is sufficient to proceed to the next subtask. The reflexive agent outputs a score and natural language comments; this embodiment uses a zero-to-fifteen-point scale and preset thresholds.

[0053] When the score reaches the threshold, the result is accepted and a successful case is saved. When the score does not reach the threshold, the written comments are converted into actionable corrections, such as reducing the brush size, decreasing the intensity, changing the perspective, repositioning the anchor point, or undoing and retrying. Automatic retries are bounded, with a maximum of three attempts per implementation. When the maximum is reached, when evaluation comments contradict each other, or when continuous modifications worsen non-target areas, the process pauses and displays a before-and-after comparison, the reason for failure, and suggested options to the user.

[0054] The bounded approach specifically includes: when the editing result does not meet the completion condition, the reflective agent generates a correction suggestion containing a description of the failure reason, the action agent adjusts the initial parameters according to the correction suggestion and then re-executes S2 and S3, and the number of retries is incremented by one; When the number of retries reaches the preset limit, the system reverts to the model state corresponding to the most recent checkpoint snapshot, presents a comparison of the views before and after editing and a description of the reason for the failure to the user, and waits for the user to provide supplementary constraints or confirm acceptance of the current editing result through natural language before continuing to execute the next subtask.

[0055] Once a subtask passes evaluation, the process moves to the next subtask, saving the task description, structured editing instructions, execution context, before-and-after views, and evaluation results as a success case. After all subtasks are completed, the edited model and process log are output. Users can still request partial corrections to the final result, requiring only the affected subtasks and their subsequent dependencies to be redesigned.

[0056] Low scores, low location reliability, user rejection, or abnormal software status will not directly push the process to continue. Instead, they will lead to partial correction, rollback, or replanning of the path, thereby preventing errors from accumulating in subsequent subtasks.

[0057] The success case library records the evaluated task descriptions, model categories, target parts, structured instructions, software environments, trajectory parameters, and visual results. When similar cases are found in subsequent tasks, the planning agent reuses its policy framework, while the action agent still needs to reselect the perspective and localization based on the current model to avoid directly using the absolute coordinates of old cases for the new model.

[0058] The interactive terminal continuously displays the current subtask, selected target area, primitives used, before and after edit previews, reflection comments, and remaining retries. Users can confirm the current result, reject and roll back, directly mark a new target, adjust constraints, pause the task, or request replanning. User feedback and automatic reflection are transmitted through the same status channel, thus enabling human-machine collaboration while maintaining the integrity of the execution record.

[0059] For various anomalies and boundary conditions, corresponding handling shall be carried out.

[0060] For accidental editing of non-target areas, use pre-edit checkpoint undo, tighten the mask, reduce brush size, or change the viewpoint, then retry within the bounded area.

[0061] If the retry limit is exceeded, stop automatic operation, retain the last successful checkpoint, and display the reason for failure, candidate parameters, and manual intervention entry point.

[0062] In response to user interruption, immediately stop new input events, wait for the current uninterruptible software operation to finish and save the state so that the task can resume from the most recent checkpoint.

[0063] The primitive does not support topology operations. For structured tasks such as Boolean operations, punching, merging objects, or retopology, invoke discrete software tools, plugins, or extended primitives; if no controlled implementation exists, report that it is not supported and do not perform approximate surface deformation.

[0064] This embodiment 1 also provides the following four typical implementation scenarios: Scenario 1: Local Surface Smoothing. The user requires "smoothing the back of the model while keeping the limbs unchanged." The planning agent sets back smoothing as a single subtask and lists the limbs as a prohibited modification area. The action agent selects the visible view of the back, generates bounding boxes and segmentation masks, selects the smearing primitive, and generates layer-by-layer contour trajectories based on the brush radius. The reflection agent compares whether the high-frequency undulations of the back have been reduced and checks whether the limb contours have been preserved; if not, it reduces the mask size or intensity and retryes.

[0065] Scenario 2: Local Shape Drag. The user request is to "make the rabbit's ears droop naturally." The planning agent sets the left and right ears as sub-tasks. The action agent selects a side-front view for each ear, recursively locates the ear tip using four partitions, generates a displacement vector based on the semantics of "downward and slightly outward," and executes it using a directional drag primitive. The reflection agent checks whether the ear roots are maintained, whether the two ears are coordinated, and whether the overall identity is preserved; if necessary, it backs down and reduces the displacement amplitude.

[0066] Scenario 3: Surface Text or Patterns. The user request is to "write text and draw faces on the surface of a pumpkin." First, the text and faces are divided into two sub-tasks. The trajectory drawing primitive generates paths from the text skeleton and pattern outline respectively, scales and projects them onto the front of the pumpkin based on the target area, and then calls the sculpting or texturing tools. The reflective agent checks whether the text legibility, pattern position, and overall pumpkin shape are preserved.

[0067] Scenario 4: Composite Interactive Editing. The user requests, "Bring the character's arms inward and write in front of their chest." The process is executed in the following order: left arm, right arm, chest preparation, and text drawing. After the arms are adjusted, a preliminary preview is displayed to the user, who can confirm the posture before continuing to draw text. If the user adds, "Do not rotate the palms," this constraint is added to the subsequent evaluation and can trigger a partial rollback of the current arm subtask.

[0068] Example 2 Please refer to Figure 8 This embodiment 2 provides an interactive multi-agent collaborative 3D model editing system, including: The task decomposition and instruction generation unit is used to plan the agent to decompose the received natural language editing instructions into multiple ordered sub-tasks, and generate structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. The target localization and primitive selection unit is used by the action agent to determine the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive localization strategy, and to select the corresponding low-dimensional editing primitive based on the editing primitive type. The trajectory generation and in-situ editing unit is used by the action agent to generate a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and to perform in-situ editing operations on the existing 3D model in 3D software through the 3D software adaptation layer, and to save a checkpoint snapshot of the current model state before performing the in-situ editing operation. The reflection evaluation and bounded correction unit is used by the reflective agent to compare the edited view with the view before editing, evaluate whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, the edited 3D model is output and the next subtask is entered. If it does not meet the conditions, a bounded correction strategy is triggered. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches a preset limit, user intervention and feedback are requested.

[0069] Example 3 This embodiment 3 also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can implement any step of an interactive multi-agent collaborative 3D model editing method.

[0070] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0072] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An interactive multi-agent collaborative 3D model editing method, characterized in that, include: S1. The planning agent decomposes the received natural language editing instructions into multiple ordered sub-tasks, and generates structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. S2. The action agent determines the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive positioning strategy, and selects the corresponding low-dimensional editing primitive based on the editing primitive type; S3. The action agent generates a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and performs in-situ editing operations on the existing 3D model in the 3D software through the 3D software adaptation layer, and saves a checkpoint snapshot of the current model state before performing the in-situ editing operation. S4. The reflecting agent compares the edited view with the view before editing, and evaluates whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, it outputs the edited 3D model and proceeds to the next subtask. If it does not meet the conditions, it triggers a bounded correction strategy. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches the preset limit, it requests user intervention and feedback.

2. The interactive multi-agent collaborative 3D model editing method according to claim 1, characterized in that, Before generating the structured editing instructions, the planning agent in S1 further includes: Information related to the natural language editing instructions is retrieved from a reference information base, which includes a 3D software operation knowledge base, user profiles, and an editing history case library. The 3D software operation knowledge base stores the functional descriptions and parameter constraints of various tools in the 3D software. The user profiles store user preference settings. The editing history case library stores the sub-task sequences of historical editing tasks and their corresponding execution parameters and reflection evaluation results.

3. The interactive multi-agent collaborative 3D model editing method according to claim 1, characterized in that, The four-part recursive positioning strategy in S2 specifically includes: The current view of the 3D software is divided into a 2×2 four-quadrant grid area. The action agent selects the grid area where the target is located according to the target description. The selected grid area is recursively subdivided and the selection operation is repeated until the pixel accuracy of the selected area meets the preset positioning threshold, so as to finally determine the anchor point position.

4. The interactive multi-agent collaborative 3D model editing method according to claim 1, characterized in that, The low-dimensional editing primitives in S2 include smudge / cover primitives, directional drag primitives, and trajectory drawing primitives; The smearing and covering primitives are suitable for tasks involving smooth, uniform engraving, texture coverage, or tasks requiring multiple repetitions within an area; The directional drag primitives are applicable to elongation, compression, bending, posture adjustment, and local protrusions; the trajectory drawing primitives are applicable to text, symbols, patterns, and high-frequency surface details.

5. The interactive multi-agent collaborative 3D model editing method according to claim 4, characterized in that, The smearing / covering primitive, the directional dragging primitive, and the trajectory drawing primitive are connected in series to form a composite editor. The smearing / covering primitive is used to solve the area coverage problem, the directional dragging primitive is used to solve the local displacement problem, and the trajectory drawing primitive is used to solve the fine path problem. For operations that do not belong to the three types of primitives, the planning agent calls the existing discrete tools in the software to expand new primitives.

6. The interactive multi-agent collaborative 3D model editing method according to claim 5, characterized in that, The generation of the screen space trajectory in S3 specifically includes: When the editing primitive type is a smear overlay primitive, the mask outline of the target area is obtained, and the mask outline is subjected to a layer-by-layer erosion operation to generate a layer-by-layer overlay trajectory from the outside to the inside as the screen space trajectory. When the editing primitive type is a directional drag primitive, the anchor point is taken as the starting point, the end point is determined according to the displacement vector in the initial parameters, and the starting point and the end point are connected to generate a straight drag trajectory as the screen space trajectory. When the editing primitive type is a trajectory drawing primitive, the skeleton of the target pattern is extracted to obtain a skeleton map, and the skeleton map is traversed to generate a continuous drawing path as the screen space trajectory.

7. The interactive multi-agent collaborative 3D model editing method according to claim 1, characterized in that, The in-situ editing operation of the existing 3D model in 3D software through the 3D software adaptation layer in S3 specifically includes: The 3D software adaptation layer converts the screen space trajectory into a sequence of native tool call commands of the 3D software. The sequence of tool call commands includes tool selection commands, parameter setting commands, and trajectory execution commands. Before executing the trajectory execution command, the 3D software adaptation layer automatically switches the editing mode of the 3D software according to the type of the candidate software tool, and restores the editing mode before execution after execution.

8. The interactive multi-agent collaborative 3D model editing method according to claim 1, characterized in that, The bounded correction strategy in S4 specifically includes: When the editing result does not meet the completion condition, the reflective agent generates a correction suggestion containing a description of the failure reason. The action agent adjusts the initial parameters according to the correction suggestion and then re-executes S2 and S3, incrementing the retry count by one. When the number of retries reaches the preset limit, the system reverts to the model state corresponding to the most recent checkpoint snapshot, presents a comparison of the views before and after editing and a description of the reason for the failure to the user, and waits for the user to provide supplementary constraints or confirm acceptance of the current editing result through natural language before continuing to execute the next subtask.

9. An interactive multi-agent collaborative 3D model editing system, characterized in that, include: The task decomposition and instruction generation unit is used to plan the agent to decompose the received natural language editing instructions into multiple ordered sub-tasks, and generate structured editing instructions for each sub-task, including candidate software tools, editing primitive types, target descriptions, initial parameters, and completion conditions. The target localization and primitive selection unit is used by the action agent to determine the anchor point position of the target area in the current view of the 3D software according to the structured editing instructions through a four-part recursive localization strategy, and to select the corresponding low-dimensional editing primitive based on the editing primitive type. The trajectory generation and in-situ editing unit is used by the action agent to generate a screen space trajectory based on the low-dimensional editing primitive and the anchor point position, and to perform in-situ editing operations on the existing 3D model in 3D software through the 3D software adaptation layer, and to save a checkpoint snapshot of the current model state before performing the in-situ editing operation. The reflection evaluation and bounded correction unit is used by the reflective agent to compare the edited view with the view before editing, evaluate whether the editing result meets the completion conditions and does not destroy non-target areas. If it meets the conditions, the edited 3D model is output and the next subtask is entered. If it does not meet the conditions, a bounded correction strategy is triggered. The bounded correction strategy includes retrying after parameter adjustment or rolling back to the checkpoint snapshot. When the number of retries reaches a preset limit, user intervention and feedback are requested.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, The computer program is executed by a processor as described in any one of claims 1-8: an interactive multi-agent collaborative 3D model editing method.

Images