Text-driven 3D mesh editing methods, model training methods, and related devices

By constructing a control group input and a neural network-generated text structure response field, local regions of a 3D mesh are located and edited, solving the problems of complexity and low efficiency in traditional 3D mesh editing and achieving efficient and accurate local editing.

CN123071591APending Publication Date: 2026-11-10HANGZHOU QUNHE INFORMATION TECHNOLOGIES CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611585085.3
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-13
Publication Date
2026-11-10

AI Technical Summary

Technical Problem

Traditional 3D mesh editing relies on manual operation by professionals, which is complex, time-consuming, and requires high user skills. Existing text-driven methods are inefficient and difficult to achieve local consistency.

Method used

By constructing a control group input, the first-stage neural network generates a text structure response field, locates the editing region and generates a target sparse support set, and combines it with the second-stage neural network to generate an editing grid, thereby achieving local editing.

Benefits of technology

It improves the accuracy and ease of use of 3D mesh editing, reduces structural drift and inconsistencies in multi-view results, and enhances editing efficiency and consistency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123071591A_ABST
    Figure CN123071591A_ABST
Patent Text Reader

Abstract

This disclosure provides a text-driven 3D mesh editing method, model training method, and related apparatus. It relates to the field of computer technology. The specific implementation scheme is as follows: a control group input is constructed based on text instructions and empty text; in the control group input, the first source condition features of the mesh to be edited are the same, and the noisy structural latent variables are the same, but the text conditions are different; the control group input is processed based on a first-stage neural network to obtain the speed difference between the first-stage neural network under the text conditions corresponding to the text instructions and the text conditions corresponding to the empty text; a text structure response field is generated based on the speed difference; the text structure response field is applied to the generation process from structural latent variables to the target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; a second-stage neural network is used to process the target sparse support set and the text instructions to obtain the editable mesh.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of space generation, spatial understanding, and artificial intelligence. Background Technology

[0002] 3D meshes are one of the most commonly used forms of 3D content representation in fields such as game development, film and television special effects, virtual reality, and industrial design. In practical applications, users often need to make local modifications to existing 3D meshes, such as adding armrests to a chair, replacing the handle of a cup, or changing the shape of a component of an object. Traditional 3D mesh editing relies on professionals manually completing the task using modeling software, requiring the adjustment of geometric elements such as vertices and faces one by one. This process is complex, time-consuming, and demands a high level of expertise from the user. With the development of generative artificial intelligence, using text commands in natural language to directly drive 3D mesh editing has become an important direction for lowering the barrier to entry for 3D content creation. Summary of the Invention

[0003] This disclosure provides a text-driven 3D mesh editing method, model training method, and related apparatus to solve or alleviate one or more technical problems in the prior art.

[0004] According to one aspect of this disclosure, a text-driven 3D mesh editing method is provided, comprising: A control group input was constructed based on text instructions and empty text. In the control group input, the first source condition features of the grid to be edited were the same, and the noisy structural latent variables were the same, but the text conditions were different. Based on the first-stage neural network processing of the control group input, the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text is obtained. The text structure response field is generated based on the speed difference; the text structure response field is used to represent the degree of change of different positions in the structural latent space by the text command; The text structure response field is applied to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; wherein, the two-stage 3D generation framework includes a first-stage neural network and a second-stage neural network. The second-stage neural network is used to process the target sparse support set and text instructions to obtain the edit mesh.

[0005] According to another aspect of this disclosure, a text-driven 3D mesh editing device is provided, comprising: The first construction module is used to construct a control group input based on text instructions and empty text. In the control group input, the first source condition features of the grid to be edited are the same, and the noisy structural latent variables are the same, but the text conditions are different. The first processing module is used to process the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text. The first generation module is used to generate a text structure response field based on the speed difference; the text structure response field is used to represent the degree of change of the text instruction to different positions in the structural latent space; The first support set generation module is used to apply the text structure response field to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner, thereby obtaining the target sparse support set; wherein, the two-stage three-dimensional generation framework includes a first-stage neural network and a second-stage neural network. The second processing module is used to process the target sparse support set and text instructions using the second-stage neural network to obtain the edit mesh.

[0006] According to another aspect of this disclosure, an electronic device is provided, comprising: At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform any of the methods described in the embodiments of this disclosure.

[0007] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform any of the methods according to embodiments of this disclosure.

[0008] According to another aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements any of the methods according to embodiments of this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the various drawings denote the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings depict only some embodiments provided according to this disclosure and should not be construed as limiting the scope of this disclosure.

[0011] Figure 1 This is a flowchart illustrating a text-driven 3D mesh editing method according to an embodiment of the present disclosure; Figure 2 This is a schematic diagram of a process for constructing a control group input based on text instructions and empty text according to an embodiment of the present disclosure; Figure 3 This is a schematic flowchart illustrating the process of obtaining speed differences according to an embodiment of the present disclosure; Figure 4 This is a schematic flowchart illustrating the generation of a text structure response field based on the speed difference according to an embodiment of the present disclosure; Figure 5 This is a schematic diagram of the process for obtaining a target sparse support set according to an embodiment of the present disclosure; Figure 6 This is a schematic diagram of the process of obtaining an edited mesh according to an embodiment of the present disclosure; Figure 7 This is a schematic flowchart of a model training method according to an embodiment of the present disclosure; Figure 8 This is a schematic diagram of the overall process of a text-driven 3D mesh editing method according to an embodiment of the present disclosure; Figure 9 This is a schematic diagram of the structure of a text-driven three-dimensional mesh editing device according to an embodiment of the present disclosure; Figure 10 This is a schematic diagram of the structure of a model training apparatus according to an embodiment of the present disclosure; Figure 11 This is a schematic diagram of an electronic device structure according to an embodiment of the text-driven 3D mesh editing method and / or model training method of the present disclosure. Detailed Implementation

[0012] The present disclosure will now be described in further detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0013] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0014] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this disclosure, "multiple" means two or more, unless otherwise explicitly specified.

[0015] It should be noted that, unless it is explicitly stated that there is a sequential order of execution between different operations, or that there is a sequential order of execution between different operations in terms of technical implementation, the execution order between multiple operations may not be significant, and multiple operations may be executed simultaneously.

[0016] In related technologies, text-driven 3D editing methods utilize pre-trained 2D image generation models to render 3D representations as multi-view images. They then use feedback signals from the 2D model to iteratively optimize the 3D representation scene by scene. However, these methods require lengthy iterative optimization for each mesh edit, resulting in low editing efficiency. Furthermore, because the feedback signals originate from the 2D image space, the 2D model lacks prior knowledge of the 3D structure, easily leading to inconsistencies in multi-view results and geometric distortion.

[0017] To achieve local editing, related technologies require users to manually select or draw the editing area in 3D space, or to determine the editing area using a 3D segmentation model and then regenerate that area. However, manually specifying the editing area increases the user's workload. Using a segmentation model requires introducing a separate segmentation model in addition to the generation model, and the segmentation result depends on predefined component categories, making it difficult to cover open semantic text commands.

[0018] In view of this, this disclosure proposes a text-driven 3D mesh editing method to solve at least one of the above problems. For example... Figure 1 The diagram shown is a flowchart of the method, including: S101, construct a control group input based on text instructions and empty text; in the control group input, the first source condition features of the grid to be edited are the same and the noisy structural latent variables are the same, but the text conditions are different.

[0019] The mesh to be edited refers to the original 3D mesh model that needs to be edited, which can consist of a set of vertices and a set of faces. For example, the mesh to be edited can come from a 3D model uploaded by the user or the result of a 3D scan reconstruction.

[0020] Text instructions refer to natural language text used to edit the mesh. For example, text instructions can describe the addition, deletion, or replacement of local components, or changes to the overall shape style.

[0021] Empty text refers to text input that does not contain any intention to modify the mesh to be edited. For example, empty text can be an empty string or a specific string, as long as it does not contain any intention to modify the mesh to be edited.

[0022] In this embodiment of the disclosure, a 3D mesh editing task refers to a task that, for an existing 3D asset, alters the local components, geometry, pose, or state of an object according to design requirements. A 3D mesh editing task can support at least one variable dimension. The variable dimension belongs to a set of complementary dimensions. This set of complementary dimensions refers to the set of dimensions used to uniformly describe the editable state of 3D objects in the mesh to be edited, including component set dimensions, shape attribute dimensions, and spatial configuration dimensions.

[0023] The part set dimension is used to modify the part set of a 3D object. A part set refers to the types and states of the parts that make up a 3D object, such as the backrest, seat, armrests, and legs of a chair. Modifying the part set can include adding, deleting, or replacing parts. For example, adding a part could be adding a storage basket under the seat; deleting a part could be removing the armrests; and replacing a part could be replacing the wooden legs with metal legs. By modifying the part set, the component composition of a 3D object can be changed, thereby altering the overall structure and function of the 3D object.

[0024] The shape attribute dimension is used to modify the geometric properties of a 3D object. Geometric properties refer to the geometric morphological features of a 3D object's surface or local components, such as thickness, curvature, proportion, density, and quantity. Modifying geometric properties can include changing at least one of the following: thickness, curvature, proportion, density, and the number of geometric elements. For example, changing the thickness can increase the thickness of a vase's wall; changing the curvature can increase the curvature at the vase's opening; changing the proportion can lengthen the proportion of a table leg; changing the density can increase the density of the surface texture; and changing the number of geometric elements can increase or decrease the number of surface protrusions or grooves. By modifying geometric properties, the surface shape and geometric details of a 3D object can be altered without changing the object's component composition or the spatial relationships between components.

[0025] The spatial configuration dimension is used to modify the pose relationships between different parts of a 3D object. This dimension represents at least one of the spatial positions, orientations, and joint relationships between existing parts of a 3D object. For example, changes along this dimension include changes in the posture of a person or animal, and at least one state change of a movable object such as opening, closing, folding, or unfolding. Modifications to pose relationships can include at least one of joint rotation, overall posture adjustment, or state switching. For example, joint rotation could be rotating a person's elbow or knee joint; overall posture adjustment could be adjusting a person's standing or sitting posture; and state switching could be changing a movable object from an open state to a closed state or from a folded state to an unfolded state. By modifying pose relationships, the relative positions and orientations between different parts of a 3D object can be changed without altering the component composition and geometric properties of the 3D object.

[0026] The three dimensions mentioned above are complementary to each other and together constitute a unified description of the 3D mesh editing task. Compared with the way related technologies organize tasks by operation names such as adding parts, deleting parts, and replacing parts, this disclosure represents the editing task as three complementary dimensions, which can uniformly describe the set of parts, continuous or discrete shape attributes, and spatial configuration between parts, thereby covering more complete editing capabilities and enabling different editing tasks to have a composable common representation.

[0027] The first source conditional feature refers to the feature representation obtained from the grid to be edited, which is obtained by feature extraction based on a neural network.

[0028] Noisy structural latent variables refer to latent variables obtained by adding noise to source structural latent variables in the structural latent space. Here, source structural latent variables refer to the structural features corresponding to the mesh to be edited in the structural latent space.

[0029] During implementation, a control group input is constructed based on text instructions and empty text. The first set of inputs consists of noisy structural latent variables, first source conditional features, and text conditions corresponding to the text instructions. The second set of inputs includes the same noisy structural latent variables and the same first source conditional features as the first set; however, the second set of inputs contains text conditions corresponding to empty text.

[0030] S102, based on the first-stage neural network processing of the control group input, obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text.

[0031] The first-stage neural network refers to a neural network used for generation within the structural latent space. Its output, crucially, is a 3D mesh generated based on modified text instructions. In practice, the first-stage neural network can, for example, include a conditional flow Transformer, a numerical solver, and a structure decoder. The conditional flow Transformer converts the input into velocity predictions, the numerical solver updates the structural latent variables in the conditional flow Transformer's input based on these velocity predictions, and this iterative update process continues. The final updated structural latent variables are then input to the structure decoder, which decodes them into the target structural latent variables.

[0032] Speed ​​difference refers to the difference between the speed prediction under the text condition corresponding to the text instruction and the speed prediction under the condition corresponding to the empty text; that is, the difference information between the two speed predictions. As explained above, speed prediction is an intermediate product of the first-stage neural network output.

[0033] S103, Generate a text structure response field based on speed differences; the text structure response field is used to represent the degree of change of different positions in the structural latent space by the text instruction.

[0034] The text structure response field is a scalar field defined at various spatial locations within the structure latent space. It represents the degree of influence of text commands on different locations within the structure latent space. A larger response degree indicates that the location needs to be modified according to the text command. A smaller response degree indicates that the relationship between the location and the text command is weaker. When the influence degree is at its default value, it means that the original structure of the mesh to be edited should be preserved as much as possible.

[0035] S104, the text structure response field is applied to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes a first-stage neural network and a second-stage neural network.

[0036] As explained above, the structural latent variables, after being decoded by the structural decoder, yield the target sparse support set. The target sparse support set refers to the set of three-dimensional coordinates of the effective voxels occupied by the mesh to be edited, used to determine the overall geometric structure of the mesh to be edited. Effective voxels refer to the voxels in the voxel mesh obtained by dividing the space into three dimensions, which are occupied by the three-dimensional structure of the mesh to be edited after being edited by text commands.

[0037] S105 utilizes a second-stage neural network to process the target sparse support set and text instructions to obtain the edit mesh.

[0038] A second-stage neural network refers to a neural network used to generate appearance features on a sparse support set of a target. For example, a second-stage neural network could be a Transformer based on sparse conditional flow.

[0039] In practice, the first-stage neural network is used to edit the spatial position of the mesh to be edited according to the text instructions to obtain the target sparse support set, and the second-stage neural network is used to generate detailed features, such as texture and material appearance features, on the target sparse support set.

[0040] The resulting edit grid is the grid obtained after the grid to be edited has been modified by text commands.

[0041] In this embodiment, a control group input with identical first source condition features and noisy structural latent variables but different text conditions is constructed for the mesh to be edited. The text structure response field is generated using the speed difference between the text conditions corresponding to the text instruction and the text conditions corresponding to empty text in the first-stage neural network. This allows the influence of the text instruction on the generation process to be isolated from factors such as the structure of the mesh to be edited and the randomness of noise. The predictive ability of the first-stage neural network is directly utilized to locate the editing region corresponding to the text instruction and its degree of change in the structural latent space. Therefore, the entire editing process not only understands "how to modify" expressed by the text instruction but also directly indicates which 3D locations should be modified, enabling different editing intensities to be applied to different 3D locations. This allows for sufficient modification of the target region while preserving the original state of non-target regions as much as possible, reducing the possibility of structural drift. Based on this, a second-stage neural network is used to generate local details on the target sparse support set in conjunction with the text instruction, resulting in an editable mesh. This achieves phased decoupling editing of geometric structure and local details, maintaining consistency between the non-editable region and the mesh to be edited while accurately responding to text instructions, thus improving the accuracy, local consistency, and ease of use of text-driven 3D mesh editing.

[0042] During implementation, a control group input is constructed based on text instructions and empty text, such as... Figure 2 As shown, it may include the following: S201, based on the structure encoder, extracts the first structural latent variables from the mesh to be edited.

[0043] A structural encoder is an encoder in a two-stage 3D generation framework that works in conjunction with the first-stage neural network. It is used to map the voxelized structure of the mesh to be edited to a low-dimensional structural latent space.

[0044] The first structural latent variable refers to the latent variable obtained in the structural latent space after the mesh to be edited is encoded by the structural encoder, and is used to characterize the overall geometric structure of the mesh to be edited.

[0045] During implementation, the mesh to be edited can be voxelized first, retaining those voxels that intersect with the mesh surface, and marking whether these voxels are occupied with 0 / 1, thus forming a three-dimensional binary voxel mesh. Then, the three-dimensional binary occupied mesh is input into the structure encoder to obtain the first structure latent variable.

[0046] S202, densely encode the first structural latent variables and convert them into first source conditional features that match the conditional channels of the first-stage neural network through an adapter.

[0047] Dense coding refers to encoding each spatial location in the first structural latent variable separately to obtain a position-by-position feature representation.

[0048] An adapter is a lightweight network module used to convert densely encoded results into inputs suitable for the first stage of a neural network.

[0049] In implementation, after the structural encoder converts the 3D binary occupied grid of the mesh to be edited into the first structural latent variable, the first structural latent variable is densely encoded to obtain dense features. Then, the dense features are input into the adapter for feature mapping, converting them into the first source conditional features that the first-stage neural network can process.

[0050] S203, add noise to the first structural latent variable to obtain the noisy structural latent variable.

[0051] During implementation, noisy structural latent variables can be generated based on expression (1): (1) In expression (1), This represents the obtained noisy structural latent variables; Indicates the first structural latent variable; Indicates the time step parameter. ; This represents the preset minimum noise standard deviation; This represents standard Gaussian noise, which follows a normal distribution with a mean of 0 and a covariance of the identity matrix, and represents random noise.

[0052] S204. Construct the first input based on the first source condition features, noisy structural latent variables, and text instructions.

[0053] In practice, the text instruction can be input into the text encoder first to obtain the text conditions corresponding to the text instruction. Then, the first source condition features, the noisy structural latent variables, and the text conditions corresponding to the text instruction are combined together as the first input.

[0054] S205. Based on the first source conditional features, noisy structural latent variables, and empty text, construct the second input; the control group input includes the first input and the second input.

[0055] In practice, the empty text can first be input into the same text encoder as in step S204 to obtain the text conditions corresponding to the empty text. Then, the first source condition features that are the same as the first input, the same noisy structural latent variables, and the text conditions corresponding to the empty text are combined together as the second input.

[0056] In this embodiment, a structural encoder maps the structure of the mesh to be edited to the structural latent space to obtain a first structural latent variable. This first structural latent variable is then densely encoded, ensuring that the structural information of the mesh at each spatial location is preserved and corresponds positionally with the noisy structural latent variable. An adapter then converts this information into a first source conditional feature that matches the conditional channel of the first-stage neural network. This allows the first-stage neural network to reference the original structure of the mesh at each location without altering its original network structure, providing a positionally source structural basis for distinguishing between the edited and preserved regions. Simultaneously, ensuring that the only variable between the two sets of inputs is the text condition ensures that the output difference of the first-stage neural network for the two sets of inputs can only be caused by text instructions. This accurately separates the effect of text instructions from factors such as the structure of the mesh itself and the randomness of noise, thereby improving the accuracy of the generated text structural response field.

[0057] During implementation, the control group input is processed based on the first-stage neural network to obtain the speed difference between the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text, such as... Figure 3 As shown, it may include the following: S301, based on the first stage neural network processing of the first input, obtains the first velocity prediction.

[0058] The first velocity prediction refers to the velocity prediction of the noisy structural latent variable output by the first-stage neural network based on the first input, namely, the first source conditional features and the text conditions corresponding to the text instructions. The first velocity prediction has the same shape as the noisy structural latent variable and is used to represent the direction and magnitude of the evolution of the noisy structural latent variable at each spatial position in the structural latent space when editing according to the text instructions.

[0059] During implementation, the first velocity prediction can be generated based on expression (2): (2) In expression (2), Indicates the first velocity prediction; Represents noisy structural latent variables; Indicates a time step; Indicates text instructions; Indicates the first source condition feature; This represents the first stage of the neural network.

[0060] S302, based on the first-stage neural network processing of the second input, obtains the second velocity prediction.

[0061] The second velocity prediction refers to the velocity prediction of the noisy structural latent variable output by the first-stage neural network based on the second input, namely the text conditions corresponding to the first source conditional features and the empty text. The second velocity prediction has the same shape as the noisy structural latent variable and is used to represent the direction and magnitude of the evolution of the noisy structural latent variable at each spatial position in the structural latent space without providing editing semantics.

[0062] During implementation, a second velocity prediction can be generated based on expression (3): (3) In expression (3), This indicates the second speed prediction; Represents noisy structural latent variables; Indicates a time step; Indicates empty text; Indicates the first source condition feature; This represents the first stage of the neural network.

[0063] S303, determine the difference between the first speed prediction and the second speed prediction to obtain the speed difference.

[0064] The velocity difference refers to the difference between the first velocity prediction and the second velocity prediction at each spatial location and in each channel.

[0065] In practice, the speed difference can be obtained by subtracting the second speed prediction from the first speed prediction element by element.

[0066] In this embodiment, the same first-stage neural network is used to process the first and second inputs, which differ only in text conditions, to obtain the first velocity prediction and the second velocity prediction. The difference between the two is then determined as the velocity difference. By separating the influence of the text instruction on the generation process from other factors such as the structure of the grid to be edited and the randomness of noise, the velocity difference can reflect the degree of change in the evolution direction of the text instruction at each spatial location in the structural latent space. This allows the first-stage neural network to directly locate the editing area corresponding to the text instruction, improving the accuracy of editing.

[0067] During implementation, a text structure response field is generated based on the speed difference, such as... Figure 4 As shown, it may include the following: S401, norm aggregation is performed on the characteristic channels of velocity differences at each spatial location to obtain the initial response field.

[0068] Characteristic channels refer to the various dimensions contained in structural latent variables at each spatial location.

[0069] Norm aggregation refers to the operation of merging the difference values ​​of multiple feature channels at each spatial location into a non-negative scalar, which is used to measure the overall size of the difference vector at that location.

[0070] The initial response field is a scalar field defined at various spatial locations in the structural latent space, obtained by norm aggregation of velocity differences. The value at each location in the initial response field is called the initial response value at that location. The larger the initial response value, the stronger the influence of the text command on that location.

[0071] During implementation, the initial response field can be generated based on expression (4): (4) In expression (4), Indicates position The L2 norm of the velocity difference, i.e., the initial response field at position. The value of ; Indicates position The corresponding first velocity prediction; Indicates position The corresponding second velocity prediction; This indicates the calculation of the L2 norm of two vectors; Indicates the first velocity prediction in the 1st... Values ​​on each feature channel; The second speed prediction is in the first... Values ​​on each feature channel; For feature channel index; This indicates the total number of feature channels.

[0072] S402, Spatial smoothing and normalization are performed on the initial response field to obtain the text structure response field.

[0073] Spatial smoothing refers to correcting the initial response value of each spatial location by using the initial response values ​​in its surrounding neighborhood, making the changes in response values ​​between adjacent locations more continuous. Normalization, on the other hand, refers to transforming response values ​​to a uniform numerical range.

[0074] During implementation, a text structure response field can be generated based on expression (5): (5) In expression (5), Indicates all positions The initial response field is formed; Indicates a space smoothing operation; This represents the smoothed initial response field; This represents the final text structure response field; Indicates the text structure response field at position The larger the value, the greater the degree of change the text instruction makes to that position; This represents a normalization operation that maps values ​​to the range of 0 to 1.

[0075] The spatial smoothing operation can be expressed by expression (6): (6) In expression (6), Indicates the smoothed position The response value, i.e., the initial response value within the neighborhood. Weighted summation; Indicated by position A set of spatial neighborhoods centered on the location; Indicates the index of adjacent positions within the neighborhood; Indicates position Position The spatial smoothing weights must be non-negative and the sum of the neighborhood weights must be 1.

[0076] In practice, after the above spatial smoothing and normalization processes, the text structure response field can be obtained. In the text structure response field, the value at each position ranges from 0 to 1. The closer the value is to 1, the greater the degree to which the text instruction changes that position. The closer the value is to 0, the weaker the relationship between that position and the text instruction; the original structure of the grid to be edited should be preserved as much as possible.

[0077] In this embodiment, by performing norm aggregation on the feature channels of velocity differences at each spatial location, the difference vector of multiple channels at each location is converted into a non-negative scalar that only reflects the magnitude of the difference. This accurately measures the intensity of the change caused by the text command at each location, resulting in an initial response field that intuitively represents the distribution of the editing location. Further spatial smoothing of the initial response field suppresses isolated high or low response points caused by network prediction fluctuations, making the high response region spatially continuous and with smooth boundaries, which better matches the characteristics of a continuous spatial distribution of a 3D mesh. This reduces the possibility of scattered structural changes or holes within the editing region during subsequent generation. The smoothing result is then normalized to unify the response fields obtained from different text commands and different editable meshes to the same numerical range. This facilitates the use of uniform parameters in the subsequent generation process, improving the accuracy, stability, and versatility of locating the editing region using the text structure response field.

[0078] In implementation, the text structural response field is applied in a spatially relevant manner to the generation process from structural latent variables to the target sparse support set in the first-stage neural network, resulting in the target sparse support set, which can be obtained as follows: Figure 5 As shown, it includes: S501 converts the text structure response field into position-dependent source-preservation weights for the editable grid; a higher source-preservation weight indicates a smaller change in the corresponding position due to the text instruction.

[0079] Source preservation weights are weights defined at each spatial location in the structural latent space to control the degree to which the original structure of the mesh to be edited is preserved at that location during the generation process.

[0080] In practice, the text structure response field can be computed once before the first-stage neural network begins sampling and remains fixed throughout the subsequent sampling of the grid to be edited, without needing to be computed repeatedly at each time step. Subsequently, the text structure response field is converted into position-dependent source-preserving weights.

[0081] During implementation, the source-preserving weights can be generated using expression (7): (7) In expression (7), Indicates position The source maintains its weight; Indicates the text structure response field at position The value of ; This represents the hyperparameter used to adjust the overall source retention strength. A higher source retention weight indicates that the location is closer to the original structure of the mesh to be edited, and the text command changes the location less. A lower source retention weight indicates that the location is generated more according to the text command.

[0082] S502, perform the following operations at each time step of the first-stage neural network (including S5021-S5024) until the current state obtained at the nth time step is reached, which is used as the latent variable of the target structure: The current state refers to the value of the structural latent variable at a certain time step in the generation process.

[0083] The target structural latent variable refers to the current state obtained at the nth time step, which is used to represent the structural features of the edit mesh in the structural latent space.

[0084] S5021, based on the first-stage neural network processing of the current state, time step parameters, first source condition features, and text instructions, obtains the editing speed.

[0085] Editing speed refers to the speed at which the first-stage neural network predicts the current state under the constraints of text instructions. It represents the direction and speed at which the current state should evolve when editing according to the text instructions.

[0086] During implementation, the editing speed can be generated based on expression (8): (8) In expression (8), Indicates editing speed; Indicates the current state; Indicates the time step parameter; Indicates text instructions; Indicates the first source condition feature; This represents the first stage of the neural network.

[0087] S5022 generates a source reference velocity based on the current state and the first structural latent variable of the mesh to be edited.

[0088] The source reference velocity is the velocity required to return the current state to the first structural latent variable of the mesh to be edited along the generation path. It represents the direction in which the current state should evolve when the mesh to be edited is not being edited.

[0089] The source reference velocity can be directly obtained from the generation path adopted by the first-stage neural network without the need for neural network calculation.

[0090] In implementation, the generation path between the structural latent variables and Gaussian noise is first defined, which can be expressed by expression (9): (9) In expression (9), The time parameter is The path state at that time, i.e., the current state; This represents the time step parameters (as described above, a total of n time steps), during the generation process. Proceeding from 1 to 0; Represents the structural latent variables without noise; Indicates Gaussian noise; Indicates the noise figure; The preset minimum noise standard deviation; At that time, the path state is noise. ;exist At that time, the path status is .

[0091] When constructing the source reference velocity, the first structural latent variable of the mesh to be edited is used as the recovery reference, that is, let the expression (9) Take the first structural latent variable On this path, the path state is related to the time parameter. Differentiating, we obtain the velocity along the path, which can be expressed by expression (10): (10) In expression (10), The derivative of the path state with respect to the time parameter represents the speed of movement along the path; the meanings of the other symbols are the same as in expression (9).

[0092] Because expression (10) contains noise However, the specific noise mixed into the current state during the actual generation process is unknown, and expression (10) cannot be directly calculated. Therefore, the noise is deduced from expression (9) using the current state and the first structure latent variable, and can be expressed by expression (11): (11) Substituting expression (11) into expression (10), eliminating the noise term and rearranging, we obtain the source reference velocity, which can be expressed by expression (12): (12) In expression (12), The source reference velocity is indicated; the meanings of the other symbols are the same as those in expression (9).

[0093] In this embodiment, the source reference velocity can be directly calculated analytically from the current state and the first structural latent variable, and is re-determined at each time step based on the current state. This does not increase the computational overhead of the neural network, and ensures that the holding region always returns to the grid to be edited from its actual state, so that the editing region and the holding region maintain a high degree of consistency with the text instructions during the generation process.

[0094] S5023, at each spatial location, the editing speed and the source reference speed are weighted and fused based on the source preservation weight to obtain the modulation speed.

[0095] Modulation speed refers to the speed at which text editing and source structure preservation are balanced at various spatial locations, and is used to actually update the current state.

[0096] In practice, the modulation speed can be generated using expression (13): (13) In expression (13), Indicates position ,aisle Modulation speed; Indicates position The source maintains its weight; Indicates the location ,aisle Editing speed; Indicates position ,aisle The source reference speed.

[0097] Specifically, when the source maintains a weight close to 1, the modulation speed is primarily determined by the source reference speed, and this position will stably return to the original structure of the mesh to be edited. When the source maintains a weight close to 0, the modulation speed is primarily determined by the editing speed, and this position will generate a new structure according to the text instructions.

[0098] S5024, update the current state based on the modulation speed to obtain the current state of the next time step.

[0099] During implementation, the current state for the next time step can be generated using expression (14): (14) In expression (14), Indicates the first The state of a step, that is, the current state of the next time step; Indicates the first The state of a step, that is, the current state; Indicates the modulation speed.

[0100] During implementation, the time step parameter gradually decreases during the generation process. Less than The difference between the two is negative, therefore the current state is updated in the opposite direction of the modulation speed, that is, it evolves from noise towards the structural latent variable. The updated state is then... As the current state for the next time step, repeat steps S5021 to S5024 until the nth time step is reached to obtain the latent variables of the target structure.

[0101] It should be noted that during the entire generation process of the first-stage neural network, the first structural latent variable, the first source conditional feature, the text structural response field, and the source hold weights remain unchanged. However, the current state, editing speed, source reference speed, and modulation speed are recalculated at each time step as the current state changes.

[0102] S503 processes the latent variables of the target structure based on the structure decoder to obtain the target occupancy structure prediction result.

[0103] The structure decoder is a decoder that works in conjunction with the aforementioned structure encoder. It is used to restore the structure latent variables in the structure latent space to the occupancy prediction with the same resolution as the occupancy grid, that is, to predict whether each voxel is occupied by the grid.

[0104] The target occupancy structure prediction result refers to the occupancy prediction of the edited mesh at each voxel position obtained after the structure decoder decodes the latent variables of the target structure.

[0105] During implementation, the target occupancy structure prediction result can be generated through expression (15): (15) In expression (15), This indicates the target's occupancy structure prediction results; Represent the latent variables of the target structure; This indicates a structure decoder.

[0106] S504: Extract the effective structure from the target occupancy structure prediction results to obtain the target sparse support set.

[0107] Effective structure refers to the voxels in the target-occupied structure prediction results that are determined to be occupied by the edited mesh, such as non-empty voxels.

[0108] During implementation, the three-dimensional coordinates of valid positions can be extracted according to the occupancy determination rules of the structure decoder to form a target sparse support set. This support set is generated by the target structure prediction and can be different from the occupancy position set of the source mesh, thus representing a change in at least one of the component set dimension, shape attribute dimension, and spatial configuration dimension.

[0109] In this embodiment, the text structure response field is converted into position-dependent source-preservation weights. At each time step of the first-stage neural network, the editing speed representing the text instruction editing direction and the source reference speed representing the original structure of the mesh to be edited are weighted and fused at each spatial location based on the source-preservation weights. This ensures that locations with significant changes in the text instruction primarily generate new structures according to the editing speed, while locations with minor changes primarily maintain the original structure according to the source reference speed. This achieves full editing of the edited area and consistency between the edited and non-edited areas during the same generation process, reducing the problem of unrelated areas being accidentally altered or insufficient editing of the edited area caused by applying a uniform operation to the entire structural latent space. It also ensures a smooth transition between the edited and preserved areas. Based on the structure decoder, the target structural latent variables are decoded and effective structures are extracted to obtain a sparse support set of the target, providing accurate geometric structural basis for the second-stage neural network to generate the edited mesh. This reduces the storage and computational overhead of subsequent processing, thereby improving the accuracy and local consistency of text-driven 3D mesh editing.

[0110] During implementation, a second-stage neural network is used to process the target sparse support set and text instructions to obtain the editing mesh, such as... Figure 6 As shown, it may include the following: S601 initializes each position of the target sparse support set to obtain the initialization noise corresponding to each position.

[0111] The target sparse support set is the set of three-dimensional coordinates of the effective voxels occupied by the edited mesh, output from the first stage. Unlike the first stage, the second stage does not change the positions of the effective voxels, but fixes each position in the target sparse support set, and only generates and updates the appearance features attached to each position.

[0112] Initialization noise refers to the initial local latent features assigned to each location when the second-stage neural network begins to generate.

[0113] For example, using This represents the flow time of the second stage, with a value between 0 and 1. The generation process of the second stage follows... The process proceeds from 1 to 0. The sampling time series for the second stage is pre-set as follows: , where L is the number of sampling steps in the second stage. The time parameter is The shape of the noise-containing features at each location is as follows: ,in The number of feature channels at each valid location. Following the initial noise distribution of the second-stage neural network, such as a standard Gaussian distribution, one feature channel is sampled at each location of the target sparse support set. Gaussian noise in dimension 1, and Gaussian noise at all locations together constitute the initialization noise. .

[0114] S602, the second source condition features, the target sparse support set, the initialization noise corresponding to each position, and the text instructions are input into the second-stage neural network so that the second-stage neural network iteratively updates the initialization noise corresponding to each position to obtain the target structured latent features.

[0115] The second source conditional feature refers to the conditional feature obtained from the mesh to be edited and used to characterize the high-level semantics of the mesh to be edited in the second-stage neural network.

[0116] The target structured latent features refer to the local latent features at each position of the target sparse support set after the second-stage neural network is generated, which are used to describe the local details of the edit mesh within each voxel.

[0117] During implementation, the initialization noise obtained from S601 can be used first. As the initial current feature. In the first... Step, to the current feature Time parameters Target sparse support set Text conditions corresponding to text instructions and characteristics of the second source conditions The input to the second-stage neural network yields a feature velocity with the same shape as the current feature. .

[0118] In practice, this characteristic velocity can be generated using expression (16): (16) In expression (16), Indicates the first The characteristic velocity of the step; Indicates the first The current characteristics of the step; Indicates the first The time parameter corresponding to the step; This represents the sparse support set of the target, used to provide the voxel coordinates of each feature; This indicates the text conditions corresponding to the text instruction; Indicates the second source condition feature; This represents the second stage of the neural network. Its network parameters.

[0119] Then update the current feature based on the feature velocity. For example, the Euler method can be used for updating, which can be expressed by expression (17): (17) In expression (17), Indicates the first The current characteristics of the step; Indicates the first The time parameter corresponding to the step; the meanings of the other symbols are the same as those in expression (16).

[0120] right Repeat the feature velocity prediction and update operation described above. Throughout the second stage, the target sparse support set, textual conditions, and second source conditional features remain unchanged, while the current features and feature velocities are updated progressively. When the iteration ends, the final result is obtained. This refers to the target's structured latent features.

[0121] It should be noted that the second stage no longer applies the weighted fusion operation of the editing speed and the source reference speed based on the source preservation weight in the first stage. This is because the structural division between the editing region and the preservation region has been determined by the first stage through the target sparse support set. The second stage only needs to generate the corresponding local detail features at the given location.

[0122] S603, based on the grid decoder, processes the target's structured latent features and the target's sparse support set to obtain the edited grid.

[0123] A mesh decoder is a tool used to decode a mesh based on the locations of sparse voxels and their structured latent features.

[0124] During implementation, the target structured latent features and their corresponding target sparse support sets are input into the mesh decoder to obtain the edited mesh, which can be expressed by expression (18): (18) In expression (18), This represents the resulting edited grid; Represents the sparse support set of the target; This represents the target's structured latent features, which is the final result obtained above. ; This indicates a grid decoder.

[0125] In this embodiment, by initializing noise at various locations of the target sparse support set and iteratively updating the features at each location with these locations fixed, the overall structure of the editable mesh is determined by the first stage, while local details are generated by the second stage. This decouples the generation of structure and details, simplifying the generation task of the second stage. Simultaneously, the second source conditional features provide local detail information of the mesh to be edited, and the text instructions provide editing semantics. Both jointly constrain feature generation, ensuring that the local details of the preserved region remain consistent with the mesh to be edited, the edited region conforms to the requirements of the text instructions, and the two are naturally connected. Furthermore, the second-stage neural network and mesh decoder are computed only on effective voxels, enabling the generation of a higher-resolution editable mesh with lower computational overhead, improving the detail quality and processing efficiency of the editable mesh.

[0126] When iteratively updating the initial noise at each position, the second-stage neural network needs to obtain local detail information of the mesh to be edited based on the second source conditional features to ensure that the local details of the preserved regions in the edited mesh are consistent with those of the mesh to be edited. In implementation, obtaining the second source conditional features can be achieved based on the following steps: Step A1: Render the mesh to be edited from multiple perspectives to obtain multiple rendered images; Multi-view rendering refers to placing multiple virtual cameras around the mesh to be edited, and imaging the mesh from different perspectives. The rendered image is the two-dimensional image captured by each virtual camera.

[0127] Step A2: Extract image features from multiple rendered images based on a visual encoder; A visual encoder is a pre-trained neural network used to extract semantic and geometric features from two-dimensional images. For example, a visual encoder can be a self-supervised pre-trained model based on the visual Transformer, such as DINOv2 (DIstillation with NO labels version 2).

[0128] Image features refer to the two-dimensional feature map obtained by the visual encoder after extracting features from each rendered image, where each pixel position corresponds to a feature vector.

[0129] During implementation, each rendered image is input into a visual encoder to obtain the image features of each rendered image.

[0130] Step A3: Fuse the image features corresponding to the effective voxel positions of the mesh to be edited to obtain the aggregated features of each effective voxel position; Effective voxel positions refer to the voxels that are occupied after the mesh to be edited is voxelized.

[0131] Aggregated features refer to the feature vector obtained by fusing the image features corresponding to each effective voxel position in multiple rendered images. It is used to comprehensively describe the local features presented by the voxel position under various viewpoints.

[0132] In practice, for each effective voxel location, the voxel center can be projected onto each rendered image using the intrinsic and extrinsic parameters of each virtual camera. The image features of that location under each viewpoint are then sampled from the corresponding image features. After removing occluded viewpoints by combining the depth map, the image features under each visible viewpoint are fused to obtain the aggregated features of that effective voxel location.

[0133] Step A4: Input the aggregated features into the sparse encoder to obtain the source appearance latent features corresponding to the mesh to be edited; A sparse encoder is an encoder used in conjunction with a mesh decoder to map aggregated features at effective voxel locations to structured latent features. For example, a sparse encoder can be a Slat (Structured LATent) encoder.

[0134] Source appearance latent features refer to the structured latent features of the mesh to be edited at each effective voxel location, used to characterize the local geometric details of the mesh to be edited.

[0135] During implementation, the aggregated features and coordinates of each effective voxel position can be input into the sparse encoder to obtain the source appearance latent features.

[0136] Step A5: Input the latent features of the source appearance into the second-stage neural network, and read the hidden features from the preset intermediate layer of the second-stage neural network as the second source conditional features.

[0137] Pre-defined intermediate layers refer to one or more pre-specified network layers in the second-stage neural network.

[0138] Hidden features refer to the features output by the second-stage neural network in a preset intermediate layer, which are in the same representation space as the features inside the second-stage neural network.

[0139] In implementation, the latent features of the source appearance can be input into the second-stage neural network, and the hidden features can be read in the preset intermediate layer. If the second-stage neural network is built based on a transformer, there are multiple recurrent blocks in the transformer. At least one of these recurrent blocks can be selected as the preset intermediate layer to obtain the features output by each preset intermediate layer as the hidden features.

[0140] In this embodiment, by performing multi-view rendering and extracting image features from the mesh to be edited, and then fusing the image features of each effective voxel position from multiple views, the aggregated features can comprehensively reflect the local information at that position, avoiding the influence of single-view occlusion or information loss. A sparse encoder maps the aggregated features to source appearance latent features, so that the local features of the mesh to be edited are represented in the same latent space as the second-stage neural network, and are calculated at the effective voxel positions, reducing computational overhead. Then, hidden features are read from a preset intermediate layer of the second-stage neural network as second source conditional features. Since these hidden features are in the same representation space as the internal features of the corresponding layer, they can be directly injected into the second-stage neural network without additional training adapters. This allows the second-stage neural network to effectively utilize the local details of the mesh to be edited, ensuring that the local details of the preserved region in the edited mesh remain consistent with those of the mesh to be edited, and that the edited region and the preserved region are naturally connected in local geometry, thereby improving the detail quality of the edited mesh.

[0141] During implementation, before constructing a control group input based on text instructions and empty text, the first-stage neural network and the second-stage neural network in the two-stage 3D generation framework can be trained to enable them to edit according to text instructions under the constraints of the original source mesh structure. This training process may include the following steps: Step B1: Based on the first source condition features of the source grid and the first text sample instruction of the first-stage neural network processing source grid, the sparse support set to be processed is obtained; Training samples may include a triple consisting of a source grid, a first text sample instruction, and a target grid. The source grid refers to the grid in the training samples before editing; the target grid is obtained by modifying the source grid using the first text sample instruction.

[0142] The first text sample instruction refers to the natural language text in the training samples that describes what kind of editing to be performed on the source grid.

[0143] The sparse support set to be processed refers to the set of three-dimensional coordinates of effective voxels generated during the training process of the first-stage neural network.

[0144] Step B2: Based on the second-stage neural network processing of the sparse support set to be processed, the first text sample instruction, and the second source condition features of the source grid, the edit grid is obtained; Step B3: Optimize the first-stage neural network and the second-stage neural network based on the target grid.

[0145] In practice, the first-stage neural network and the second-stage neural network can be optimized based on the first loss between the editing speed output by the first-stage neural network and the first target speed, and the second loss between the predicted speed output by the second-stage neural network and the second target speed.

[0146] In this embodiment of the disclosure, before constructing the control group input, an editing grid is generated using a first-stage neural network and a second-stage neural network based on the first source condition features, the second source condition features, and the first text sample instruction of the source grid. The target grid is used as the ground value to optimize the two-stage neural network. This enables the speed prediction given by the trained first-stage neural network under both text conditions and empty text conditions to accurately reflect the editing intent of the text instruction, laying the foundation for accurately locating the editing area through speed differences in the subsequent editing stage.

[0147] Based on the same concept, this disclosure also provides a model training method for training the first-stage neural network and the second-stage neural network in the aforementioned two-stage 3D generation framework. For example... Figure 7 As shown, the model training method may include the following: S701, construct a control group input based on text sample instructions and empty text; in the control group input, the first source condition features of the source grid are the same and the noisy structural latent variables are the same, but the text conditions are different.

[0148] S702, based on the first-stage neural network processing of the control group input, obtains the speed difference of the first-stage neural network under the text conditions corresponding to the text sample instruction and the text conditions corresponding to the empty text.

[0149] S703 generates a text structure response field based on velocity differences; the text structure response field is used to represent the degree of change of different positions in the structural latent space by the text sample instruction.

[0150] S704, the text structure response field is applied to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner, to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes a first-stage neural network and a second-stage neural network.

[0151] S705 utilizes a second-stage neural network to process the target sparse support set and text sample instructions to obtain an editable mesh.

[0152] In implementation, a second-stage neural network can be used to process the target sparse support set and text sample instructions to obtain the prediction speed. Based on the prediction speed, the features at each position of the target sparse support set are iteratively updated to obtain the target structured latent features. Then, a grid decoder is used to process the target structured latent features and the target sparse support set to obtain the edit grid.

[0153] S706 optimizes the first-stage neural network and the second-stage neural network based on the target grid.

[0154] In implementation, the first-stage neural network and the second-stage neural network can be optimized based on the first loss between the editing speed output by the first-stage neural network and the first target speed, and the second loss between the predicted speed output by the second-stage neural network and the second target speed. The first target speed and the second target speed are generated based on the target grid; the target grid is the ground truth grid obtained by editing the source grid using text sample instructions.

[0155] When training the flow model, a target velocity needs to be provided for the model to learn, i.e., a reference value that the model's prediction should approach. This can be achieved using known real target meshes and random noise. First, the mixing method between the two is defined, then the rate of change of the mixing result with time parameters is calculated, and this rate of change is used as the target velocity. In implementation, the first loss mentioned in the embodiments of this disclosure can be obtained based on the following steps: The target mesh can first be voxelized into an occupied mesh, and then mapped to the structural latent space through a structural encoder to obtain structural latent variable reference information, denoted as . Regenerate Gaussian noise with the same shape as the structural latent variable reference information. The number and arrangement of elements in Gaussian noise are both similar to Same. And select the time parameter. This is used to control the degree of mixing between structural latent variable reference information and noise, and to obtain noisy structural latent variable reference information, which can be expressed by expression (19): (19) In expression (19), This indicates that noisy structural latent variable reference information has been obtained; Indicates reference information for structural latent variables; Indicates Gaussian noise; Represents the preset minimum noise figure; where, when hour, ;when hour, Therefore, change This allows us to obtain training inputs with varying degrees of mixing.

[0156] For the selected structural latent variable reference information and Gaussian noise, fix , and As shown in expression (20), expression (19) can be rearranged as follows: (20) In expression (20), Not following Change, equal Multiply by a fixed array of values. Therefore, For each small increase, The change is The array represents the exact value of the rate of change. It can be used to determine the first target velocity during training, which can be expressed by expression (21): (twenty one) In expression (21), This indicates the speed of the first target.

[0157] Here, "first target velocity" represents the rate of change of the latent variable values ​​in the structural latent variable reference information with respect to time parameters. During calculation, each element in the array is multiplied by the noise value (…). The operation of subtracting the latent variable value yields the complete first target velocity array, eliminating the need for another model prediction.

[0158] For example, considering only one element in the array, take the latent variable value. Noise value and order ,but The target speed is .when from Increase to hour, from Reduce to The rate of change The negative sign indicates that the value of the element changes with... Increases and decreases. The actual latent variable contains multiple elements, each of which is calculated in the same way.

[0159] During implementation, the noisy structural latent variables of the source grid, the first source condition features of the source grid, and the text conditions corresponding to the text sample instructions are input into the first-stage neural network to obtain the editing speed.

[0160] Subtract the editing speed from the first target speed element by element, square the differences and take the average to obtain the first loss, which can be expressed by expression (22): (twenty two) In expression (22), Indicates the first loss. This represents the total number of elements involved in the first loss calculation; Indicates the element number; The first number representing editing speed One element; The first target velocity is represented by the th Each element.

[0161] Similarly, the second target velocity can be generated using the same method as the first target velocity. The generation process for the second target velocity is the same as that for the first target velocity. The difference is that the first target velocity is generated using structural latent features, while the second target velocity is generated using the appearance latent features of the target mesh. The generation method for the appearance latent features is as described in steps A1-A4 above, and will not be repeated here. Furthermore, the Gaussian noise required for calculating the second target velocity, and the time step... , and Determined independently.

[0162] In implementation, the second target loss is calculated using the prediction speed and the second target speed of the second-stage neural network iterative update. The first and second losses are then combined to optimize the first and second-stage neural networks.

[0163] In this embodiment, by executing a processing flow consistent with actual editing during training—namely, generating a text structure response field based on a control group input with only different text conditions—and applying it to the first-stage generation process in a spatially relevant manner, followed by the second-stage neural network generating the editing mesh, the two-stage neural networks directly adapt to the editing method based on the text structure response field during training, thus reducing the difference between training and inference. Furthermore, the two-stage neural networks are optimized using the target mesh as the ground truth. This enables the first-stage neural network to learn to generate the correct target structure according to the text sample instructions under the constraints of the original source mesh structure, and to concentrate speed differences more on the areas that truly need editing, improving the accuracy of the text structure response field in locating the editing area. Simultaneously, the second-stage neural network learns to generate correct local geometric details, thereby enhancing the accuracy and local consistency of text-driven 3D mesh editing.

[0164] In summary, the overall flowchart of the text-driven 3D mesh editing method provided in this embodiment is as follows: Figure 8 As shown, it includes: S801, extract the first source condition features of the mesh to be edited.

[0165] In practice, the grid to be edited can be voxelized into an occupied grid, and the occupied grid can be mapped to the structural latent space through a structural encoder to obtain the first structural latent variable. Then, the first structural latent variable is densely encoded and converted into the first source conditional feature that matches the conditional channel of the first stage neural network through an adapter.

[0166] S802, extract the second source condition features of the mesh to be edited.

[0167] The mesh to be edited is rendered from multiple perspectives to obtain multiple rendered images. Image features are extracted from the rendered images based on a visual encoder. The image features corresponding to each effective voxel position under multiple perspectives are projected and fused to obtain aggregated features, and the source appearance latent features are obtained through a sparse encoder. Then, the source appearance latent features, along with time parameters and empty text conditions, are input into the second-stage neural network, and hidden features are read from the preset intermediate layer as the second source conditional features.

[0168] S803, construct the control group input, and obtain the speed difference based on the first-stage neural network.

[0169] Noise is added to the first structural latent variable to obtain a noisy structural latent variable. The noisy structural latent variable and the first source conditional feature are combined with the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text, respectively, to obtain the first input and the second input. These two inputs are then fed into the first-stage neural network to obtain the first speed prediction and the second speed prediction, respectively. The difference between the two is the speed difference.

[0170] S804 generates a text structure response field and converts it into source-preserving weights.

[0171] Norm aggregation is performed on each feature channel representing the velocity difference at each spatial location, followed by spatial smoothing and normalization to obtain the text structure response field. This text structure response field is then converted into source preservation weights; the larger the value of the text structure response field at a given location, the smaller the source preservation weight at that location. The source preservation weights remain unchanged in subsequent generation processes.

[0172] S805, based on the first-stage neural network processing of the current state, time step parameters, first source condition features, and text conditions, obtains the editing speed.

[0173] S806 generates a source reference velocity based on the current state and the first structural latent variable, and then performs a weighted fusion of the editing velocity and the source reference velocity at each spatial location based on the source preservation weight to obtain the modulation velocity.

[0174] S807: Based on the modulation speed, the current state is updated through the numerical solver to obtain the current state for the next time step. If the nth time step has not been reached, return to S805; if the nth time step has been reached, use the obtained current state as the latent variable of the target structure and continue to execute S808.

[0175] S808 processes the latent variables of the target structure based on the structure decoder, obtains the target occupied structure prediction results, extracts the effective structure from them, and obtains the target sparse support set.

[0176] S809 initializes noise at each position of the target sparse support set as the current noisy feature.

[0177] In step S810, the current noisy feature is used as the initial current feature. The text instruction, time parameter, target sparse support set, second source conditional feature, and the initial current feature are input into the second-stage neural network to obtain the feature velocity. The current feature is updated based on the feature velocity. If step L has not been reached, S810 continues; if step L has been reached, the obtained current feature is used as the target structured latent feature, and step S811 continues.

[0178] S811, based on the grid decoder, processes the target's structured latent features and the target's sparse support set to obtain the edited grid.

[0179] In summary, the text-driven 3D mesh editing method provided in this embodiment utilizes a control group input with only different text conditions to directly obtain the region of action of the text instruction in the structural latent space from the speed difference of the first-stage neural network, without manually specifying the editing region or additionally training the segmentation model; then, by using source-preserving weights to fuse the editing speed and the source reference speed at each spatial location, the editing region is fully edited according to the text instruction, while the non-editing region retains the original structure of the mesh to be edited; finally, the second-stage neural network combines the second source condition features to generate local geometric details, resulting in an edited mesh that accurately responds to text instructions while maintaining local consistency.

[0180] Based on the same technical concept, this disclosure also provides a text-driven three-dimensional mesh editing device 900, such as... Figure 9 As shown, it includes: The first construction module 901 is used to construct a control group input based on text instructions and empty text; in the control group input, the first source condition features of the grid to be edited are the same and the noisy structural latent variables are the same, but the text conditions are different. The first processing module 902 is used to process the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text. The first generation module 903 is used to generate a text structure response field based on the speed difference; the text structure response field is used to represent the degree of change of the text instruction to different positions in the structural latent space; The first support set generation module 904 is used to apply the text structure response field to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner, so as to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes a first-stage neural network and a second-stage neural network. The second processing module 905 is used to process the target sparse support set and text instructions using the second-stage neural network to obtain the edit mesh.

[0181] The first building module is used for: Extracting the first structural latent variables from the mesh to be edited based on the structural encoder; The first structural latent variables are densely encoded and converted into first source conditional features that match the conditional channels of the first-stage neural network through an adapter; Adding noise to the first structural latent variable yields a noisy structural latent variable; The first input is constructed based on the first source conditional features, noisy structural latent variables, and text instructions. The second input is constructed based on the first source conditional features, noisy structural latent variables, and empty text; the control group input includes the first input and the second input.

[0182] The first processing module is used for: The first speed prediction is obtained by processing the first input based on the first-stage neural network. The second speed prediction is obtained by processing the second input based on the first-stage neural network. The difference between the first and second speed predictions is determined to obtain the speed difference.

[0183] The first generation module is used for: Norm aggregation is performed on the characteristic channels of velocity differences at each spatial location to obtain the initial response field; Spatial smoothing and normalization are performed on the initial response field to obtain the text structure response field.

[0184] The first support set generation module is used for: The text structure response field is converted into position-dependent source-preservation weights for the editable grid; a higher source-preservation weight indicates that the text instruction causes a smaller change to the corresponding position. In the first stage of the neural network, the following operations are performed at each time step until the current state obtained at the nth time step is reached, thus obtaining the latent variables of the target structure: The editing speed is obtained by processing the current state, time step parameters, first source condition features, and text instructions based on the first-stage neural network. Generate a source reference velocity based on the current state and the first structural latent variables of the mesh to be edited; At each spatial location, the editing speed and the source reference speed are weighted and fused based on the source preservation weight to obtain the modulation speed; The current state is updated based on the modulation speed to obtain the current state at the next time step; The target occupancy structure prediction result is obtained by processing the latent variables of the target structure using a structure decoder. The effective structure is extracted from the target occupancy structure prediction results to obtain the target sparse support set.

[0185] Based on the current state and the first structural latent variables of the mesh to be edited, a source reference velocity is generated, including: The source reference speed is generated based on the following expression: in, Indicates the source reference velocity; This is the preset minimum noise standard deviation; This is the current state; These are the first structural latent variables; Indicates time, during the generation process Proceeding from 1 to 0.

[0186] The second processing module is used for: Initialize each position of the target sparse support set to obtain the initialization noise corresponding to each position; The second source conditional features, the target sparse support set, the initialization noise corresponding to each position, and the text instructions are input into the second-stage neural network so that the second-stage neural network iteratively updates the initialization noise corresponding to each position to obtain the target structured latent features. An edited mesh is obtained by processing the target's structured latent features and sparse support set using a mesh decoder.

[0187] Also includes: The mesh to be edited is rendered from multiple perspectives to obtain multiple rendered images; Image features are extracted from multiple rendered images based on a visual encoder; The image features corresponding to the effective voxel positions of the grid to be edited are fused to obtain the aggregated features of each effective voxel position; The aggregated features are input into the sparse encoder to obtain the source appearance latent features corresponding to the mesh to be edited; The latent features of the source appearance are input into the second-stage neural network, and the hidden features are read from the preset intermediate layer of the second-stage neural network as the second source conditional features.

[0188] It also includes a training module, which performs the following operations before constructing a control group input based on text instructions and empty text: Based on the first source condition features and the first text sample instruction of the source grid processed by the first-stage neural network, a sparse support set to be processed is obtained. Based on the second-stage neural network processing of the sparse support set to be processed, the first text sample instruction, and the second source condition features of the source grid, the edit grid is obtained. Based on the target grid, optimize the first-stage neural network and the second-stage neural network.

[0189] Based on the same technical concept, this disclosure also provides a model training device 1000, such as... Figure 10 As shown, it includes: The second construction module 1001 is used to construct a control group input based on text sample instructions and empty text; in the control group input, the first source condition features of the source grid are the same and the noisy structural latent variables are the same, but the text conditions are different. The third processing module 1002 is used to process the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text sample instruction and the text conditions corresponding to the empty text. The second generation module 1003 is used to generate a text structure response field based on the speed difference; the text structure response field is used to represent the degree of change of different positions in the structural latent space by the text sample instruction; The second support set generation module 1004 is used to apply the text structure response field to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner, so as to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes a first-stage neural network and a second-stage neural network. The fourth processing module 1005 is used to process the target sparse support set and text sample instructions using the second-stage neural network to obtain the edit grid; Optimization module 1006 is used to optimize the first-stage neural network and the second-stage neural network based on the target grid.

[0190] Figure 11 This is a structural block diagram of an electronic device according to an embodiment of the present disclosure. Figure 11As shown, the electronic device includes a memory 1110 and a processor 1120. The memory 1110 stores a computer program that can run on the processor 1120. The number of memories 1110 and processors 1120 can be one or more. The memory 1110 can store one or more computer programs, which, when executed by the electronic device, cause the electronic device to perform the method provided in the above-described method embodiments. The electronic device may also include a communication interface 1130 for communicating with external devices and performing data exchange and transmission.

[0191] If the memory 1110, processor 1120, and communication interface 1130 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 11 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0192] Optionally, in a specific implementation, if the memory 1110, processor 1120 and communication interface 1130 are integrated on a single chip, the memory 1110, processor 1120 and communication interface 1130 can communicate with each other through an internal interface.

[0193] It should be understood that the aforementioned processor can be a Central Processing Unit (CPU), or other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Machines (ARM) architecture.

[0194] Further, optionally, the aforementioned memory may include read-only memory and random access memory, and may also include non-volatile random access memory. The memory may be volatile or non-volatile, or may include both. Non-volatile memory may include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may include random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), SyncLink DRAM (SLDRAM), and Direct RAMBUS RAM (DR RAM).

[0195] In the description of the embodiments of this disclosure, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this disclosure. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0196] In the description of the embodiments disclosed herein, unless otherwise stated, " / " means "or". For example, A / B can mean A or B. The "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0197] In the description of embodiments of this disclosure, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of this disclosure, unless otherwise stated, "a plurality of" means two or more.

[0198] The above description is merely an exemplary embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A text-driven 3D mesh editing method, characterized in that, include: A control group input was constructed based on text instructions and empty text. In the control group input, the first source condition features of the grid to be edited are the same, and the noisy structural latent variables are the same, but the text conditions are different; Based on the processing of the control group input by the first-stage neural network, the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text is obtained. A text structure response field is generated based on the speed difference; the text structure response field is used to represent the degree of change of the text instruction at different positions in the structural latent space; The text structure response field is applied to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; wherein, the two-stage 3D generation framework includes the first-stage neural network and the second-stage neural network. The second-stage neural network is used to process the target sparse support set and the text instructions to obtain an editable mesh.

2. The method according to claim 1, characterized in that, The input for constructing the control group based on text instructions and empty text includes: Extracting the first structural latent variables from the mesh to be edited based on the structural encoder; The first structural latent variable is densely encoded and converted into a first source conditional feature that matches the conditional channel of the first stage neural network through an adapter; Add noise to the first structural latent variable to obtain the noisy structural latent variable; Based on the first source condition features, the noisy structural latent variables, and the text instructions, a first input is constructed; A second input is constructed based on the first source conditional features, the noisy structural latent variables, and the empty text; the control group input includes the first input and the second input.

3. The method according to claim 2, characterized in that, The process of processing the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text includes: Based on the first stage neural network processing of the first input, a first speed prediction is obtained; Based on the first-stage neural network processing of the second input, a second speed prediction is obtained; The difference between the first speed prediction and the second speed prediction is determined to obtain the speed difference.

4. The method according to claim 1, characterized in that, The generation of the text structure response field based on the speed difference includes: Norm aggregation is performed on each characteristic channel of the velocity difference at each spatial location to obtain the initial response field; The initial response field is spatially smoothed and normalized to obtain the text structure response field.

5. The method according to any one of claims 1-4, characterized in that, The step of applying the text structure response field in a spatially relevant manner to the generation process from structural latent variables to the target sparse support set in the first-stage neural network to obtain the target sparse support set includes: The text structure response field is converted into a position-dependent source-preservation weight of the editable grid; the higher the source-preservation weight, the smaller the change of the text instruction to the corresponding position; In each time step of the first stage of the neural network, the following operations are performed until the current state obtained at the nth time step is reached, which is used as the latent variable of the target structure: The editing speed is obtained based on the current state, time step parameters, first source condition features, and text instructions processed by the first-stage neural network. Based on the current state and the first structural latent variable of the mesh to be edited, a source reference velocity is generated; At each spatial location, the editing speed and the source reference speed are weighted and fused based on the source retention weight to obtain the modulation speed; The current state is updated based on the modulation speed to obtain the current state at the next time step; Based on the processing of the latent variables of the target structure using the structure decoder, the target occupancy structure prediction result is obtained; The effective structure is extracted from the target occupancy structure prediction results to obtain the target sparse support set.

6. The method according to claim 5, characterized in that, The generation of the source reference velocity based on the current state and the first structural latent variable of the mesh to be edited includes: The source reference velocity is generated based on the following expression: in, Indicates the source reference velocity; This is the preset minimum noise standard deviation; This refers to the current state; For the first structural latent variable; Indicates time, during the generation process Proceeding from 1 to 0.

7. The method according to claim 1, characterized in that, The process of using the second-stage neural network to process the target sparse support set and the text instructions to obtain the edit mesh includes: Initialize each position of the target sparse support set to obtain the initialization noise corresponding to each position; The second source conditional features, the target sparse support set, the initialization noise corresponding to each position, and the text instruction are input into the second stage neural network, so that the second stage neural network iteratively updates the initialization noise corresponding to each position to obtain the target structured latent features. The edited mesh is obtained by processing the target structured latent features and the target sparse support set using a mesh decoder.

8. The method according to claim 7, characterized in that, Also includes: The mesh to be edited is rendered from multiple perspectives to obtain multiple rendered images; Image features are extracted from the multiple rendered images based on a visual encoder; The image features corresponding to the effective voxel positions of the grid to be edited are fused to obtain the aggregated features of each effective voxel position; The aggregated features are input into a sparse encoder to obtain the source appearance latent features corresponding to the mesh to be edited; The latent features of the source appearance are input into the second-stage neural network, and the hidden features are read from the preset intermediate layer of the second-stage neural network as the second source conditional features.

9. The method according to claim 1, characterized in that, Before constructing the control group input based on text instructions and empty text, the following is also included: Based on the first source condition features and the first text sample instruction of the source grid processed by the first-stage neural network, a sparse support set to be processed is obtained. The editing grid is obtained by processing the sparse support set to be processed, the first text sample instruction, and the second source condition feature of the source grid based on the second-stage neural network; Based on the target grid, optimize the first-stage neural network and the second-stage neural network.

10. A model training method, characterized in that, include: A control group input is constructed based on text sample instructions and empty text; in the control group input, the first source condition features of the source grid are the same and the noisy structural latent variables are the same, but the text conditions are different. Based on the processing of the control group input by the first-stage neural network, the speed difference of the first-stage neural network under the text conditions corresponding to the text sample instruction and the text conditions corresponding to the empty text is obtained. A text structure response field is generated based on the speed difference; the text structure response field is used to represent the degree of change of the text sample instruction to different positions in the structural latent space; The text structure response field is applied to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes the first-stage neural network and the second-stage neural network. The second-stage neural network is used to process the target sparse support set and the text sample instructions to obtain an editing grid; Based on the target grid, optimize the first-stage neural network and the second-stage neural network.

11. A text-driven three-dimensional mesh editing device, characterized in that, include: The first construction module is used to construct the control group input based on text instructions and empty text; In the control group input, the first source condition features of the grid to be edited are the same, and the noisy structural latent variables are the same, but the text conditions are different; The first processing module is used to process the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text instruction and the text conditions corresponding to the empty text. The first generation module is used to generate a text structure response field based on the speed difference; the text structure response field is used to represent the degree of change of the text instruction to different positions in the structural latent space; The first support set generation module is used to apply the text structure response field to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes the first-stage neural network and the second-stage neural network. The second processing module is used to process the target sparse support set and the text instructions using the second-stage neural network to obtain an editable mesh.

12. A model training device, characterized in that, include: The second construction module is used to construct a control group input based on text sample instructions and empty text; in the control group input, the first source condition features of the source grid are the same and the noisy structural latent variables are the same, but the text conditions are different. The third processing module is used to process the control group input based on the first-stage neural network to obtain the speed difference of the first-stage neural network under the text conditions corresponding to the text sample instruction and the text conditions corresponding to the empty text. The second generation module is used to generate a text structure response field based on the speed difference; the text structure response field is used to represent the degree of change of the text sample instruction to different positions in the structural latent space; The second support set generation module is used to apply the text structure response field to the generation process from structural latent variables to target sparse support set in the first-stage neural network in a spatially related manner, so as to obtain the target sparse support set; wherein, the two-stage three-dimensional generation framework includes the first-stage neural network and the second-stage neural network. The fourth processing module is used to process the target sparse support set and the text sample instructions using the second-stage neural network to obtain an edit grid; An optimization module is used to optimize the first-stage neural network and the second-stage neural network based on the target grid.

13. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; characterized in that, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the method of any one of claims 1-10.

14. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-10.

15. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method according to any one of claims 1-10.