A complex semantic-oriented progressive text-guided 3D model generation method and system

Abstract

The application provides a progressive text-guided three-dimensional model generation method and system for complex semantic hints, and belongs to the technical field of computer graphics and artificial intelligence generated content cross technology. A 3D semantic component positioning module based on zero-shot learning and a geometric perception double-path optimizer are used. A positioning strategy based on zero-shot learning is used in combination with a visual large model to automatically identify and position the to-be-edited area in the three-dimensional model. Then, the geometric perception double-path optimizer is used to drive the positioning area to evolve towards the target semantics while constraining the non-editing area to remain unchanged. Finally, a high-fidelity three-dimensional model is output. The application solves the technical problems that text-based three-dimensional technology is prone to semantic ambiguity when processing complex logical hints, and that catastrophic forgetting occurs in non-associated areas when editing locally.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of computer graphics and artificial intelligence content generation, and in particular relates to a progressive text-guided 3D model generation method and system for complex semantics. Background Technology

[0002] Currently, with the widespread application of various digital twin technologies, the demand for diverse and high-quality 3D models in scenarios such as smart colleges and smart factories is increasing. Text-to-3D (TTL) AI-generated content, as a key technology capable of efficiently converting text prompts into 3D content that fits real-world scenarios, can not only be applied to digital twin systems to quickly build virtual mappings highly consistent with real-world scenes, but is also suitable for other application scenarios with a strong demand for 3D assets, enabling seamless integration of generated content into various virtual environments.

[0003] To ensure the practical usability of generated models, it is particularly important to research highly consistent 3D content generation methods for complex semantic cues. However, existing text-guided 3D model generation technologies suffer from the following problems: traditional text-based 3D generation methods rely on the prior knowledge of 2D diffusion models, which easily lead to problems such as missing objects, attribute mismatches, and structural disorder when dealing with complex semantic cues involving multiple objects, attributes, and spatial relationships. This makes it difficult to meet the requirements of digital twin scenarios for rich detail and structural accuracy.

[0004] Existing progressive editing methods lack fine-grained semantic control over 3D models, making it difficult to accurately identify and locate the parts that need modification. This leads to a heavy reliance on manual intervention in the editing process, and the final edited results are often inaccurate, failing to achieve efficient and automated 3D content adaptation. Furthermore, most existing methods are limited to overall regeneration or parametric template editing, lacking fine-grained control based on semantic parts. When performing local, progressive adaptation of 3D assets, the global optimization characteristics can easily cause geometric drift or texture degradation in non-editable areas.

[0005] Based on the above analysis, the problems and shortcomings of the existing technology are as follows: (1) Traditional text-based 3D methods are prone to object missing, attribute mismatch and structural disorder when dealing with complex semantic prompts, resulting in poor semantic consistency. (2) Existing progressive generation methods lack semantic-level fine control, making it difficult to automatically and accurately locate the parts to be edited in the 3D model. (3) Existing methods cannot effectively maintain the geometric consistency of non-edited areas when performing local incremental generation, which can easily lead to background structure destruction or texture distortion. Summary of the Invention

[0006] To address the aforementioned issues, the first aspect of this invention proposes a progressive text-guided 3D model generation method for complex semantic prompts. This method employs a 3D semantic component localization module based on zero-shot learning and a geometry-aware dual-path optimizer. A zero-shot learning-based localization strategy combined with a large visual model is used to automatically identify and locate the editable regions in the 3D model. Then, the geometry-aware dual-path optimizer drives the localized regions to evolve towards the target semantic meaning while constraining the non-editable regions to remain unchanged, ultimately outputting a high-fidelity 3D model.

[0007] Preferably, the following specific processes are included: Step 1: Input source description text to generate an initial 3D scene representation; Step 2: Based on the initial 3D scene representation, and simultaneously inputting source description text and target description text, the 3D semantic component localization module based on zero-shot learning is used to identify the region to be edited, and generate a multi-view geometric consistency mask sequence and a 3D editing bounding box. Step 3: The geometry-aware dual-path optimizer receives the multi-view geometric consistency mask sequence and the 3D edit bounding box as input data, and decouples the optimization process of the 3D scene into joint processing of the edit path and the preservation path: In the editing path, a text-driven gradient with source semantic suppression mechanism is applied to the area to be edited indicated by the mask, and incremental semantic information is extracted for redrawing and generation; While preserving the path, geometric anchoring constraints and boundary-aware gradient smoothing constraints are applied to the non-editable areas indicated by the mask to maintain the geometric and photometric consistency of the original scene background. Finally, the 3D scene parameters are updated, and a high-fidelity 3D model with local incremental generation is output.

[0008] Preferably, the 3D semantic component localization module based on zero-shot learning specifically includes a sampling rendering unit, a proxy image generation and geometric restoration unit, an object detection and segmentation unit, and a multi-view fusion and aggregation unit; The sampling and rendering unit is used to perform multi-view discrete sampling of the three-dimensional scene by adjusting the camera pose, and simultaneously output the RGB image and normalized depth map of the current view using a differentiable rendering function. The proxy image generation and geometric restoration unit uses the conditional diffusion model InstructPix2Pix to generate a proxy image containing incremental objects by taking the RGB image and incremental semantics as conditional inputs; and takes the proxy image and the normalized depth map as inputs, and uses the monocular depth estimation model ZoeDepth to perform geometric restoration operations to obtain the corrected depth map. The target detection and segmentation unit extracts incremental text representing the newly added object by comparing the source description text and the target description text; then, using the proxy image and the incremental text as input, it uses the zero-shot object detection model Grounding DINO to obtain the bounding box of the incremental object; and uses the bounding box as input, it uses the segmentation basic model SAM to generate a pixel-level semantic mask. The multi-view fusion and aggregation unit, through a spatial aggregation strategy based on a robust voting mechanism and a depth consistency check, based on the corrected depth map and pixel-level semantic mask, finally aggregates to obtain the multi-view geometric consistency mask sequence and the 3D editing bounding box.

[0009] Preferably, the specific data processing procedure of the proxy image generation and geometric restoration unit is as follows: In the proxy image generation stage: the RGB image of the current viewpoint is used as the initial input, the incremental text is extracted as the editing instructions and semantic control signals, and the RGB image is redrawn using the conditional diffusion model InstructPix2Pix to generate the proxy image that merges the target incremental object and maintains the consistency of the original scene background; In the geometric restoration stage: the pixel-level relative depth map of the proxy image is predicted using the monocular depth estimation model ZoeDepth; then, the background region in the normalized depth map is extracted as a geometric anchor point, the linear transformation parameters are solved by the least squares method, and the pixel-level relative depth map is aligned to the physical depth space of the original scene using the linear transformation parameters to obtain the corrected depth map.

[0010] Preferably, the specific processing procedure of the multi-view fusion and aggregation unit is as follows: The 3D scene space is discretized into a voxel mesh, and potential editable voxels are selected by calculating the confidence accumulation function of the voxels. For the selected potential editable voxels, a depth consistency check is performed by calculating the difference between their theoretical depth in the camera coordinate system and the corrected depth map. Voxels with a difference exceeding a preset threshold are removed, and valid voxels are retained. The bounding boxes are aligned with the calculation axes of all valid voxels to determine the final 3D editable bounding box. Finally, a spatial-semantic joint verification strategy is adopted to project the 3D editing bounding box back to the image plane of each viewpoint to generate a geometric constraint mask, and then denoise the intersection of the geometric constraint mask with the pixel-level semantic mask to obtain the final multi-view geometric consistency mask sequence.

[0011] Preferably, the step of selecting potential editable voxels by calculating the confidence cumulative function of voxels specifically involves: Define confidence cumulative function Calculate voxels The weighted sum of the projection points falling within the foreground regions of the semantic masks corresponding to each viewpoint:

[0012] in, A three-dimensional voxel after discretization of the three-dimensional scene space; For the voxel The cumulative confidence level; For the voxel The set of visible viewpoints; The total number of visible viewpoints; i Represents the first element in the set. One perspective; The weights are based on projection centrality, with higher weights for projection points closer to the mask center. This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise; For the first Camera projection matrix for each viewpoint, Indicates voxels Project to the Two-dimensional pixel coordinates on a viewpoint image plane; For the first Pixel-level semantic mask corresponding to each viewpoint; Based on the calculated cumulative confidence value By setting appropriate confidence thresholds, a potential set of editable voxels can be selected.

[0013] Preferably, the dual-path optimizer includes editing the path and preserving the path; The dual-path optimizer receives the multi-view geometric consistency mask sequence as input and linearly weights and aggregates the incremental fractional distillation loss, geometric anchoring reconstruction loss, and boundary-aware gradient smoothing loss to construct the overall optimization objective function:

[0014] in, The total number of viewpoints. These are the editing loss weights, the preservation loss weights, and the smoothing loss weights, respectively. For the first Incremental fractional distillation loss from each perspective For the first Geometric anchoring reconstruction loss from each perspective For the first Boundary-aware gradient smoothing loss from multiple perspectives; The editing path strategy is as follows: Utilizing the multi-view geometrically consistent mask sequence, a text-driven editing gradient is applied only to the target component region indicated by the mask. Fractional distillation sampling (SDS) using a source semantic suppression mechanism is employed to remove common feature components belonging to the source text from the target gradient, extracting incremental semantic information. Its optimization objective is defined as:

[0015] in, For 3D scene parameters, For diffusion time step, For added random noise, The time-step weighting function, For the noise prediction network of the pre-trained diffusion model, As a latent variable, The target description text, The source description text, The source semantic suppression coefficient, The rendered image component corresponding to the region to be edited; The path-preserving strategy is as follows: apply geometric anchoring constraints to the non-edited areas indicated by the mask, requiring the current rendering result to remain aligned with the source model in both photometric and geometric dimensions; at the same time, introduce boundary-aware gradient stitching constraints to minimize the magnitude of the image gradient and depth gradient in the transition zone between the edited area and the background area, thereby eliminating visual seams. The geometric anchoring reconstruction loss is defined as:

[0016] in, For the multi-view geometric consistency mask sequence, the first... i A mask for each perspective. Used to lock the non-editable area; This represents an element-wise multiplication operation; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Rendering RGB images from various perspectives; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Depth map from multiple perspectives; These are the weighting coefficients for the geometric depth loss; This represents the square of the L2 norm.

[0017] The second aspect of the present invention provides a progressive text-guided 3D model generation system for complex semantic prompts, including a 3D semantic component localization module and a geometry-aware dual-path optimizer as described in the first aspect; it also includes a data acquisition module and a model rendering module; The data acquisition module is used to acquire source 3D scene data, source description text and target description text, and perform preprocessing. The model rendering module is used to render the updated 3D scene parameters and output the final 3D model image or video.

[0018] Compared with the prior art, the present invention has the following beneficial effects: First, this invention is a 3D model generation method and system based on zero-shot learning localization, geometric perception optimization, and diffusion model priors, enabling precise automatic localization of editable regions within a 3D model. This invention resolves the paradox of creating incremental objects from scratch through multi-view rendering and the InstructPix2Pix proxy generation strategy. The geometric perception dual-path optimizer provided by this invention coordinates editing drive and background anchoring, achieving a balance between precise local editing and maintaining global consistency. This system endows this invention with high-fidelity generation capabilities when handling complex semantic cues (such as multi-object interaction and attribute binding), effectively avoiding object missingness and feature aliasing.

[0019] Secondly, this invention provides a progressive text-guided 3D model generation method and system for complex semantic prompts. It achieves high-precision local incremental generation of 3D assets without manual data annotation, particularly involving deep learning and AIGC technologies. Automated editing of 3D models is achieved through a semantic localization module and a dual-path optimizer. This invention introduces a source semantic suppression mechanism in the editing path, effectively preventing excessive redrawing of existing content. It also introduces geometric anchoring and boundary smoothing constraints in the path preservation mechanism, achieving a Masked-PSNR of 32.84 dB, significantly exceeding existing technologies. This makes it suitable for applications with extremely high requirements for geometric consistency, such as digital twins and virtual reality content construction.

[0020] Third, the technical solution of the present invention solves the technical problems of semantic ambiguity in text-based 3D technology when processing complex logical prompts, and the forgetting of non-related areas during local editing. Attached Figure Description

[0021] Figure 1 This is a flowchart of a progressive text-guided 3D model generation method for complex semantic prompts provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the progressive text-guided 3D model generation method for complex semantic prompts provided in this embodiment of the invention; Figure 3 This is a structural diagram of a progressive text-guided 3D model generation system for complex semantic prompts provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the geometry-aware dual-path optimizer structure provided in an embodiment of the present invention.

[0022] Figure 5 This is a diagram illustrating the generation effect of the method of the present invention in Embodiment 2 of the present invention.

[0023] Figure 6 This is a diagram showing the generation effect of the baseline method MVDream in Embodiment 2 of the present invention. Detailed Implementation

[0024] Example 1: like Figure 1 As shown in the embodiment of the present invention, the progressive text-guided 3D model generation method for complex semantic prompts includes the following steps: The initial 3D scene representation is generated from the input source description text; Based on the initial 3D scene representation, and simultaneously inputting source description text and target description text, the 3D semantic part localization module based on zero-shot learning is used to identify the region to be edited, and generate a multi-view geometric consistency mask sequence and a 3D editing bounding box. The geometry-aware dual-path optimizer receives the multi-view geometric consistency mask sequence and the 3D edit bounding box as input data, and decouples the optimization process of the 3D scene into joint processing of the edit path and the preservation path: In the editing path, a text-driven gradient with source semantic suppression mechanism is applied to the area to be edited indicated by the mask, and incremental semantic information is extracted for redrawing and generation; While preserving the path, geometric anchoring constraints and boundary-aware gradient smoothing constraints are applied to the non-editable areas indicated by the mask to maintain the geometric and photometric consistency of the original scene background. Finally, the 3D scene parameters are updated, and a high-fidelity 3D model with local incremental generation is output.

[0025] Initialize the 3D scene and text input, and obtain the source 3D scene representation, source description text, and target description text; The region to be edited is identified by a 3D semantic part localization module based on zero-shot learning, and a multi-view geometric consistency mask sequence is generated. The geometry-aware dual-path optimizer receives the mask constraints output by the localization module, performs incremental generation optimization on the 3D scene, and returns the final 3D model.

[0026] As a preferred embodiment, such as Figure 2As shown, the progressive text-guided 3D model generation method for complex semantic prompts provided in this embodiment of the invention specifically includes the following steps: S0. Generate an initial 3D scene representation, specifically including: the system receiving the source description text input by the user. Using a pre-trained text-to-3D generation model (text-to-3D base model), an initial 3D scene representation corresponding to the source description text is generated. .

[0027] S1. Initialize the positioning module and data input, specifically including: based on the initial 3D scene representation. Source description text With target description text The 3D semantic part localization module based on zero-shot learning first adjusts the camera pose. Multi-view discrete sampling is performed on the 3D scene. Differentiable rendering functions are used. Simultaneously, it calculates the color integral and depth information of light on the pixel plane, and outputs an RGB image that reflects texture details. Normalized depth maps that explicitly capture the spatial topology of objects .

[0028] S2. View-guided 2D proxy image generation and geometric restoration, specifically including: To address the object omission problem in incremental generation tasks, an instruction-guided proxy image generation strategy is employed, introducing the InstructPix2Pix architecture as an image editor for the generation process. First, the rendered RGB image of the current viewpoint is used as the input image, and the target description text is used as the editing instruction, inputting it into a pre-trained conditional diffusion model (InstructPix2Pix). The model redraws the input image according to the semantic instructions, generating a proxy image that includes the incremental target object while maintaining consistency with the original scene background. Subsequently, a geometric restoration operation is performed, introducing the monocular depth estimation model ZoeDepth to predict the pixel-level relative depth map of the proxy image. To address the scale inconsistency problem, the background region in the original rendered depth map is used as a geometric anchor point. The linear transformation parameters are solved using the least squares method, aligning the predicted relative depth map to the physical depth space of the original scene, resulting in a corrected depth map.

[0029] S3. Zero-shot-based object detection and segmentation, specifically including: the system extracts incremental text descriptions representing newly added objects. Proxy image The Grounding DINO model, with incremental text input, outputs bounding boxes that tightly surround the newly added object region. Subsequently, a cascaded fine-grained segmentation strategy is employed, using the bounding boxes as geometric cues input to the SAM (Segment Anything Model) to generate pixel-level semantic masks. Furthermore, noise is filtered out through connected component analysis to obtain a clean target region mask. .

[0030] S4. Robust multi-view geometric fusion and 3D edit box aggregation, specifically including: To overcome the uncertainties caused by single-view detection noise and depth errors, this invention abandons the traditional strict frustum intersection strategy and instead adopts a spatial aggregation strategy based on a robust voting mechanism. First, the 3D scene space is discretized into a voxel mesh, and a confidence accumulation function is defined. Calculate voxels The weighted sum of the projection points falling within the foreground regions of the semantic masks corresponding to each viewpoint:

[0031] in, A three-dimensional voxel after discretization of the three-dimensional scene space; For the voxel The cumulative confidence level; For the voxel The set of visible viewpoints; Let i be the total number of visible viewpoints; i represents the i-th viewpoint in the set. One perspective; The weights are based on projection centrality, with higher weights for projection points closer to the mask center. This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise; For the first Camera projection matrix for each viewpoint, Indicates voxels Project to the Two-dimensional pixel coordinates on a viewpoint image plane; For the first Pixel-level semantic mask corresponding to each viewpoint.

[0032] Based on the above calculations, the cumulative confidence value A corresponding confidence threshold is set to filter out a set of potential editable voxels. Then, a depth consistency check is performed on these potential voxels, calculating the difference between their theoretical depth in the camera coordinate system and the corrected depth map. Voxels with differences exceeding a preset threshold are removed, and the final valid voxels are retained. All valid voxels are then computed to align their bounding boxes (AABB) to determine the final 3D editable bounding box. .

[0033] Finally, to obtain pixel-level accurate guidance signals, a "spatial-semantic joint verification" strategy is adopted. The calculated 3D editing bounding box... Projecting back onto the image planes from each viewpoint generates a geometric constraint mask. The key is to combine this geometric constraint mask with the original fine-grained semantic mask generated by the Segmentation Base Model (SAM). The intersection is then determined. This step utilizes 3D geometric constraints to remove background noise from the SAM mask while preserving sub-pixel details at the edges, ultimately yielding a multi-view consistent mask sequence. .

[0034] S5. Geometric perception dual-path optimization, the structure of which is as follows: Figure 4 As shown; specifically including: The geometry-aware dual-path optimizer receives a mask sequence. The optimization process is decoupled into edit paths and maintain paths.

[0035] (1) Edit path: Only for the target component area indicated by the mask ( Text-driven editing gradients are applied. Semantic-suppressed fractional distillation sampling (SDS) is employed, introducing a source semantic suppression mechanism when calculating the gradient to remove common feature components belonging to the source text from the target gradient. Its optimization objective is defined as:

[0036] in, For 3D scene parameters, For diffusion time step, For added random noise, The time-step weighting function, For the noise prediction network of the pre-trained diffusion model, As a latent variable, The target description text, The source description text, The source semantic suppression coefficient, The rendered image component corresponding to the region to be edited.

[0037] (2) Preserve Path: For the non-editable area indicated by the mask ( Apply geometric anchoring constraints and boundary smoothing constraints. Geometric anchoring reconstruction loss. By minimizing the non-editable regions in the RGB images and depth maps before and after editing Norm, forces the background region to remain unchanged:

[0038] in, For the multi-view geometric consistency mask sequence, the first... i A mask for each perspective. Used to lock the non-editable area; This represents an element-wise multiplication operation; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Rendering RGB images from various perspectives; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Depth map from multiple perspectives; These are the weighting coefficients for the geometric depth loss; This represents the square of the L2 norm.

[0039] Boundary smoothing loss It's not simply pixel smoothing, but rather introducing boundary-aware gradient stitching constraints within the gradient domain. First, a transition band is defined by morphological dilation of the mask boundaries. Within this region, the magnitudes of the image gradient and depth gradient are minimized. This forces the signal variations of the generated content at the boundaries to satisfy the Lipschitz continuity condition, thus mathematically smoothing gradient conflicts and eliminating visual seams.

[0040]

[0041] in, For the first Boundary smoothing loss from each perspective; This is the transition zone region defined after morphological dilation of the boundary of the mask indication area; The transition zone region Two-dimensional pixels within; For the first Rendering an RGB image from a different perspective at the pixel level Image gradient at the location; For the first Depth map from various perspectives at pixel level Depth gradient at that location; These are the weighting coefficients for the depth gradient loss; This represents the square of the L2 norm, used to calculate the magnitude of the gradient.

[0042] S6. Overall Loss Function Optimization, specifically including: Linearly weighting and aggregating the incremental fractional distillation loss, geometric anchoring reconstruction loss, and boundary smoothing loss to construct the overall optimization objective:

[0043] in, The total number of viewpoints. These are the editing loss weights, the preservation loss weights, and the smoothing loss weights, respectively. For the first Incremental fractional distillation loss from each perspective For the first Geometric anchoring reconstruction loss from each perspective For the first A boundary-aware gradient smoothing loss from multiple perspectives.

[0044] In embodiments of the present invention, it is typically set This provides stronger rigidity to the geometric anchoring, ensuring a stable background. These six steps enable the incremental generation of high-fidelity 3D models for complex semantic cues.

[0045] like Figure 3 As shown in the figure, this embodiment of the invention also provides a progressive text-guided 3D model generation system for complex semantic prompts. The overall structure includes four parts: a data acquisition module, a 3D semantic component localization module, a geometry-aware dual-path optimizer, and a rendering and output module. The structures of the 3D semantic component localization module and the geometry-aware dual-path optimizer are as described above. The main function of the data acquisition module is to load and preprocess the 3D scene and text. The rendering and output module is responsible for converting the optimized 3D parameters into a visualized 3D model or video stream.

[0046] This embodiment of the invention illustrates the execution flow of the entire system for user operation. First, the user inputs an initial 3D model and corresponding incremental prompts. The system automatically invokes the semantic localization module, generating proxy images through multi-view rendering and InstructPix2Pix, and automatically calculating the precise 3D location where the incremental semantics should exist using Grounding DINO and SAM technologies. Subsequently, the system activates a dual-path optimizer, driving the generation of the geometry and texture of the corresponding object for the incremental semantic prompts within the editing path, while strictly locking the non-editing areas within the path to prevent deformation. After optimization, the system outputs the final high-precision 3D model.

[0047] Example 2: The technical solution and technical effects of the present invention will be further explained below with reference to specific experiments.

[0048] In one embodiment of digital twin content construction, the technical solution of this invention serves as an automated adaptation service for 3D assets. When handling the complex semantic cue "a soldier wearing medieval armor, a cowboy hat, and a silver longsword riding a wooden pig statue," traditional methods often produce models with geometrically overlapping soldiers and pig statues, armor color leakage onto the pig statue, and the cowboy hat not being generated. Applying the technical solution of this invention, the system first accurately identifies the spatial regions of the "pig statue," "silver longsword," and "cowboy hat" through a semantic localization module. Subsequently, when generating the texture of the wooden pig statue, the dual-path optimizer uses geometric anchoring constraints to force the soldier's armor area to maintain its original color, effectively avoiding attribute mismatch. The generation effect of this method is as follows: Figure 5 As shown, the generation effect of the baseline method MVDream is as follows: Figure 6 The experimental data shows that, in the self-built complex semantic prompt test set CSA-100, the model generated by the method of this invention improves the accuracy of visual question answering (B-VQA) to 0.478, which is far higher than that of traditional techniques, verifying the superiority of this invention in fine-grained semantic control.

[0049] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0050] While the specific embodiments of the present invention have been described above, they are not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A progressive text-guided 3D model generation method for complex semantic prompts, characterized in that: The system employs a zero-shot learning-based 3D semantic component localization module and a geometry-aware dual-path optimizer. It uses a zero-shot learning-based localization strategy combined with a large visual model to automatically identify and locate the editable regions in the 3D model. Then, the geometry-aware dual-path optimizer drives the localized regions to evolve towards the target semantics while constraining the non-editable regions to remain unchanged, finally outputting a high-fidelity 3D model.

2. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 1, characterized in that: The specific processes include the following: Step 1: Input source description text to generate an initial 3D scene representation; Step 2: Based on the initial 3D scene representation, and simultaneously inputting source description text and target description text, the 3D semantic component localization module based on zero-shot learning is used to identify the region to be edited, and generate a multi-view geometric consistency mask sequence and a 3D editing bounding box. Step 3: The geometry-aware dual-path optimizer receives the multi-view geometric consistency mask sequence and the 3D edit bounding box as input data, and decouples the optimization process of the 3D scene into joint processing of the edit path and the preservation path: In the editing path, a text-driven gradient with source semantic suppression mechanism is applied to the area to be edited indicated by the mask, and incremental semantic information is extracted for redrawing and generation; While preserving the path, geometric anchoring constraints and boundary-aware gradient smoothing constraints are applied to the non-editable areas indicated by the mask to maintain the geometric and photometric consistency of the original scene background. Finally, the 3D scene parameters are updated, and a high-fidelity 3D model with local incremental generation is output.

3. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 2, characterized in that: The 3D semantic component localization module based on zero-shot learning specifically includes a sampling rendering unit, a proxy image generation and geometric restoration unit, an object detection and segmentation unit, and a multi-view fusion and aggregation unit. The sampling and rendering unit is used to perform multi-view discrete sampling of the three-dimensional scene by adjusting the camera pose, and simultaneously output the RGB image and normalized depth map of the current view using a differentiable rendering function. The proxy image generation and geometric restoration unit uses the conditional diffusion model InstructPix2Pix to generate a proxy image containing incremental objects by taking the RGB image and incremental semantics as conditional inputs; and takes the proxy image and the normalized depth map as inputs, and uses the monocular depth estimation model ZoeDepth to perform geometric restoration operations to obtain the corrected depth map. The target detection and segmentation unit extracts incremental text representing the newly added object by comparing the source description text and the target description text; then, using the proxy image and the incremental text as input, it uses the zero-shot object detection model Grounding DINO to obtain the bounding box of the incremental object. The bounding box is used as input to generate a pixel-level semantic mask using the segmentation basic model SAM. The multi-view fusion and aggregation unit, through a spatial aggregation strategy based on a robust voting mechanism and a depth consistency check, based on the corrected depth map and pixel-level semantic mask, finally aggregates to obtain the multi-view geometric consistency mask sequence and the 3D editing bounding box.

4. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 3, characterized in that: The specific data processing procedure of the proxy image generation and geometric restoration unit is as follows: In the proxy image generation stage: the RGB image of the current viewpoint is used as the initial input, the incremental text is extracted as the editing instructions and semantic control signals, and the RGB image is redrawn using the conditional diffusion model InstructPix2Pix to generate the proxy image that merges the target incremental object and maintains the consistency of the original scene background; In the geometric restoration stage: the pixel-level relative depth map of the proxy image is predicted using the monocular depth estimation model ZoeDepth; then, the background region in the normalized depth map is extracted as a geometric anchor point, the linear transformation parameters are solved by the least squares method, and the pixel-level relative depth map is aligned to the physical depth space of the original scene using the linear transformation parameters to obtain the corrected depth map.

5. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 3, characterized in that: The specific processing procedure of the multi-view fusion and aggregation unit is as follows: The 3D scene space is discretized into a voxel mesh, and potential editable voxels are selected by calculating the confidence accumulation function of the voxels. For the selected potential editable voxels, a depth consistency check is performed by calculating the difference between their theoretical depth in the camera coordinate system and the corrected depth map. Voxels with a difference exceeding a preset threshold are removed, and valid voxels are retained. The bounding boxes are aligned with the calculation axes of all valid voxels to determine the final 3D editable bounding box. Finally, a spatial-semantic joint verification strategy is adopted to project the 3D editing bounding box back to the image plane of each viewpoint to generate a geometric constraint mask, and then denoise the intersection of the geometric constraint mask with the pixel-level semantic mask to obtain the final multi-view geometric consistency mask sequence.

6. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 5, characterized in that: The process of selecting potential editable voxels by calculating the cumulative confidence function of voxels specifically involves: Define confidence cumulative function Calculate voxels The weighted sum of the projection points falling within the foreground regions of the semantic masks corresponding to each viewpoint: in, A three-dimensional voxel after discretization of the three-dimensional scene space; For the voxel The cumulative confidence level; For the voxel The set of visible viewpoints; The total number of visible viewpoints; i Represents the first element in the set. One perspective; The weights are based on projection centrality, with higher weights for projection points closer to the mask center. This is an indicator function that takes the value 1 when the condition inside the parentheses is true, and 0 otherwise; For the first Camera projection matrix for each viewpoint, Indicates voxels Project to the Two-dimensional pixel coordinates on a viewpoint image plane; For the first Pixel-level semantic mask corresponding to each viewpoint; Based on the calculated cumulative confidence value By setting appropriate confidence thresholds, a potential set of editable voxels can be selected.

7. The progressive text-guided 3D model generation method for complex semantic prompts as described in claim 2, characterized in that: The dual-path optimizer includes path editing and path preservation; The dual-path optimizer receives the multi-view geometric consistency mask sequence as input and linearly weights and aggregates the incremental fractional distillation loss, geometric anchoring reconstruction loss, and boundary-aware gradient smoothing loss to construct the overall optimization objective function: in, The total number of viewpoints. These are the editing loss weights, the preservation loss weights, and the smoothing loss weights, respectively. For the first Incremental fractional distillation loss from each perspective For the first Geometric anchoring reconstruction loss from each perspective For the first Boundary-aware gradient smoothing loss from multiple perspectives; The editing path strategy is as follows: Utilizing the multi-view geometrically consistent mask sequence, a text-driven editing gradient is applied only to the target component region indicated by the mask. Fractional distillation sampling (SDS) using a source semantic suppression mechanism is employed to remove common feature components belonging to the source text from the target gradient, extracting incremental semantic information. Its optimization objective is defined as: in, For 3D scene parameters, For diffusion time step, For added random noise, The time-step weighting function, For the noise prediction network of the pre-trained diffusion model, As a latent variable, The target description text, The source description text, The source semantic suppression coefficient, The rendered image component corresponding to the region to be edited; The path-preserving strategy is as follows: apply geometric anchoring constraints to the non-edited areas indicated by the mask, requiring the current rendering result to remain aligned with the source model in both photometric and geometric dimensions; at the same time, introduce boundary-aware gradient stitching constraints to minimize the magnitude of the image gradient and depth gradient in the transition zone between the edited area and the background area, thereby eliminating visual seams. The geometric anchoring reconstruction loss is defined as: in, For the multi-view geometric consistency mask sequence, the first... i A mask for each perspective. Used to lock the non-editable area; This represents an element-wise multiplication operation; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Rendering RGB images from various perspectives; and The currently optimized 3D scene and the source 3D scene are respectively in the 3D scene at ... Depth map from multiple perspectives; These are the weighting coefficients for the geometric depth loss; This represents the square of the L2 norm.

8. A progressive text-guided 3D model generation system for complex semantic prompts, characterized in that: It includes a 3D semantic component localization module and a geometry-aware dual-path optimizer as described in any one of claims 1 to 7; it also includes a data acquisition module and a model rendering module; The data acquisition module is used to acquire source 3D scene data, source description text and target description text, and perform preprocessing. The model rendering module is used to render the updated 3D scene parameters and output the final 3D model image or video.

Images