A 3D scene editing method based on multi-view consistency constraints and interactive segmentation

By employing a 3D scene editing method based on multi-view consistency constraints and interactive segmentation, the problems of insufficient consistency in multi-view editing and difficulty in accurately segmenting local targets in existing technologies are solved, achieving efficient and accurate 3D scene editing and improving editing efficiency and result consistency.

CN123118987APending Publication Date: 2026-11-17NANJING UNIV OF AERONAUTICS & ASTRONAUTICS
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611098060.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-23
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing 3D scene editing technologies suffer from problems such as insufficient consistency in multi-view editing, low efficiency in batch editing, difficulty in accurately segmenting and constraining local targets, and long overall editing time, resulting in low editing efficiency and inconsistent results.

Method used

A 3D scene editing method based on multi-view consistency constraints and interactive segmentation is adopted. By reconstructing the initial 3D Gaussian scene, locating and segmenting the region to be edited, performing multi-view consistency editing, and combining encoder caching and parallel decoding strategies, local feature merging and global feature merging are performed. Finally, target region constraint fusion and iterative optimization are performed to generate multi-view consistent editing results.

Benefits of technology

It improves the efficiency and accuracy of 3D scene editing, reduces redundant calculations and invalid iterations during the editing process, enhances cross-view consistency, and ensures locally controllable and efficient editing results.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118987A_ABST
    Figure CN123118987A_ABST
Patent Text Reader

Abstract

This invention discloses a 3D scene editing method based on multi-view consistency constraints and interactive segmentation. It performs local and global feature merging based on cross-view feature similarity to enhance editing consistency across different views and reduce the computational overhead of the editing model. During the diffusion model inference stage, encoder caching and parallel decoding strategies reduce redundant computation of encoder features during diffusion sampling, thereby accelerating the 3D scene editing process. Simultaneously, the interactive segmentation module combines user prompts and depth information to generate consistent target region segmentation results across different views, determining the 3D target region to be edited, thus achieving accurate editing of local targets and reducing erroneous modifications to the background region. This invention enables consistent editing of large batches of multi-view images under limited computing resources, improving the efficiency, view consistency, and local editing accuracy of 3D scene editing, and reducing artifacts during the editing process.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of computer graphics and artificial intelligence 3D content generation technology, and relates to a 3D scene editing method based on multi-view consistency constraints and interactive segmentation. Background Technology

[0002] With the development of computer vision and 3D reconstruction technologies, 3D scene editing has become an important technology in fields such as virtual reality, digital media production, and intelligent interaction. Traditional 3D scene editing mainly relies on manual modeling and texture modification, which suffers from complex operation, high cost, and low editing efficiency. In recent years, with the development of 3D Gaussian sputtering and diffusion models, using 2D image editing models to achieve intelligent 3D scene editing has gradually become a research hotspot.

[0003] However, existing 3D scene editing techniques still have shortcomings. Many current methods employ per-view editing, meaning that image editing is performed only on a single rendering viewpoint at a time, and the 3D scene is updated accordingly. Due to the lack of cross-view consistency constraints, the same object is prone to inconsistencies in color, texture, or structure under different views, often requiring multiple rounds of rendering and optimization to obtain stable results, leading to low editing efficiency. Secondly, while some methods improve efficiency through multi-view batch input, their batch size is usually small due to limitations in GPU memory overhead and the inference complexity of image editing models. As the number of views increases, insufficient GPU memory and a significant increase in inference latency easily occur, making it difficult to simultaneously achieve large-batch editing capabilities and cross-view consistency.

[0004] Furthermore, accurate localization of the target region is a significant challenge in local 3D scene editing tasks. One type of method relies primarily on text-driven 2D segmentation or object detection models to determine the editing area. However, in complex backgrounds, with small targets, fine-grained regions, or occluded scenes, problems such as inaccurate localization, coarse segmentation boundaries, and inconsistencies across viewpoints often arise. Other 3D segmentation methods, while capable of acquiring the target region in 3D space, typically rely on scene-level training or additional complex annotations, resulting in high costs and making them difficult to directly apply to interactive 3D editing. Due to the lack of stable and consistent target region constraints, existing local editing methods are prone to mistakenly modifying non-target regions, affecting the accuracy and naturalness of the results.

[0005] Therefore, there is an urgent need for a new 3D scene editing method to solve the technical problems existing in the current technology, such as insufficient consistency of multi-view editing, low efficiency of batch editing, difficulty in accurately segmenting and constraining local targets, and long overall editing time, so as to achieve efficient, accurate and consistent 3D scene editing for real-world scenes. Summary of the Invention

[0006] Purpose of the invention: To address the aforementioned existing technologies, a 3D scene editing method based on multi-view consistency constraints and interactive segmentation is proposed.

[0007] Technical solution: A 3D scene editing method based on multi-view consistency constraints and interactive segmentation, the method comprising the following steps:

[0008] Step S1: Reconstruct the initial 3D Gaussian scene: Acquire multi-view image data of the target scene, perform motion recovery structure processing on the multi-view image data to obtain the camera intrinsic parameters, extrinsic parameters and sparse point cloud of the scene corresponding to each view, and initialize the 3D Gaussian scene representation based on the obtained sparse point cloud.

[0009] Step S2: Locate and segment the area to be edited: Based on the interactive point prompts and depth information of the reference viewpoint, the two-dimensional prompts are back-projected to the three-dimensional space to obtain the three-dimensional prompts of the target area; then projected to other views to generate multi-view two-dimensional guidance prompts, combined with the segmentation model to obtain the target area mask of each viewpoint, and mapped back to the three-dimensional Gaussian scene to obtain the three-dimensional Gaussian mask of the area to be edited.

[0010] Step S3: Perform multi-view consistent editing: Input the original images from multiple views into the image editing model in batches for encoding. Based on the feature similarity between different views, perform local feature merging and global feature merging. In the diffusion model inference process, the encoder caching and parallel decoding strategy is adopted to finally output the multi-view edited result image.

[0011] Step S4, Target Region Constraint Fusion: Based on the target region mask, perform region constraints and fusion on the obtained multi-view editing result image to achieve target region editing and non-target region preservation;

[0012] Step S5: Iterative optimization and generation of the final edited scene: Using the constrained editing results as a supervision signal, the parameters of the 3D Gaussian scene representation are updated, and the 3D Gaussian scene representation is rendered, edited and updated in multiple rounds until the preset number of iterations is reached or the stopping condition is met, and the edited target 3D scene is output to achieve efficient 3D scene editing with consistent multi-view perspectives and local controllability.

[0013] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the three-dimensional scene editing method.

[0014] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the three-dimensional scene editing method.

[0015] Beneficial effects: This invention reduces repetitive calculations and invalid iterations during the editing process by employing batch editing, encoder caching and parallel decoding, and early stopping optimization strategies, thereby improving the efficiency of 3D scene editing and shortening the overall editing time.

[0016] The view token merging method proposed in this invention can achieve collaborative constraints on editing information between different viewpoints, thereby improving the cross-view consistency of 3D scene editing results and reducing inconsistencies in color, texture, and structure.

[0017] This invention introduces an interactive segmentation and three-dimensional region constraint mechanism to achieve accurate positioning and locally controllable editing of the target area to be edited, reduce erroneous editing of non-target areas, and improve the accuracy and naturalness of the editing results.

[0018] This invention enables efficient and accurate editing of 3D scenes under limited computing resources, and has great practical value. Attached Figure Description

[0019] Figure 1 This is a flowchart of the method of the present invention;

[0020] Figure 2 This is a schematic diagram of the three-dimensional scene editing framework structure of the method of the present invention, wherein the real person photos involved are from the publicly published paper "NeRF-Art: Text-Driven Neural Radiance FieldsStylization";

[0021] Figure 3 This is a flowchart of the algorithm for the diffusion inference parallel decoding strategy proposed in this invention. Detailed Implementation

[0022] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0023] like Figure 1 As shown, a 3D scene editing method based on multi-view consistency constraints and interactive segmentation includes the following steps:

[0024] Step S1: Reconstruct the initial 3D Gaussian scene.

[0025] This step is used to reconstruct the corresponding 3D scene, providing a basic scene representation for subsequent editing of the target scene into a 3D scene. First, multi-view image data of the target scene is acquired. Then, Structure from Motion (SfM) processing is performed on the multi-view image data to obtain the camera intrinsic and extrinsic parameters and sparse point cloud of the scene corresponding to each viewpoint. Finally, a 3D Gaussian scene representation is initialized based on the obtained sparse point cloud.

[0026] The implementation process of step S1 specifically includes the following steps:

[0027] S101. A camera or other imaging device is used to continuously capture images of the static target scene. During the capture process, the camera moves along a circumferential or translational trajectory, ensuring that the shooting perspective covers multiple observation directions of the target scene, thus obtaining continuous video data. The video data is divided into frames at equal time intervals, and images whose common visible area ratio of adjacent frames meets preset requirements are selected to form a multi-view image sequence. The multi-view image sequence obtained in this step is the original visual input of the entire method, corresponding to the real geometric structure and appearance information of the target scene. During the selection process, sufficient overlap between adjacent frames is ensured to guarantee the feature matching accuracy of subsequent 3D reconstruction.

[0028] S102. Perform motion reconstruction (SfM) processing on the multi-view image sequence obtained in step S101, and sequentially perform feature point extraction and matching, camera self-calibration, pose joint optimization, and 3D point triangulation calculation, and finally output the camera intrinsic parameters, camera extrinsic parameters, and sparse point cloud of the target scene corresponding to each view.

[0029] The camera intrinsic parameter matrix includes inherent optical parameters such as camera focal length, principal point coordinates, and lens distortion coefficient; the camera extrinsic parameters correspond to the camera's rotation matrix and translation vector in the world coordinate system for each viewpoint image; the scene sparse point cloud is composed of the three-dimensional world coordinates of highly saliency feature points in the scene, which can roughly restore the overall geometric contour and spatial distribution of the scene, and is used to provide initial spatial anchor points for subsequent three-dimensional Gaussian elements.

[0030] S103. Based on the sparse point cloud output in step S102, the initial 3D Gaussian scene is constructed and optimized using 3D Gaussian Splatting (3DGS) technology. The implementation process is as follows:

[0031] First, Gaussian element initialization is performed: a 3D Gaussian element is generated centered on each 3D feature point in the sparse point cloud; each 3D Gaussian element contains four types of optimizable parameters: 3D spatial coordinates, 3D covariance matrix, spherical harmonic color coefficients, and opacity. The covariance matrix is ​​used to characterize the shape, size, and spatial orientation of the Gaussian element, and the spherical harmonic color coefficients are used to characterize the viewpoint-dependent appearance color. During the initialization phase, the covariance is set to an isotropic small-scale initial value, the color parameters are sampled and assigned values ​​from the original image of the corresponding viewpoint, and the opacity is set to a preset initial value.

[0032] Subsequently, differentiable rendering optimization is performed: a Gaussian sputtering differentiable rendering pipeline is used to project and rasterize all 3D Gaussian elements into rendered images corresponding to each viewpoint; using the original multi-view images from step S101 as supervised ground truth, a joint loss function combining pixel-level L1 loss and perceptual loss is constructed, and the parameters of all 3D Gaussian elements are iteratively updated through gradient backpropagation; during the optimization process, adaptive splitting and redundancy pruning operations are performed on the Gaussian elements according to the local detail density of the scene, until the error between the rendered image and the original image converges to a preset threshold, resulting in the optimized initial 3D Gaussian scene, denoted as scene. .

[0033] Step S2: Locate and segment the area to be edited.

[0034] The purpose of this step is to accurately locate and segment the target editing object in the initial 3D Gaussian scene, transforming the user's simple interaction from a single perspective into a consistent 2D segmentation mask and 3D spatial mask across different perspectives. This provides spatial boundary information for subsequent local region constraint editing and directional updates of 3D Gaussian parameters, fundamentally preventing the accidental modification of non-target areas. This step achieves cross-view propagation of interactive prompts based on 3D geometric projection and generates region masks using an interactive segmentation model, specifically through the following sub-steps:

[0035] S201. Based on the camera intrinsic parameter matrix obtained in step S1 Select the transformation matrix from the camera coordinate system to the world coordinate system corresponding to the reference viewpoint. The initial 3D Gaussian scene is rendered using the 3D Gaussian sputtering differentiable rendering pipeline. Rendering is performed to obtain a color rendered image corresponding to the reference viewpoint. and pixel-by-pixel depth map Among them, depth map Each pixel value corresponds to the depth distance of a point on the scene surface along the camera's optical axis from that viewpoint.

[0036] In the image Input interactive prompts, including positive and negative prompts: positive prompts are marked within the target area to be edited, indicating the segmentation of the target subject; negative prompts are marked within non-target areas to be excluded, used to correct segmentation boundaries and eliminate background interference. Record the two-dimensional pixel coordinates corresponding to each interactive prompt. .

[0037] S202. Based on the two-dimensional pixel coordinates and corresponding depth values ​​of the interactive prompt points, the two-dimensional prompt points are transformed into three-dimensional spatial prompt points in the world coordinate system through camera geometric inverse projection, realizing the dimensionality upgrade of interactive information from a two-dimensional plane to a three-dimensional space. The specific calculation process is as follows:

[0038] Get the pixel coordinates of the interaction point In depth map The corresponding depth value Combined with the inverse of the camera intrinsic parameter matrix The coordinates of the three-dimensional point in the camera coordinate system are calculated. :

[0039]

[0040] Camera-to-world transformation matrix combined with reference viewpoint Convert 3D points in the camera coordinate system to homogeneous world coordinates. :

[0041]

[0042] in, The table shows the homogeneous world coordinates corresponding to the interactive prompt points, which are globally unique and not constrained by the observation position from a single perspective.

[0043] The inverse projection calculation is performed on each of the positive and negative interactive cue points to obtain a set of 3D cue points in the world coordinate system, including foreground and background 3D cue points. This step ensures the spatial uniformity of interactive cue points from a geometric perspective and is the core foundation for achieving consistent segmentation across different viewpoints.

[0044] S203. Project the global 3D cue points onto all target viewpoints to generate 2D guidance cue points. Combine this with the segmentation model to generate 2D segmentation masks for each viewpoint, and further map these masks to obtain 3D Gaussian-grained region masks. The specific process is as follows:

[0045] For any target perspective Obtain the transformation matrix from the world coordinate system to the camera coordinate system corresponding to this viewpoint. , 3D prompt points Projecting the image onto the image plane at this viewpoint yields the corresponding two-dimensional guidance and prompt points. The computational relationship satisfies the homogeneous coordinate equation:

[0046]

[0047] In the formula This represents an equation in a homogeneous coordinate system. During calculation, first... Perform nonhomogeneization processing, then combine with the intrinsic parameter matrix. Multiply to obtain pixel coordinates. Perform projection calculations on all target viewpoints to obtain the foreground and background guide points for each viewpoint.

[0048] The foreground and background cue points corresponding to each viewpoint are input into the image segmentation mode of the interactive segmentation model SAM2 (Segment Anything Model 2), and the segmentation model outputs a binary mask of the target region for the corresponding viewpoint. In the mask, a pixel value of 1 indicates that it belongs to the target area, and a value of 0 indicates that it belongs to a non-target area.

[0049] The 2D segmentation masks of each viewpoint are reverse-mapped to the 3D Gaussian scene: Based on the correspondence between the camera pose of each viewpoint and the pixel-Gaussian unit of differentiable rendering, the proportion of each 3D Gaussian unit that is determined to be the target region in all visible viewpoints is counted; when the proportion is higher than the preset proportion threshold, the 3D Gaussian unit is marked as the target Gaussian unit to be edited, otherwise it is marked as a non-target Gaussian unit; finally, a 3D Gaussian mask covering the entire scene is obtained, which is used to identify the 3D spatial region to be edited.

[0050] This step outputs two-dimensional segmentation masks and three-dimensional Gaussian masks for each viewpoint, providing precise spatial constraints for subsequent editing region constraints and scene parameter directional updates.

[0051] Step S3: Perform multi-view consistency editing.

[0052] This step is used to perform instruction-driven batch consistency editing on multi-view scene images, aiming to solve the technical problems of poor cross-view consistency in per-view editing, high memory overhead in large-batch joint editing, and low computational efficiency in diffusion inference in existing technologies. This step takes the multi-view original images rendered in step S1 as input, uses block processing to reduce single-step memory peaks, and achieves multi-view semantic alignment and editing consistency constraints through a two-level mechanism of intra-block local feature merging and cross-block global feature merging. Simultaneously, encoder caching and parallel decoding strategies are introduced during diffusion inference to compress computational overhead, ultimately outputting a cross-view consistent multi-view edited result image. Figure 3 As shown, this is achieved through the following sub-steps:

[0053] S301. The multi-view rendering image to be edited obtained in step S1 is... The images are sorted according to the pose spatial adjacency of the corresponding cameras to form a batch view sequence with continuous view transitions and high scene overlap. The sorted batch view sequence is then input into the Instruct-Pix2Pix instruction-driven diffusion image editing model. The model's built-in VAE encoder performs dimensionality reduction encoding on the images of each view to obtain the corresponding latent space feature tensor. ,in This indicates the total number of views included in the batch. This represents the total number of latent space feature tokens (corresponding to spatial dimensions) corresponding to a single image after VAE encoding. Channel dimension representing latent space characteristics.

[0054] This step compresses the high-dimensional pixel space image into a low-dimensional latent space, reducing the computational cost of subsequent diffusion inference while preserving scene semantics and structural information. The output multi-view latent space features provide the input basis for subsequent block segmentation and feature merging.

[0055] S302, Encoded multi-view latent space features Divide the data into consecutive blocks according to the viewpoint order to obtain... Feature blocks , of which Feature blocks , The number of views contained in this chunk, and the total number of views across all chunks satisfy the following condition: By using a block-based approach, the memory usage and computational burden during large-scale collaborative editing can be reduced.

[0056] S303. In the process of inference using the diffusion model, an encoder caching and parallel decoding strategy is adopted to cache the reusable encoded features between adjacent denoising moments and to process the corresponding decoding process in parallel, so as to reduce the computational overhead of repeated encoding and improve inference efficiency.

[0057] S304. Perform a similarity-driven merging operation on the multi-view feature tokens within a single block. This reduces the total number of tokens to decrease the computational complexity of self-attention and aligns features in semantically similar regions within the block, enhancing cross-view editing consistency. The specific process is as follows:

[0058] Latent space features of the current block Split into anchor set Source set Where the anchor set serves as the feature alignment benchmark, and the source set serves as the features to be merged, the number of views corresponding to both satisfy the following condition: .

[0059] Flattening the view dimension and spatial dimension of the anchor set and source set respectively, we obtain a two-dimensional feature matrix. and .

[0060] Calculate the cosine similarity matrix between all source tokens and anchor tokens. , of which Source token and the first The similarity value of the anchor tokens is:

[0061]

[0062] In the formula Represents the vector dot product. This represents the magnitude of the vector.

[0063] For each source token Match the anchor token with the highest similarity to obtain the optimal matching index:

[0064]

[0065] Then according to similarity value Sort in descending order and select the top with the highest similarity. Each source token is used to merge its features into the corresponding matching anchor token; the remaining... Source tokens with low similarity were retained; among them The value can be:

[0066]

[0067] In the formula Indicates rounding down. This is the token merging ratio coefficient, used to control the degree of feature compression.

[0068] After local feature merging, the total number of tokens participating in self-attention computation within the block is reduced, and the efficiency of self-attention computation is significantly improved. At the same time, semantically corresponding and spatially homogeneous regions in different perspectives achieve feature fusion, ensuring cross-perspective consistency of editing results within the block from the feature level.

[0069] S305. Based on local feature merging, a global token library is introduced. This enables cross-block information exchange and feature alignment, resolving issues such as discontinuity in editing style between blocks and insufficient global consistency caused by block-based processing. A global token library is also included. Representative edit feature tokens used to store historical blocks serve as a shared carrier for cross-block information transfer. The specific execution process is as follows:

[0070] Let the first Features of each block after local merging When processing this block, use a preset probability. Randomly swapped local block features With global token library The roles of the source token and anchor token are determined, and cross-block token merging is performed according to the local feature merging rules in S304 to obtain the globally merged block features. When the global token store When local block features are used as the anchor set and the local block features are used as the source set, the current block features are aligned with the global features, absorbing the editing semantics and style information of historical blocks to ensure the consistency of the editing effect; when local block features are used as the anchor set and the global token library When used as a source set, the global token is aligned with the current block features, integrating the scene features and editing information of the current block into the global token library to achieve iterative updates of global features.

[0071] After the cross-block merge is completed, the source tokens that were not merged will be added to the global token store. Replace historical tokens with low representativeness in the token library, maintain the scenario adaptability and information timeliness of the global token library, and enhance the global consistency of multi-view editing across all batches.

[0072] S306. After the self-attention calculation is completed, an anti-merging operation is performed on the compressed feature tokens to restore the feature dimension to the token size at the input, so as to match the input requirements of the subsequent cross-attention layer and decoding module. The specific process is as follows:

[0073] Based on the token matching mapping relationship recorded in the local and global merging stages, the merged anchor token features are reverse-mapped to the corresponding source token positions, so that the source tokens obtain edit features consistent with the matching anchor tokens; the output features after demerging are split into two parts: a global token and a local token; the global token is stored in the global token library. In the middle, the global features are used for merging in subsequent blocks; the local tokens are restored to the number of views and the size of the spatial tokens when the current block is input, and used as the input of the subsequent cross-attention layer to ensure that the interaction dimensions of text instructions and image features are matched.

[0074] S307. Using the VAE decoder, perform upsampling decoding on the latent space feature tokens that have completed all diffusion denoising steps, mapping the low-dimensional latent space features back to the RGB pixel space, and generating multi-view editing result images corresponding to each viewpoint. This is for use in subsequent editing of area constraints and updates to the 3D scene.

[0075] Step S4: Target region constraint fusion.

[0076] This step is used to spatially limit and smoothly blend the multi-view full-image editing results output in step S3. Combined with the target region segmentation mask generated in step S2, the editing effect is strictly constrained within the specified target area, while completely preserving the original scene appearance of non-target areas. This solves the problems of background mismodification and jagged edges and splicing marks that easily occur in full-image editing of diffusion models, and provides accurate local supervision signals for subsequent parameter iteration updates of the 3D Gaussian scene. Specifically, it is implemented through the following sub-steps:

[0077] S401. Use the binary segmentation mask of the target region from each viewpoint generated in step S2. As a spatial template, the multi-view edited image output in step S3 is used. Pixel-level region filtering constraints are applied: target region pixels with a mask value of 1 are fully preserved in the edited image. The corresponding pixel values ​​in the image ensure that the text-driven editing effect is fully effective within the target area; pixels in non-target areas with a mask value of 0 are rendered using the original rendered image obtained in step S1. Replace the corresponding pixel values ​​in the image to mask any unintended interference from the diffusion editing process on the background area.

[0078] This step is a spatial hard constraint, which defines the boundary where the editing takes effect at the pixel level, and prevents the editing effect from spreading to non-target areas from the source. The output preliminary constrained image has a hard boundary transition between the target area and the background, with obvious jaggedness and splicing marks, and needs to be smoothed and blended in the next step to improve the visual naturalness.

[0079] S402. A Gaussian blur operator is used to smooth the binary segmentation mask, generating a continuous soft mask with gradual transitions at the edges. Then, based on the soft mask, the original rendered image and the edited image are weighted and fused pixel by pixel to obtain the final constrained edited image. The specific calculation method is as follows:

[0080]

[0081] in This represents a Gaussian blur operation, used to convert a discrete binary mask containing only 0 and 1 values ​​into a continuous soft mask with values ​​ranging from [0,1]. The width of the edge transition band can be controlled by adjusting the size and standard deviation of the Gaussian kernel to suit different editing needs. This represents a pixel-by-pixel element-wise multiplication operation; To constrain the output image after fusion.

[0082] The effect of this fusion mechanism is as follows: the soft mask value inside the target area is close to 1, the edited image dominates, and the editing effect is fully preserved; the soft mask value of the background area is close to 0, the original rendered image dominates, and the original appearance remains unchanged; the soft mask at the boundary of the target area smoothly transitions between 0 and 1, realizing a natural connection between the edited area and the background area, eliminating jagged edges and splicing marks caused by hard mask cutting, and improving the visual naturalness of the local editing results.

[0083] Step S5: Iterate and optimize to generate the final edit scene.

[0084] This step is the closed-loop optimization and output stage of 3D scene editing. It transforms the 2D constraint editing results output in step S4 into parameter updates for a 3D Gaussian scene. Through multiple rounds of iterative "rendering-editing-constraint-optimization," the 2D editing effect is gradually solidified into the 3D representation. This addresses the technical issues of insufficient single-round update intensity and inadequate convergence of 3D spatial cross-view consistency, ultimately resulting in a target 3D scene with accurate editing effects and consistent performance across all views. This step employs a joint loss function to ensure optimization accuracy and introduces an early stopping mechanism to reduce invalid iterations, balancing editing quality and operational efficiency. Specifically, it is implemented through the following sub-steps:

[0085] S501. Using the constrained edited image output in step S4 as the supervised ground truth, perform a test on the current 3D Gaussian scene. The parameters are iteratively optimized, and early stopping judgment is performed based on the semantic consistency index. The specific implementation process is as follows:

[0086] A joint loss function combining pixel-level L1 loss and perceptual loss is used as the supervision for parameter optimization. The expression of the loss function is as follows:

[0087]

[0088] in, For the current number The viewpoint image obtained from a 3D Gaussian scene rendering step. Edit the target image after applying constraints to the corresponding viewpoint; The average absolute error loss per pixel is used to constrain the numerical consistency of the rendering result with the target image in terms of color and structure, ensuring the stability and convergence of the optimization process. To detect similarity loss, the difference is calculated based on the high-level features extracted by the pre-trained deep visual network, which is used to improve the visual naturalness and realism of the editing results and avoid the problem of pixel values ​​matching but visual disharmony. , These are preset weighting coefficients used to balance the contribution ratios of the two types of losses, and can be adapted and adjusted according to the scenario type and editing needs.

[0089] Based on a 3D Gaussian sputtering differentiable rendering pipeline, the gradient of the joint loss is backpropagated to the optimizable parameters of each 3D Gaussian element, including 3D spatial coordinates, 3D covariance matrix, spherical harmonic color coefficients, and opacity. The Gaussian element parameters are then updated using a gradient descent algorithm. Since the constrained image in non-target regions is identical to the original rendered image, the loss gradient in the corresponding regions approaches zero. Therefore, only the 3D Gaussian elements within the target region undergo targeted updates, naturally achieving parameter isolation for local editing.

[0090] Using CLIP directional similarity index A semantic consistency check is performed on the editing effect of the current step. This metric quantifies the degree of semantic matching between the rendered image and the text editing instructions. If within a preset number of consecutive steps... If the metrics stop improving or even decline, it indicates that the semantic optimization potential of this round of editing has been fully realized. Continuing the iteration would easily introduce artifacts and yield no substantial benefits. In this case, the early stop mechanism is triggered, prematurely terminating the current round of parameter optimization and entering the next round of full-process iteration. This mechanism can effectively reduce invalid calculations and improve overall editing efficiency.

[0091] S502. Based on the single-round optimized 3D Gaussian scene, perform multiple rounds of full-process iterative optimization to ensure that the editing effect fully converges in 3D space. The specific process is as follows:

[0092] After each round of parameter optimization, a new round of multi-view images is obtained by re-rendering based on the updated 3D Gaussian scene. The entire process of multi-view consistency editing in step S3, target region constraint fusion in step S4, scene parameter update and effect detection in this step is repeated sequentially. Through multiple iterations, the editing effect of the target region is gradually enhanced, the deviation of 3D geometry and appearance is continuously corrected, the editing differences between different perspectives are reduced, and the stability and consistency of the editing effect are ensured in all space and all perspectives.

[0093] The iteration stops when the preset total number of iterations is reached, or when the error between the rendered result and the editing target converges to a preset threshold, and the optimized target 3D Gaussian scene is output. The editing effect of this scene is embedded in the parameters of the 3D Gaussian elements, supporting real-time rendering from any viewing angle, ultimately achieving efficient 3D scene editing with consistent multi-viewpoints and local controllability.

[0094] Based on the updated 3D Gaussian scene from the previous iteration, the multi-view image corresponding to the next iteration is re-rendered, and the process of multi-view consistency editing, scene parameter updating, and editing effect detection is repeated until the preset number of iterations is reached, and the edited target 3D scene is output.

[0095] In summary, the 3D scene editing method based on multi-view consistency constraints and interactive segmentation described in this embodiment constructs a training-free 3D scene editing framework. Combined with... Figure 2 As shown, the framework mainly consists of two functional modules: an interactive 3D segmentation module and a batch editing module. The functions and technical effects of each module are as follows:

[0096] The interactive 3D segmentation module receives interactive prompts from the user at a reference viewpoint. Combining the scene depth map with the camera projection geometry, it upscales the 2D interactive prompts to 3D space and propagates them to all viewpoints, generating a consistent 2D segmentation mask for the target region across all viewpoints. This mask is then further mapped to a 3D Gaussian-granular mask for the region to be edited. This module enables accurate positioning and spatial boundary constraints of the target region, providing a spatial basis for controlled local editing, effectively reducing erroneous modifications to non-target areas, and improving the accuracy and visual naturalness of the editing results.

[0097] The batch editing module performs instruction-driven batch consistency editing on multi-view scene rendered images, using the constrained editing results as a supervision signal to iteratively update the parameters of the 3D Gaussian scene. In terms of editing efficiency, this module reduces peak memory usage for single-step inference through block-based batch processing, reduces redundant computations during diffusion inference through encoder caching and parallel decoding strategies, and eliminates invalid iteration steps using early stopping optimization strategies, thus improving the overall efficiency of 3D scene editing. Regarding view consistency, this module achieves collaborative constraints and semantic alignment of editing features across different views through a two-level view token merging mechanism (local and global), effectively improving the inconsistency of target color, texture, and structure under different views and enhancing the cross-view consistency of editing results.

[0098] The framework described above does not require additional model training or fine-tuning for specific scenarios. Under limited computing resources, it can simultaneously ensure the operational efficiency of 3D scene editing, cross-view consistency, and local editing accuracy, while reducing the probability of editing artifacts. It is applicable to technical fields such as 3D content generation, virtual reality, and digital media production.

[0099] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described three-dimensional scene editing method.

[0100] A computer-readable storage medium storing a computer program, which, when executed by a processor, implements the aforementioned three-dimensional scene editing method.

[0101] The above description is merely a preferred embodiment of the present invention, and while it is quite specific and detailed, it should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention.

Claims

1. A 3D scene editing method based on multi-view consistency constraints and interactive segmentation, characterized in that, The method includes the following steps: Step S1: Reconstruct the initial 3D Gaussian scene: Acquire multi-view image data of the target scene, perform motion recovery structure processing on the multi-view image data to obtain the camera intrinsic parameters, extrinsic parameters and sparse point cloud of the scene corresponding to each view, and initialize the 3D Gaussian scene representation based on the obtained sparse point cloud. Step S2: Locate and segment the area to be edited: Based on the interactive point prompts and depth information from the reference viewpoint, the two-dimensional prompts are back-projected into the three-dimensional space to obtain the three-dimensional prompts of the target area; Then project it onto other perspectives to generate multi-view two-dimensional guidance prompts. Combine the segmentation model to obtain the target region mask for each perspective, and map it back to the three-dimensional Gaussian scene to obtain the three-dimensional Gaussian mask of the region to be edited. Step S3: Perform multi-view consistent editing: Input the original images from multiple views into the image editing model in batches for encoding. Based on the feature similarity between different views, perform local feature merging and global feature merging. In the diffusion model inference process, the encoder caching and parallel decoding strategy is adopted to finally output the multi-view edited result image. Step S4, Target Region Constraint Fusion: Based on the target region mask, perform region constraints and fusion on the obtained multi-view editing result image to achieve target region editing and non-target region preservation; Step S5: Iterative optimization and generation of the final edited scene: Using the constrained editing results as a supervision signal, the parameters of the 3D Gaussian scene representation are updated, and the 3D Gaussian scene representation is rendered, edited and updated in multiple rounds until the preset number of iterations is reached or the stopping condition is met, and the edited target 3D scene is output to achieve efficient 3D scene editing with consistent multi-view perspectives and local controllability.

2. The 3D scene editing method based on multi-view consistency constraints and interactive segmentation according to claim 1, characterized in that, Step S1 includes: S101. Capture the target scene with a camera to obtain video data of the corresponding scene, and perform frame-by-frame processing on the video data to extract a multi-view image sequence. S102. Perform motion recovery structure reconstruction on the multi-view image sequence to obtain the camera intrinsic parameters, camera extrinsic parameters and sparse point cloud of the target scene corresponding to each view. S103. Using 3D Gaussian sputtering technology, the obtained sparse point cloud is initialized with multiple 3D Gaussian elements. Then, the parameters of all 3D Gaussian elements are iteratively optimized through the Gaussian sputtering differentiable rendering pipeline, finally obtaining the optimized initial 3D Gaussian scene. .

3. The 3D scene editing method based on multi-view consistency constraints and interactive segmentation according to claim 1, characterized in that, The implementation process of step S2 includes: S201, Based on the intrinsic parameter matrix shared by all views And the transformation matrix from the camera coordinate system to the world coordinate system corresponding to the reference viewpoint. The initial 3D Gaussian scene is rendered using the 3D Gaussian sputtering differentiable rendering pipeline. Rendering is performed to obtain a color rendered image corresponding to the reference viewpoint. and pixel-by-pixel depth map ; in the image The input interactive prompt information includes positive prompts marked in the target area to be edited, and negative prompts marked in the non-target area to be excluded; S202. Obtain the two-dimensional pixel coordinates corresponding to the interactive prompt point. and its depth map The corresponding depth value The 3D cue points in the world coordinate system are obtained through camera geometric inverse projection calculation: in, The coordinates of a 3D point in the camera coordinate system. The homogeneous world coordinates corresponding to the interactive prompt point; S203, Based on the perspectives of each target The corresponding world coordinate system to camera coordinate system transformation matrix , 3D prompt points Projecting the image plane onto the target viewpoint generates corresponding two-dimensional guidance prompts. The foreground and background guidance prompts of each viewpoint are input into the image segmentation mode of the interactive segmentation model SAM2 to obtain the binary mask of the target region for each viewpoint. The two-dimensional segmentation mask of each viewpoint is reverse-mapped to the three-dimensional Gaussian scene. The proportion of each three-dimensional Gaussian element that is determined to be the target region in all visible viewpoints is counted. When the proportion is higher than the preset proportion threshold, the three-dimensional Gaussian element is marked as the target Gaussian element to be edited, and finally the three-dimensional Gaussian mask of the region to be edited is obtained.

4. The 3D scene editing method based on multi-view consistency constraints and interactive segmentation according to claim 1, characterized in that, The implementation process of step S3 includes: S301. Render the multi-view image to be edited. The images are sorted according to the pose spatial adjacency of the corresponding cameras to form a batch view sequence with continuous view transitions and high scene overlap. The sorted batch view sequence is then input into the Instruct-Pix2Pix instruction-driven diffusion image editing model. The model's built-in VAE encoder performs dimensionality reduction encoding on the images of each view to obtain the corresponding latent space feature tensor. ,in This represents the total number of views included in the batch. This represents the total number of latent space feature tokens corresponding to a single image. The channel dimension is a feature of the latent space; S302, Encoded multi-view latent space feature tensor Divide the data into consecutive blocks according to the viewpoint order to obtain the desired results. Feature blocks , of which Feature blocks , The number of views contained in this chunk, and the total number of views across all chunks satisfy the following condition: ; S303. In the multi-step denoising inference process of the diffusion model, an encoder caching and parallel decoding strategy is adopted: the encoder output features are recalculated only in the preset key denoising time steps, and the cached encoder features are directly reused in non-key denoising time steps; the decoding process of multiple perspectives within the same block is parallelized to reduce the amount of repeated calculations and improve inference efficiency. S304. Perform local feature merging on the multi-view feature tokens within a single block, specifically: merge the latent space features of the current block. Split into anchor set Source set The number of views corresponding to both satisfy Flatten the view dimension and spatial dimension of the anchor set and source set, and calculate the cosine similarity matrix between all source tokens and anchor tokens; match the anchor token with the highest similarity for each source token, and select the top [anchor tokens with the highest similarity]. Each source token is merged into its corresponding anchor token, and the remaining source tokens are retained; among them , This is the token merging ratio coefficient. ; S305. Introduce a global token library to store representative feature tokens of historical blocks, and perform global feature merging on the block features after local merging: process the first... When there are blocks, use a preset probability The local block features are randomly swapped with the source / anchor roles of the global token store. Cross-block token merging is then performed according to the local feature merging rules to obtain the globally merged block features. Add unmerged source tokens to the global token library, replace historical tokens with lower representativeness in the library, and update the global token library. S306. After the self-attention calculation is completed, according to the token matching mapping relationship recorded in the merging stage, the compressed feature tokens are de-merged and restored to the token size at the time of input. The de-merged features are split into global tokens and local tokens. The global tokens are stored in the global token library for subsequent block segmentation, and the local tokens are restored to the view size of the current block and used as the input of the subsequent cross-attention layer. S307. For the latent space feature tokens that have completed all diffusion denoising steps, an upsampling decoding operation is performed using a VAE decoder to map the low-dimensional latent space features back to the RGB pixel space, generating multi-view editing result images corresponding to each viewpoint. .

5. The 3D scene editing method based on multi-view consistency constraints and interactive segmentation according to claim 1, characterized in that, The implementation process of step S4 includes: S401. Using the binary segmentation mask of the target region from each viewpoint generated in step S402. As a spatial template, edit the resulting image from multiple perspectives. Apply pixel-level region constraints: for target region pixels with a mask value of 1, the corresponding pixel values ​​in the edited image are preserved to ensure the editing effect takes effect; for non-target region pixels with a mask value of 0, the original rendered image is used. Replace the corresponding pixel values ​​to shield the background area from unwanted interference from diffusion editing; S402. Applying Gaussian blurring operator to the binary segmentation mask. A smoothing process is performed to obtain a continuous soft mask with values ​​ranging from [0,1]. Based on the soft mask, the original rendered image and the edited image are weighted and fused pixel by pixel to obtain the constrained edited image. The fusion formula is: in Indicates the Gaussian blur operation; This represents a pixel-by-pixel element-wise multiplication operation.

6. The 3D scene editing method based on multi-view consistency constraints and interactive segmentation according to claim 1, characterized in that, The implementation process of step S5 includes: S501. Using the constrained edited image output in step S4 as the supervised ground truth, perform a test on the current 3D Gaussian scene. The parameters are iteratively optimized, and early stopping detection of editing effects is performed: a pixel-level L1 loss and perception are constructed. The joint loss function of the loss combination is expressed as follows: in, For the current number The viewpoint image obtained from a 3D Gaussian scene rendering step. Edit the target image after applying constraints to the corresponding viewpoint; This is the average absolute error loss per pixel; To perceive similarity loss; , These are preset weighting coefficients; Based on a 3D Gaussian sputtering differentiable rendering pipeline, the gradient of the joint loss is backpropagated to the optimizable parameters of each 3D Gaussian element, and the parameters are updated using a gradient descent algorithm; CLIP directional similarity index is employed. Perform semantic consistency detection on the editing effect of the current step. If the indicator no longer improves or decreases within a preset number of consecutive steps, trigger the early stop mechanism to terminate the current round of parameter optimization in advance. S502. After each round of parameter optimization, a new round of multi-view images is obtained by re-rendering based on the updated 3D Gaussian scene. The multi-view consistency editing of step S3, the target region constraint fusion of step S4, and the scene parameter update and effect detection of this step are repeated in sequence. When the number of iterations reaches the preset total number of iterations, or the error between the rendering result and the editing target converges to the preset threshold, the iteration stops and the edited target 3D Gaussian scene is output.

7. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the three-dimensional scene editing method as described in any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the three-dimensional scene editing method as described in any one of claims 1 to 6.