A sketch-guided three-dimensional gaussian splash scene image controllable editing method and device

Abstract

The application relates to a sketch-guided controllable editing method and device for a three-dimensional Gaussian splash scene image, which comprises the following steps: receiving a two-dimensional sketch contour drawn by a user from at least two reference visual angles; taking the depth of each collected visual angle image pixel point by using a three-dimensional Gaussian splash model and performing reverse projection to obtain three-dimensional space coordinate points; determining a sampling direction and generating a dense three-dimensional sampling point set according to the three-dimensional space coordinate points; re-projecting the sampling point set to a reference visual angle image coordinate system, matching the sampling point set with the user sketch contour, and determining a target three-dimensional editing space region; generating enhanced guide information and inputting the enhanced guide information into an image generation model, processing the enhanced guide information through a cross-visual angle consistency control mechanism, and obtaining a multi-visual angle consistent editing image; initializing a new three-dimensional Gaussian primitive in the target region, updating attribute parameters of the new three-dimensional Gaussian primitive under the supervision of the editing image, and obtaining an edited three-dimensional model. The application realizes accurate positioning and contour control at a pixel level, solves the multi-visual angle consistency problem, has high editing efficiency, and is suitable for high-quality three-dimensional geometric editing.

Description

Technical Field

[0001] This invention relates to the fields of computer graphics and 3D image processing technology, and in particular to a method and apparatus for controllable editing of 3D Gaussian splash scene images based on sketch guidance. Background Technology

[0002] With the rapid development of 3D reconstruction and generation technologies, the cost of acquiring high-quality 3D digital assets has been significantly reduced. However, reconstructed or generated 3D scenes and objects often require further fine-tuning and artistic processing by humans. Therefore, how to provide users with an intuitive and highly controllable 3D editing method, making 3D editing as precise and convenient as drawing on 2D images, has become a crucial technical problem that urgently needs to be solved in the field of 3D image processing.

[0003] Although existing methods have made some progress, based on their technical approaches, they mainly suffer from the following types of insurmountable technical defects: First, text prompts lack sufficient control. Most existing 3D editing methods rely on text prompts as the input method for editing commands. However, textual language has inherent limitations in its expressive power regarding the specific shape, outline details, and precise spatial position of objects, making it difficult to achieve fine-grained control over the geometry of a 3D scene. For example, when adding a rose to a 3D scene, text cannot accurately describe the degree of petal folding and spatial orientation.

[0004] Second, the positioning accuracy of the 3D editing area is insufficient. Existing methods typically use 3D bounding boxes to specify the editing area, but 3D bounding boxes can only provide a coarse cubic region positioning and cannot accurately describe the outline shape of the area to be edited. More seriously, 3D bounding boxes cannot locate spatial regions in the scene that did not originally exist, which fundamentally limits the operation of adding new 3D objects to the scene.

[0005] Third, editing methods based on implicit 3D representations (such as Neural Radiation Fields, NeRF) are inefficient and of questionable quality. The implicit representation characteristics of Neural Radiation Fields make direct manipulation of scene geometry extremely difficult, resulting in high computational overhead, slow convergence speed, and blurred object outlines and significant loss of geometric details in the editing results.

[0006] Fourth, the score distillation sampling (SDS) guided method introduces distortion. When utilizing a pre-trained diffusion model, this method ignores its inherent generation inconsistencies and low robustness, which can easily lead to texture distortion and overly smoothed image quality defects in the generated 3D assets.

[0007] Fifth, the consistency of multi-view image editing is poor. Pre-trained 2D image generation models only possess 2D image knowledge and can only provide image supervision signals from a single viewpoint. When used for multi-view 3D editing, the images generated from each viewpoint exhibit severe inconsistencies in texture, semantics, and geometry, directly leading to contradictory artifacts in the final 3D editing result and low editing quality.

[0008] Sixth, existing sketch-assisted methods have limited control. Although existing sketch-assisted 3D editing methods attempt to use sketches to specify 3D editing areas, they only apply fuzzy constraints to the editing area through loss function constraints. The actual outline and shape of the added objects are still mainly determined by text prompts, and the sketch has extremely limited control over the shape of the editing result.

[0009] In summary, existing technologies have not yet effectively resolved the contradiction between sketch-guided precise spatial positioning and consistent high-quality editing across multiple perspectives.

[0010] It should be noted that the information disclosed in the background section above is only for understanding the background of this application, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0011] The technical problem this application aims to solve is "how to address the difficulty in conveying user intent (especially precise geometric shape and spatial location intent) to the model through natural language and simple interaction methods in existing 3D editing methods, and ultimately reflecting it as a high-quality, multi-view consistent editing result." To this end, this application provides a sketch-guided, controllable editing method and apparatus for 3D Gaussian splash scene images.

[0012] The technical solution adopted in this application to solve the above-mentioned technical problems is as follows.

[0013] This application provides a sketch-guided method for controllable editing of 3D Gaussian splash scene images, applied to the geometric image editing of 3D Gaussian splash models, including the following steps: S1. Steps for acquiring 3D scene image data: Acquire the 3D Gaussian splash model data of the 3D scene to be edited. The 3D Gaussian splash model data includes the position, covariance, opacity and color attribute parameters of multiple 3D Gaussian primitives in the scene; acquire the multi-view acquired images and corresponding camera calibration parameters obtained by multi-view image acquisition of the 3D scene. The camera calibration parameters include the intrinsic parameter matrix and extrinsic parameter matrix of each view camera. S2. Sketch Interactive Input Steps: Receive two-dimensional sketch outlines drawn by the user from at least two different viewpoints through the image interactive interface, and convert the two-dimensional sketch outlines into two-dimensional reference mask images of the corresponding reference viewpoints. S3. Steps for determining the 3D editing region: For each pixel in the image corresponding to each acquisition viewpoint, the depth value corresponding to the pixel is obtained using the depth map rendered by the 3D Gaussian splash model. Based on the intrinsic and extrinsic parameter matrices in the corresponding camera calibration parameters, the 2D pixel is back-projected into the 3D scene space to obtain its corresponding 3D spatial coordinates. For each back-projected 3D spatial coordinate point, its normalized observation direction vector relative to the camera optical center is calculated. A dense 3D sampling point set is uniformly generated along the normalized observation direction vector within the depth range at a preset sampling density. The dense 3D sampling point set is projected onto the image plane of each reference viewpoint camera and matched with the 2D reference mask image under the corresponding reference viewpoint to determine the target 3D editing space region and the transformation mask image corresponding to each viewpoint. S4. Mask image denoising and edge enhancement steps: Perform noise point removal and edge information enhancement processing based on convolution filtering on the transformed mask image obtained in step S3 to obtain an enhanced mask image, and extract edge feature maps from the enhanced mask image as guiding information; S5. Multi-view consistent image editing steps: Input the edge feature map and text description information into the pre-trained conditional diffusion generation model, and perform joint denoising generation processing on the multi-view image through the cross-view attention control mechanism to obtain a multi-view consistent edited image; S6. 3D Model Optimization and Reconstruction Steps: Initialize new 3D Gaussian primitives within the target 3D editing space region. Use the edited image with consistent multi-view perspectives as the supervision target signal. By minimizing the combined loss function between the rendered image and the edited image, the attribute parameters of the 3D Gaussian primitives are iteratively optimized using the stochastic gradient descent algorithm to obtain the edited 3D scene model.

[0014] In some embodiments, step S3 specifically includes: S31. 3D spatial back projection of image pixels: For each pixel in the image corresponding to each acquisition viewpoint, the depth value corresponding to the pixel is obtained by using the depth map rendered by the 3D Gaussian splash model. Based on the intrinsic and extrinsic parameter matrices of the camera at the acquisition viewpoint, the 2D pixel is back projected onto the 3D scene space to obtain its corresponding 3D spatial coordinates. S32. Dense 3D point set sampling along the observation direction: For each 3D spatial coordinate point obtained by back projection, calculate its normalized observation direction vector relative to the camera optical center, and uniformly generate a dense 3D sampling point set along the normalized observation direction vector within the depth range at a preset sampling density N. S33. Reprojection of dense 3D points to reference viewpoint: Project the dense 3D sampling point set onto the image plane of each reference viewpoint camera to obtain the 2D projection coordinates on the reference viewpoint; S34. Three-dimensional region determination based on multi-view mask matching: For each pixel in the image corresponding to the acquisition viewpoint, determine whether there is at least one three-dimensional sampling point in the dense three-dimensional sampling point set associated with it whose projection falls within the corresponding two-dimensional reference mask area in all reference viewpoints. If the condition is met, the pixel is determined to be within the projection range of the target three-dimensional editing space. The three-dimensional sampling points that meet the condition constitute the geometric representation of the target three-dimensional editing space. S35. Based on the determination result of step S34, generate a transformation mask image corresponding to each acquisition viewpoint. The transformation mask image is used to identify all pixels that are determined to be located within the projection range of the target 3D editing space under that acquisition viewpoint.

[0015] In some embodiments, in step S32, the preset sampling density N is set to 500, and the depth offset is set to 0.1; in step S5, the multi-view editing uses 48 uniform sampling views, and the image resolution of each view is 512×512 pixels; in step S6, the weight coefficients of the combined loss function are set to L1 loss weight of 100, perceptual loss weight of 1, and anchor loss weight of 5, and the maximum number of optimization iterations is set to 7500; in the denoising process, the convolution kernel size is 5×5 and the counting threshold Tc is 9, and in the edge enhancement process, the convolution kernel size is 3×3 and the threshold is 8; the inference step range of cross-view attention control is from step 20 to step 50, and the range of U-Net attention control layers is from layer 10 to layer 15.

[0016] In some embodiments, the noise reduction process includes: The transformation mask image tensor is convolved using a k×k convolution kernel. The number of pixels in the k×k local neighborhood of each pixel that are equal to the preset mask label value T is counted. Pixels whose count exceeds the counting threshold Tc are determined to be valid mask pixels, and their values ​​are set to the preset mask label value T. The remaining pixels remain unchanged. Edge enhancement processing includes: creating a color mask M, setting the pixel position of color T to 1, performing a convolution operation on the color mask M using a convolution kernel of size k×k, normalizing and thresholding the convolution result, and generating a binarized enhancement mask B as the enhancement mask image.

[0017] In some embodiments, the cross-view attention control mechanism in step S5 specifically includes: (a) Cross-view unified attention calculation: In the inference denoising process of the conditional diffusion generative model, image features from different perspectives are treated as a unified whole for cross-view attention calculation. The feature representation of the source perspective image is used as the key and value, and the feature representation of the target perspective image is used as the query. Cross-view information interaction is achieved through attention mapping. (b) Enhanced batch consistency guided by reference images: A pre-selected reference edit image is inserted at the first input position of each processing batch, and the reference edit image serves as a global guiding signal in the attention calculation of that batch.

[0018] In some embodiments, step S6 specifically includes: S61. Region constraint initialization: Randomly sample several points from the three-dimensional point set of the target three-dimensional editing space region as the position coordinates of the new three-dimensional Gaussian primitives. The other attributes of each new three-dimensional Gaussian primitive, except for the position, are initialized by randomly sampling from the original three-dimensional Gaussian primitives and reusing their attribute values. S62. Iterative optimization: A combined loss function is used to measure the difference between the current 3D model rendered image and the target edited image. The combined loss function includes pixel-level L1 loss, perceptual loss based on a pre-trained visual feature network, and anchor point loss. The attribute parameters are iteratively updated using a stochastic gradient descent algorithm. S63. Periodic Reset: During the optimization iteration process, the current position of all new 3D Gaussian primitives is detected every preset number of steps, and 3D Gaussian primitives that exceed the boundary of the target 3D editing space region are randomly reset to the inside of the region; S64. Adaptive Density Adjustment and Pruning: During the optimization process, the distribution density of 3D Gaussian primitives in the current editing area is periodically evaluated. New 3D Gaussian primitives are added to areas with insufficient density, while 3D Gaussian primitives that contribute little to the rendering result are removed.

[0019] In some embodiments, the multi-view consistency image editing step in step S5 further includes: After editing, the mask region of the multi-view consistent edited image output by the conditional diffusion generation model is extracted and superimposed on the original multi-view acquired image to complete the local image editing fusion. The enhanced mask image generated in step S4 is used to distinguish between the edited area and the non-edited area, thereby ensuring that the edited content only applies to the area specified by the two-dimensional sketch outline and does not affect other parts of the multi-view acquired image.

[0020] In some embodiments, a sketch-guided controllable editing device for a 3D Gaussian splash scene image is also provided, comprising: The 3D scene data acquisition module is used to acquire the 3D Gaussian splash model data, multi-view acquired images, and camera calibration parameters of the 3D scene to be edited; The sketch interactive input module is used to receive two-dimensional sketch outlines drawn by the user from at least two different perspectives through the image interactive interface and convert them into reference mask images; The 3D editing region determination module is used to perform 3D spatial back-projection, dense 3D point set sampling and reference viewpoint reprojection on pixels according to camera calibration parameters, and determine the target 3D editing space region and the transformation mask of each viewpoint through multi-view mask matching. The mask image preprocessing module is used to perform noise point removal and edge information enhancement processing on the transformed mask image based on convolution filtering; The multi-view consistent editing module, including an edge feature extraction submodule and a cross-view attention control submodule, is used to generate multi-view consistent edited images through a pre-trained conditional diffusion generation model. The 3D model optimization and reconstruction module is used to initialize 3D Gaussian primitives within the 3D editing area and reconstruct the edited 3D scene model through iterative optimization based on multi-view image supervision.

[0021] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method of this application.

[0022] In some embodiments, an electronic device is also provided, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method of this application. The processor includes a graphics processing unit (GPU), and the rendering and optimization process of the three-dimensional Gaussian splash model is accelerated by parallel computing through the GPU.

[0023] The present invention has the following beneficial effects: This invention enables high-quality, high-precision, and multi-viewpoint consistent editing of 3D scenes. Its advantages are as follows: By receiving 2D sketch outlines drawn by the user from at least two different perspectives, this invention intuitively and accurately transforms the user's intent regarding precise geometric shapes and spatial positions into a machine-processable 2D reference mask image, laying a high-precision input foundation for subsequent processing. Through a series of operations, including backprojection using a depth map rendered by a 3D Gaussian splash model, dense sampling along the viewing direction vector, and reprojection to the reference viewpoint for matching with the 2D reference mask image, the 2D sketch information is precisely mapped to 3D space, defining the target 3D editing space region and achieving pixel-level precise positioning from user intent to 3D space. The "mask image denoising and edge enhancement step" performs convolutional filtering-based noise removal and edge information enhancement on the transformed mask image, improving the quality of the mask image and providing clear and accurate regional constraints for subsequent image generation, thereby ensuring the accuracy of the generated content outline. By inputting extracted edge feature maps and textual descriptions into a pre-trained conditional diffusion generation model and employing a cross-view attention control mechanism for joint denoising, this approach ensures that the generated edited images maintain high consistency in texture and semantics across different viewpoints, effectively eliminating multi-view contradictory artifacts. Furthermore, by initializing new 3D Gaussian primitives within the target 3D editing space and using the multi-view consistent edited image as the supervised target for iterative optimization based on a combined loss function, a high-quality 3D scene model is ultimately reconstructed that conforms to the user's sketch-defined geometry and spatial location while maintaining multi-view consistency. In summary, this application enables users to convey complex editing intentions through intuitive and precise sketch interactions, ultimately obtaining high-quality, multi-view consistent 3D editing results. It successfully solves the technical challenge in existing 3D editing methods where user intentions (especially precise geometric shape and spatial location intentions) are difficult to convey to the model through natural language and simple interaction methods, ultimately resulting in high-quality, multi-view consistent editing results.

[0024] It should be noted that the systematic solution to the aforementioned technical problems does not rely on the independent function of any single technical feature, but rather on the deep synergy and sequential coordination between each step. Specifically: The "Sketch Interaction Input Step" is the starting point for capturing and transforming the user's precise intent. Without this step, all subsequent processing will lose its high-precision input basis and target. The "3D Editing Region Determination Step" is the core bridge that accurately links the 2D user intent with 3D space. Its series of operations are indispensable, jointly ensuring the accuracy of 3D spatial positioning. Without any step such as dense sampling or reprojection matching, the positioning accuracy will drop significantly. The "Mask Image Denoising and Edge Enhancement Step" optimizes the quality of the output result of the core bridge, ensuring the input quality of subsequent generation steps. Without this step, noise and blurred edges will interfere with the generated model, affecting the accuracy of the final shape. The "Multi-View Consistency Image Editing Step" is one of the keys to solving the multi-view consistency problem. Its "Cross-View Attention Control Mechanism" is an important means to achieve this. If there is only multi-view input without this mechanism, the consistency of the output cannot be guaranteed. The "3D model optimization and reconstruction step" is the final process of solidifying the results of the preceding steps into a high-quality 3D model. Its constraint of "initializing new 3D Gaussian elements within the target 3D editing space" ensures that editing only applies to the user-specified area, while the optimization of "using consistent edited images from multiple perspectives as supervisory target signals" drives the model to learn consistent editing content. Therefore, these various technical features are interdependent and work closely together to form a complete solution. Through the synergistic cooperation of these features, the technical challenges of accurately conveying user intent and generating high-quality, consistent multi-view editing results are effectively solved.

[0025] Other beneficial effects of the present invention will be further described below. Attached Figure Description

[0026] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein: Figure 1 This is a schematic diagram of the overall processing flow of the method of the present invention; Figure 2 This is a schematic diagram of the sketch-guided mask transformation process, including sub-steps such as back projection, dense sampling, reprojection, and mask matching. Figure 3 This is a visualization diagram of two-dimensional image editing and processing, where (a) is the result of mask transformation, (b) is the result of noise point removal, (c) is the result of edge enhancement, (d) is the result of edge extraction, and (e) is the result of local editing and fusion. Figure 4 This is a schematic diagram of the cross-perspective attention control mechanism.

[0027] Figure 5 Pseudocode flowchart for 3D Gaussian primitive initialization and optimization; Figure 6 The figure shows the qualitative comparison results with existing methods GaussianEditor and Instruct-NeRF2NeRF; Figure 7 Demonstration of flexible editing effects under different sketch inputs; Figure 8 This is a comparison chart of ablation experiment results. Detailed Implementation

[0028] The embodiments of the present invention will be described in detail below. It should be emphasized that the following description is merely exemplary and not intended to limit the scope and application of the present invention.

[0029] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of embodiments of the present invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0030] The embodiments of this application will be further described below with reference to the accompanying drawings.

[0031] This invention proposes a sketch-guided method for controllable editing of 3D Gaussian splash scene images.

[0032] In some embodiments, the method is applied to geometric image editing of a three-dimensional Gaussian splash model, such as... Figure 1 As shown, it includes the following steps: Step S1, 3D scene image data acquisition. Acquire the 3D Gaussian splash model data of the 3D scene to be edited. The model data includes the position, scale, rotation, opacity, and color attribute parameters of multiple 3D Gaussian primitives in the scene. Acquire multi-view acquired images of the 3D scene and the corresponding camera calibration parameters. The camera calibration parameters include the intrinsic and extrinsic parameter matrices of each camera viewpoint.

[0033] In some embodiments, the 3D Gaussian splatter model represents the 3D scene using an explicit set of 3D Gaussian primitives, each parameterized by a mean vector, covariance matrix, color, and opacity. For each pixel on the image plane, its color value is calculated by accumulating the alpha blend of all 3D Gaussian primitives along the direction of the observation ray after depth sorting. This explicit representation allows for direct addition, deletion, and attribute modification of 3D Gaussian primitives in the scene, providing a data foundation for subsequent precise geometric editing.

[0034] Step S2: Multi-view sketch interactive input and mask generation. Multi-view acquired images are presented to the user through an image interaction interface. Two-dimensional sketch outlines drawn by the user from at least two different viewpoints are received. These two-dimensional sketch outlines mark the target shape and position of the area to be edited on the image. The two-dimensional sketch outlines are converted into two-dimensional mask images of the corresponding viewpoints, serving as a two-dimensional reference mask for subsequent 3D editing area determination and 2D image editing (i.e., the original mask drawn by the user in step S2, also known as the initial editing mask).

[0035] Step S3: Determine the 3D editing area based on sketch-guided mask transformation. For example... Figure 2 As shown, based on the camera calibration parameters, the reference mask is automatically transformed from the sketch drawing viewpoint to any target viewpoint, while simultaneously determining the 3D spatial region to be edited.

[0036] In some embodiments, this step specifically includes the following sub-steps.

[0037] Sub-step S3-1: 3D spatial backprojection of image pixels. For each pixel in the acquired image captured by the target view camera, the depth value corresponding to that pixel is obtained using the depth map rendered by the 3D Gaussian splash model. Then, based on the intrinsic and extrinsic parameter matrices of the target view camera, the 2D pixel is backprojected into the 3D scene space to obtain its corresponding 3D spatial coordinates. The backprojection calculation formula is: , where d is the depth value corresponding to the pixel.

[0038] Sub-step S3-2 involves sampling a dense 3D point set along the observation direction. For each 3D spatial coordinate point obtained from back projection, its normalized observation direction vector relative to the camera's optical center is calculated. Along the normalized observation direction, a series of depth sampling values ​​are uniformly generated within a preset depth range at a preset sampling density N. Combined with the observation direction and the camera's optical center coordinates, a dense 3D sampling point set is constructed. This dense sampling strategy ensures sufficient sampling along the entire observation ray corresponding to the target viewpoint pixel, avoiding the omission of 3D regions due to depth estimation errors.

[0039] In some embodiments, the "preset depth range" is an interval based on the depth of the current backprojection point, rather than the depth range of the entire scene. Specifically, for pixels... back projection depth value The sampling range is ,in This is the preset depth offset. The value of is related to the scene scale and is usually set to 5%-10% of the scene depth range. N=500 points are sampled uniformly within this range to ensure that even if there is some error in the depth estimation, the actual surface location can still be covered.

[0040] Sub-step S3-3: Reprojection of dense 3D points to the reference viewpoint. Project the dense 3D sampled point set onto the image plane of the reference viewpoint camera used for each sketch, and obtain its 2D projected coordinates on the reference viewpoint. The projection formula is: .

[0041] Sub-steps S3-4 involve determining the 3D region based on multi-view mask matching. For each 2D pixel query point in the target viewpoint, it is determined whether at least one of its associated N dense 3D sampling points falls within the corresponding reference mask region in the projection of all reference viewpoints. If this condition is met, the 2D pixel query point is determined to be within the projection range of the target 3D editing region. Simultaneously, the 3D sampling points that meet the condition constitute the geometric representation of the target 3D editing region. Through these sub-steps, this invention not only completes the automatic transformation of the reference mask from the sketch drawing viewpoint to each target viewpoint, but also accurately determines the 3D spatial region to be edited in the form of a 3D point set during the transformation process, achieving a precise mapping from a 2D sketch to a 3D editing region.

[0042] In some embodiments, regarding the source and accuracy of the depth map from the three-dimensional spatial back projection, it should be noted that: In this method, the depth map is obtained based on a rendering pipeline using a 3D Gaussian splash model. Specifically, for each pixel in the image... Its depth value is calculated using the following formula:

[0043] in, To affect pixels The Gaussian set, For the first The depth value of each Gaussian element along the observed ray. The formula uses a weighted summation method to combine the depth contribution of each Gaussian pixel according to its opacity weight, resulting in a pixel-level desired depth value.

[0044] This depth calculation method essentially utilizes the alpha compositing mechanism in the 3DGS rendering pipeline to estimate surface depth. Although there may be some deviations at edges and in transparent areas, the depth accuracy is sufficient to support subsequent operations in the application scenarios of this method, for the following reasons:

[0045] (1) This method does not require absolutely accurate depth positioning of a single point. Instead, it constructs a set of candidate points in three-dimensional space by dense sampling (sampling density N=500) along the observation direction in step S3-2. Even if there is a deviation in the depth of a single pixel, the real surface position can still be covered by dense sampling within a preset range before and after the depth value.

[0046] (2) The multi-view mask matching mechanism in step S3-3 plays a "correction" role - only three-dimensional points that fall within the mask area under all sketch drawing views are retained. This multi-view consistency constraint effectively filters out erroneous points introduced by depth deviation.

[0047] (3) The mask denoising and enhancement process in step S4 further eliminates residual noise points, ensuring the accuracy of the final edited area.

[0048] Step S4 involves image denoising and edge enhancement using a transformation mask (i.e., the multi-view mask obtained after three-dimensional spatial mapping in step S3, also known as a mapping mask). Figure 3 As shown, since the mask transformation process in step S3 involves sampling and projection in three-dimensional space, the transformed multi-view mask image may contain noisy pixels due to floating-point precision and discrete sampling. To improve the quality of subsequent two-dimensional image editing, denoising and edge enhancement preprocessing are required for the transformed mask image.

[0049] In some embodiments, this step specifically includes the following sub-steps.

[0050] Sub-step S4-1: Noise removal from the mask image based on convolution filtering. Local neighborhood convolution analysis is performed on the transformed mask image tensor.

[0051] A k×k convolution kernel is used to convolve the tensor of the transform mask image. The number of pixels in the k×k local neighborhood of each pixel that equals a preset mask marker value T is counted. Pixels whose count exceeds a threshold Tc are identified as valid mask pixels, and their values ​​are set to the preset mask marker value T. The remaining pixels remain unchanged. This process utilizes the local statistical properties of convolution to effectively identify and remove isolated noise pixels in the transform mask, improving the spatial continuity of the mask image.

[0052] Sub-step S4-2 enhances mask edge information based on convolutional filtering. A mask is created, with pixels of color T set to 1. A k×k all-one convolution kernel is used to convolve the mask. The convolution result is normalized and thresholded to generate a binarized enhanced mask (i.e., the final mask after denoising and enhancement in step S4, also known as a refined mask). This process distinguishes between the internal and edge regions of the mask through convolution, enhancing the clarity and integrity of the mask edges, enabling subsequent edge feature extraction to obtain more accurate contour information.

[0053] Step S5: Multi-view consistency image editing based on cross-view attention control. A pre-trained fully convolutional edge detection network is used to extract precise edge feature maps from the masked image enhanced in Step S4. The edge feature maps and user-provided text descriptions are then fed into a pre-trained conditional diffusion generation model to perform image editing on the images acquired from each viewpoint.

[0054] In some embodiments, the pre-trained models involved in this method are all publicly available open-source models, including but not limited to: (1) Basic diffusion model: Stable Diffusion v1.5 (released by Stability AI), the infrastructure for image generation.

[0055] (2) Conditional control model: ControlNet-softedge (proposed by Lvmin Zhang et al.) is used for conditional guided generation based on edge maps. This model accepts soft edge maps as conditional inputs and guides the diffusion model to generate images consistent with edge structures.

[0056] (3) Edge detection model: PidiNet (Pixel Difference Network), used to extract soft edge feature maps from the input image, which serve as conditional inputs to ControlNet. All of the above are open-source models, not self-developed models.

[0057] In some embodiments, the text prompt is entered by the user during editing. The user input in this method consists of two parts: (1) a sketch drawn by the user from one or more perspectives, which specifies the editing area and target shape; and (2) a text prompt provided by the user, which specifies the semantic information of the edited content (such as "a red hat"). The sketch and text prompt work together, with the sketch controlling the spatial location and geometry of the edit, and the text prompt controlling the texture, color, and semantic content of the edit.

[0058] In some embodiments, to address the inconsistency in texture and semantics of multi-view editing results, this invention improves the attention calculation mechanism in the reasoning process of the conditional diffusion generative model, such as... Figure 4 As shown, a cross-perspective attention control mechanism is proposed, which includes the following two aspects.

[0059] Firstly, unified attention calculation across viewpoints is implemented. During the inference denoising process of the conditional diffusion generative model, image features from different viewpoints are no longer treated as independent batches for separate attention calculations. Instead, they are treated as a unified whole for cross-viewpoint attention calculation. The feature representations of the source viewpoint image are used as keys and values, and the feature representations of the target viewpoint image are used as queries. Cross-viewpoint information interaction is achieved through attention mapping. The attention calculation formula is: CrossViewAttention(Q, Ks, Vs) = Softmax(Q·Ks^T / √d) · Vs. Through this mechanism, the target viewpoint image can query and reference the texture and semantic information of the source viewpoint image during the generation process, ensuring that the editing results of multiple viewpoints within the same batch remain consistent in texture and semantics.

[0060] Secondly, reference image-guided batch consistency is enhanced. A pre-selected reference editing image is inserted at the first input position of each processing batch. The reference image serves as a global guiding signal in the attention calculation of that batch, enabling the editing results between different batches to maintain semantic and textural coherence.

[0061] After editing, the masked region of the generated image is extracted and superimposed onto the original acquired image, completing the local image editing. The binarized enhancement mask generated in step S4 is used to distinguish between the edited and non-edited areas, ensuring that the edited content only affects the area specified in the sketch and does not affect other parts of the original image. The image fusion formula is: , where ⊙ represents element-wise multiplication, and M_enhanced is the binarization enhancement mask.

[0062] In some embodiments, the cross-view attention control in step S5 is based on modifying the computational logic of the self-attention layer in the pre-trained Stable Diffusion model U-Net, rather than adding a new network layer. The specific implementation is as follows:

[0063] (1) Multi-view batch processing organization: Latent variables from multiple perspectives are concatenated along the batch dimension and then uniformly input into U-Net. Assume there are... From each perspective, the shape of the input tensor is... ,in For the number of channels, , The size of the feature map.

[0064] (2) Cross-view self-attention modification: In layers 10 to 15 of U-Net, in the original self-attention calculation, the query for each view ( The previous method only performed attention calculations with the Key and Value from its own perspective. This method modifies this so that the Key and Value from the source perspective (the perspective from which the user drew the sketch) are broadcast to the attention calculations of all perspectives.

[0065] in For the first A query from multiple perspectives and For the source's perspective, Key and Value The feature dimension of the key. This modification ensures that each viewpoint references the same source viewpoint features during the denoising process, thus achieving consistency in multi-view editing.

[0066] (3) Selection criteria for layer and step range: Layers 10 to 15 correspond to the lower resolution intermediate layers in U-Net. These layers capture global semantic information and structural layout, making them suitable for cross-view consistency constraints. Steps 20 to 50 correspond to the stage of generating the main structure and content during the denoising process. In the early steps (<20 steps), noise dominates, and cross-view constraints are not very meaningful; in the later steps (>40 steps), the content is basically determined, and applying constraints at this time may introduce artifacts instead.

[0067] (4) This modification does not destroy the generative capability of the pre-trained model because: a) the source of the Key / Value is replaced only within a specific layer and step range, while the remaining layers and steps maintain the original behavior; b) the Key / Value comes from the normal processing results of the same model from the source perspective, and the semantic space is consistent.

[0068] Step S6: Initialization and iterative optimization reconstruction of Gaussian elements within the 3D editing region. New 3D Gaussian elements are initialized within the 3D editing region determined in step S3. Using the multi-view editing image generated in step S5 as the supervised target signal, iterative optimization is performed to ensure that the newly added 3D Gaussian elements accurately represent the edited content, thus achieving geometric editing of the 3D model.

[0069] In some embodiments, such as Figure 5 As shown, the specific steps include the following.

[0070] Sub-step S6-1: Random initialization of the region constraints of the 3D Gaussian primitives. Several points are randomly sampled from the 3D point set of the 3D editing region determined in step S3, serving as the position coordinates of the new 3D Gaussian primitives. Each new Gaussian primitive's attribute set includes position, scale, rotation, and opacity parameters. Attributes other than position are initialized by randomly sampling from existing 3D Gaussian primitives and reusing their attribute values, with the scale parameter adjusted using a preset scaling factor. This initialization strategy ensures that the attribute values ​​of the new Gaussian primitives are distributed within a reasonable range, which is beneficial for rapid convergence in subsequent optimizations.

[0071] Sub-step S6-2 involves iterative optimization based on multi-view image supervision. A combined loss function is used to measure the difference between the current 3D model rendered image and the target edited image. The combined loss function includes three terms: pixel-level L1 loss, perceptual loss based on a pre-trained visual feature network, and anchor point loss. The loss function formula is: ,in I and J These are the rendered image and the target image, respectively. The attribute parameters are iteratively updated using the stochastic gradient descent algorithm. .

[0072] The definition and function of anchor loss are explained below: Anchor point loss is used to prevent 3D Gaussian primitives from deviating too far from their initial positions during optimization, ensuring that the scene structure outside the editing area is not accidentally altered. Its mathematical definition is:

[0073] in, For the current set of attributes of Gaussian elements, This is the set of position coordinates of the current Gaussian element. For editing area, For the first The current position of each Gaussian element. This is its initial position before optimization. The loss imposes an L2 norm constraint on all Gaussian elements located within the edit region, ensuring that they do not deviate too far from their initial positions during optimization.

[0074] In the combined loss function, the anchor loss, L1 pixel loss, and perceptual loss together constitute the optimization objective: in , , These are the weighting coefficients for each loss term. Weights of the anchor point loss. Set to a small value (e.g., 0.01) to ensure that the basic structural stability of the scene is maintained without excessively restricting the degree of freedom of optimization.

[0075] Sub-step S6-3 involves the periodic reset of Gaussian primitives outside the region. During the optimization iteration, some 3D Gaussian primitives may move outside the 3D editing region due to gradient updates. Every preset number of steps, it checks whether the current position of all new Gaussian primitives is still within the 3D editing region, and randomly resets Gaussian primitives that have exceeded the region boundary to valid positions inside the region. This periodic reset mechanism ensures that editing operations are strictly limited to the 3D region specified in the sketch and do not affect other parts of the original 3D scene.

[0076] In some embodiments, the periodic reset of the Gaussian element is described as follows: (1) The purpose of this reset strategy: During the optimization process of 3D Gaussian splashing, due to the cumulative effect of gradient updates, some Gaussian primitives in the editing area may gradually drift to the outside of the area. Without constraints, these "escaping" Gaussian primitives will produce unwanted visual artifacts outside the editing area, while reducing the number of Gaussian primitives in the editing area and affecting the rendering quality.

[0077] (2) The criterion for determining whether a Gaussian primitive is "out of the editing area" is: project the three-dimensional position of the Gaussian primitive onto the image plane of all relevant viewpoints and check whether it falls within the refined mask of that viewpoint. If the projected position falls outside the mask at any viewpoint, the Gaussian primitive is considered to have exceeded the editing area and needs to be reset.

[0078] (3) “Random reset” is not completely random: The reset operation repositions the Gaussian element that has run out of the edit area to a random legal position within the edit area, but retains its other attributes (such as color, opacity, scale, etc.) unchanged. In this way, the Gaussian element can start from the new position in subsequent optimization steps and be driven by the gradient to a more suitable position.

[0079] (4) This strategy complements the anchor loss: the anchor loss limits the movement of Gaussian elements through soft constraints (loss penalty), while periodic reset acts as a hard constraint to handle extreme drift. The two work together to ensure the stability of the optimization process. The reset operation is executed once every 100 iterations, which has little impact on the continuity of the optimization process.

[0080] (5) The effectiveness of the strategy was verified in the experiment: compared with the ablation experiment without using the reset strategy, the rendering quality (PSNR) of the edited area was improved by about 0.5-1dB after using the strategy, and no visible artifacts were introduced.

[0081] Sub-step S6-4: Adaptive density adjustment and pruning of Gaussian primitives. During optimization, density evaluation is performed periodically to assess the 3D Gaussian primitive distribution density of the current editing region. New Gaussian primitives are added to areas with insufficient density to increase detail representation, while Gaussian primitives that contribute less to the rendering result are removed, improving optimization efficiency and rendering quality.

[0082] In some embodiments, the evaluation criteria in adaptive density control are as follows: (1) Density evaluation: Based on the cumulative gradient of Gaussian elements in the image space. During the optimization process, the gradient magnitude contributed by each Gaussian element to the image pixel during rendering is recorded. When the average gradient magnitude of a Gaussian element exceeds a preset threshold... If the region is deemed to require a finer representation, a cloning or splitting operation is performed on the Gaussian unit: if the Gaussian unit is small in scale, a copy is cloned and its position is finely adjusted; if the scale is large, it is split into two smaller Gaussian units.

[0083] (2) Contribution assessment: based on the opacity of Gaussian elements. Opacity below a threshold... High-contribution Gaussian elements are considered insufficient for rendering and are pruned. This avoids a large number of low-contribution Gaussian elements consuming computational resources.

[0084] (3) The above mechanism adopts the adaptive density control strategy proposed in the original paper on 3D Gaussian Splatting (Kerbl et al., 2023) and applies it specifically within the editing region to ensure that the Gaussian Splatden density of the editing region can fully express the new editing content.

[0085] After the above iterative optimization until the preset maximum number of iterations is reached or the loss function converges to below the preset threshold, the edited 3D Gaussian splash model is obtained, which can then render a high-quality edited image that is consistent with the sketch outline from any perspective.

[0086] In some embodiments, a sketch-guided controllable editing device for a 3D Gaussian splash scene image is also provided, comprising: The 3D scene data acquisition module is used to acquire the 3D Gaussian splash model data, multi-view acquired images, and camera calibration parameters of the 3D scene to be edited; The sketch interactive input module is used to receive two-dimensional sketch outlines drawn by the user from at least two different perspectives through the image interactive interface and convert them into reference mask images; The 3D editing region determination module is used to perform 3D spatial back-projection, dense 3D point set sampling and reference viewpoint reprojection on pixels according to camera calibration parameters, and determine the target 3D editing space region and the transformation mask of each viewpoint through multi-view mask matching. The mask image preprocessing module is used to perform noise point removal and edge information enhancement processing on the transformed mask image based on convolution filtering; The multi-view consistent editing module, including an edge feature extraction submodule and a cross-view attention control submodule, is used to generate multi-view consistent edited images through a pre-trained conditional diffusion generation model. The 3D model optimization and reconstruction module is used to initialize 3D Gaussian primitives within the 3D editing area and reconstruct the edited 3D scene model through iterative optimization based on multi-view image supervision.

[0087] In some embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method of this application.

[0088] In some embodiments, an electronic device is also provided, including a memory and a processor. The memory stores a computer program that can run on the processor. When the processor executes the computer program, it implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method of this application. The processor includes a graphics processing unit (GPU), and the rendering and optimization process of the three-dimensional Gaussian splash model is accelerated by parallel computing through the GPU.

[0089] The following will further describe specific embodiments of the present invention. These embodiments are merely illustrative and do not mean that the present invention is limited to the following examples.

[0090] Example 1, Hardware Implementation Environment. This example is implemented on the following hardware and software environment: The computing device is equipped with an NVIDIA RTX 4090 GPU with 24GB of video memory. The operating system is Ubuntu 20.04 LTS. The deep learning framework is PyTorch 2.0. The differentiable rasterization rendering process of the 3D Gaussian splash model is executed in parallel on the GPU using a custom CUDA operator. 3D scene reconstruction uses COLMAP to obtain multi-view camera calibration parameters, and the 3D Gaussian splash model is iteratively trained 30,000 times to obtain the initial 3D model. When drawing sketches, the camera calibration parameters are retained, including rotation matrix, translation vector, X-axis field of view, Y-axis field of view, image width, and image height.

[0091] Example 2, Key Parameter Settings. The number of multi-view editing perspectives is 48, with uniform sampling. The single-view image resolution is 512×512 pixels. The dense point sampling density N is set to 500, with uniform sampling along the viewing direction. The depth offset d is set to 0.1 to prevent surface floating-point precision from affecting the results. The loss function weight parameters are set as follows: L1 loss weight is set to 100, perceptual loss weight is set to 1, and anchor point loss weight is set to 5. The optimization process parameters are set as follows: the maximum number of optimization iterations is set to 7500, new Gaussian primitive initialization is performed in the first 1000 steps, density adjustment and pruning are performed every 100 steps, and out-of-region point reset is performed every 30 steps. The mask processing parameters are set as follows: the noise removal convolution kernel size is 5×5, and the counting threshold Tc is 9; the edge enhancement convolution kernel size is 3×3, and the threshold is 8. The attention control parameters are set as follows: the parameters of the conditional diffusion generation model are all set to the default values, the inference step range of cross-view attention control is from step 20 to step 50, and the range of U-Net attention control layers is from layer 10 to layer 15.

[0092] Example 3, Processing Time Performance. The processing time on an NVIDIA RTX 4090 GPU is as follows: Sketch mask transformation: approximately 20 seconds. Multi-view 2D image editing: approximately 3 minutes, with 48 images processed in 8 batches, limited by GPU memory size. 3D model optimization and reconstruction: approximately 5 to 12 minutes, with optimal results achieved within 20 seconds; the specific time depends on the number of Gaussian points in the scene and the consistency across multiple views.

[0093] Example 4: Quantitative evaluation of editing effects. For example... Figure 6 and Figure 7 As shown, the quantitative comparative experimental results are as follows. Regarding user preference rate, the method of this invention achieves 81.36%, compared to 6.45% for GaussianEditor Edit mode, 5.79% for GaussianEditor Add mode, and 6.4% for Instruct-NeRF2NeRF. Regarding CLIP similarity, the method of this invention reaches 0.2498, compared to 0.2049 for GaussianEditor Edit mode, 0.2041 for GaussianEditor Add mode, and 0.2090 for Instruct-NeRF2NeRF. These data demonstrate that the method of this invention significantly outperforms existing methods in both user preference and instruction alignment.

[0094] The detailed explanation of user preference rate and CLIP similarity data is as follows: 1. User preference experiment: (1) Participants: A total of 46 participants, of which 37 had a background in computer vision / graphics (graduate students and above) and 9 were users without a professional background.

[0095] (2) Evaluation scenarios: There are a total of 14 test scenarios, divided into three categories: 5 human scenarios, 5 object scenarios, and 4 indoor scenarios.

[0096] (3) Evaluation process: Each participant independently views the editing results of each scene (this method vs. other comparison methods) and makes a preference selection based on three dimensions: consistency of multiple perspectives, editing fidelity, and overall visual quality. Each person can only choose one preferred method per scene.

[0097] (4) Calculation of preference rate: The preference rate was calculated separately for each scene category, and then the weighted average of the preference rates of the three categories was taken as the overall preference rate. Due to the different number of scenes in each category (5 people, 5 objects, and 4 indoor scenes), there is a slight difference between the overall preference rate after weighted averaging and the result of directly dividing by the total number of votes. Specifically, the total number of votes was 46 people × 14 scenes = 644 votes, but 81.36% was not obtained by directly rounding 644, but by the result of weighted averaging of the preference rates of the three categories. In the calculation of the preference rate of each category, rounding to two decimal places before averaging will introduce a small cumulative rounding error, but this error does not exceed ±0.5% and does not affect the validity of the conclusion.

[0098] 2. CLIP Similarity Assessment: (1) The model used is CLIP ViT-B / 32.

[0099] (2) Calculation method: Extract CLIP image features from the edited area of ​​the edited image and calculate the cosine similarity with the CLIP text features extracted from the user-input text prompt.

[0100] (3) 0.2498 is the average CLIP similarity score for all test scenarios.

[0101] 3. Fairness of the experiment: (1) All methods are run on the same test scenario, using the same user sketch input and text description.

[0102] (2) The hardware environment is uniformly NVIDIA RTX 4090 GPU (24GB video memory).

[0103] (3) The comparison methods (GaussianEditor, etc.) all use their official open source code and recommended parameter configurations.

[0104] Example 5, ablation experiment verification. For example... Figure 8As shown, firstly, the necessity of 3D region determination and Gaussian primitive initialization is verified. After removing the random initialization module and allowing all Gaussian primitives to participate in optimization, the quality of the editing result significantly decreases, and the original scene is improperly modified. For example, the Gaussian primitive of the hat is incorrectly modified from the original hair point, and abnormal color blocks appear on the background wall. This proves the necessity of the Gaussian primitive initialization strategy based on 3D region constraints for maintaining editing accuracy and the integrity of the original scene. Secondly, the necessity of cross-view attention control is verified. After removing the cross-view attention control module, there are obvious contradictions in texture details in the edited images from different perspectives, and the optimization process cannot converge to a consistent 3D result. This proves the key role of the cross-view attention control mechanism in achieving consistent 3D editing across multiple perspectives.

[0105] The strategy and computational feasibility of dense 3D point sampling in the above embodiments are explained as follows: (1) The sampling range does not cover all pixels of the entire image, but only the pixel area covered by the sketch drawn by the user in step S2. In a typical editing scenario, the user sketch usually only covers a small part of the image (usually no more than 10%-20% of the total pixels), so the actual number of pixels to be processed is much smaller than the 262,144 pixels of the entire image.

[0106] (2) The sampling and mask matching process employs batch GPU parallel computing. Assume the mask region contains... Each pixel is sampled. At this point, it is necessary to... If matching is performed from each perspective, the overall time complexity is O(n). Through the massive parallel processing of GPUs, the amount of computation is manageable.

[0107] (3) The following optimization strategies were also adopted in the actual implementation: a) Pixels to be processed were pre-screened using a two-dimensional mask to avoid invalid calculations of the background area; b) PyTorch’s batch matrix operation was used to efficiently complete the reprojection of three-dimensional points to two-dimensional pixels; c) A chunked processing strategy was adopted to process in batches when the mask area is large in order to control the memory usage.

[0108] (4) Basis for choosing sampling density N=500: This value was determined experimentally. Too low a sampling density (e.g., N<200) may lead to omissions of 3D regions due to depth estimation errors, while too high a sampling density (e.g., N>1000) will bring unnecessary computational overhead and have limited improvement on the results. N=500 achieves a good balance between accuracy and efficiency.

[0109] (5) The “approximately 20 seconds” processing time on the NVIDIA RTX 4090 GPU in Example 3 is the total time including the complete editing process, of which the dense sampling and matching steps account for only a small portion.

[0110] The following explains the setting of convolution kernels and thresholds in mask denoising and enhancement: (1) 5×5 convolution kernel and counting threshold The denoising design with a threshold of 9 uses a 5×5 convolutional kernel covering 25 pixel locations. A count threshold of 9 means a pixel is only retained if it has at least 9 (36%) mask pixels in its 5×5 neighborhood. This design is based on the following considerations: a) isolated noise points in the mask typically lack sufficient similar pixels in their 5×5 neighborhood and are therefore filtered out; b) a 36% retention threshold is lenient enough to preserve the main mask area and edge transitions, while being strict enough to eliminate sparse noise. Threshold values ​​between 7 and 12 show similar results, with 9 being the optimal value in experiments.

[0111] (2) Edge enhancement design with 3×3 convolution kernel and threshold of 8: The 3×3 convolution kernel covers 9 pixel positions. The threshold of 8 means that when there are at least 8 mask pixels in the 3×3 neighborhood of a non-masked pixel, the pixel is included in the mask. This is equivalent to only expanding the area by 1 pixel width outside the mask edge to ensure that the edge is complete and continuous without excessive expansion.

[0112] (3) The above parameters have been experimentally verified to perform well on images with a resolution of 512×512. For images with different resolutions, the kernel size can be adjusted proportionally (for example, 7×7 and 5×5 kernels can be used for 1024×1024 resolution), and the threshold can be adjusted proportionally to the kernel area.

[0113] The technical solution of the present invention has the following beneficial technical effects.

[0114] First, it achieves pixel-level precision in 3D editing area localization. Through a sketch-guided mask transformation module, the outline of the user-drawn 2D sketch is precisely mapped to a 3D spatial editing area with pixel-level localization accuracy, far exceeding the coarse localization accuracy of existing 3D bounding box methods. This allows for the addition of entirely new objects that were not previously present in the 3D scene.

[0115] Secondly, it significantly improves the shape controllability of 3D editing. Users only need to draw simple 2D sketch outlines from at least two perspectives to precisely control the shape, position, and size of the object being edited, making 3D geometric editing operations as intuitive and precise as drawing on a 2D image. In user preference evaluation experiments, the method of this invention achieved a user preference rate of 81.36%, far exceeding the 6.45% and 5.79% of existing methods GaussianEditor and 6.4% of Instruct-NeRF2NeRF.

[0116] Third, it effectively solves the problem of consistency in multi-view editing. Through a cross-view attention control mechanism, it ensures that multi-view edited images maintain consistency in texture and semantics, eliminating the 3D artifacts caused by editing conflicts between viewpoints in traditional methods. In the CLIP similarity evaluation, this invention achieves a score of 0.2498, significantly outperforming GaussianEditor's 0.2049 and 0.2041, and Instruct-NeRF2NeRF's 0.2090.

[0117] Fourth, it significantly improves the efficiency of 3D editing. Based on explicit representation using 3D Gaussian splashing, compared to methods based on implicit representation using neural radiation fields, rendering and optimization efficiency is significantly improved. On computing devices equipped with NVIDIA RTX 4090 GPUs, mask transformation processing takes only about 20 seconds, multi-view image editing processing takes about 3 minutes, and 3D model optimization processing generally takes 5 to 12 minutes. In optimal conditions, acceptable editing results can be obtained within 20 seconds.

[0118] Fifth, it has strong versatility. The method of this invention is applicable to both 3D scene editing and 3D object editing, and can edit various types of 3D Gaussian splash model instances.

[0119] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0121] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0123] The background section of this invention may include background information about the problems or environment in which the invention is being developed, and is not necessarily a description of prior art. Therefore, the content included in the background section does not constitute an admission of prior art by the applicant.

[0124] The above description provides a further detailed explanation of the present invention in conjunction with specific / preferred embodiments, and it should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various substitutions or modifications can be made to these described embodiments without departing from the concept of the present invention, and all such substitutions or modifications should be considered within the scope of protection of the present invention. In the description of this specification, the reference to terms such as "an embodiment," "some embodiments," "preferred embodiment," "example," "specific example," or "some examples," etc., indicates that the specific features, structures, materials, or characteristics described in connection with that embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions, and modifications can be made herein without departing from the scope of protection of the patent application.

Claims

1. A sketch-guided controllable editing method for three-dimensional Gaussian splatting scene image, applied to the geometric image editing process of three-dimensional Gaussian splatting model, characterized in that, Includes the following steps: S1. Steps for acquiring 3D scene image data: Acquire the 3D Gaussian splash model data of the 3D scene to be edited. The 3D Gaussian splash model data includes the position, covariance, opacity, and color attribute parameters of multiple 3D Gaussian primitives in the scene; acquire the multi-view acquired images and corresponding camera calibration parameters obtained by multi-view image acquisition of the 3D scene. The camera calibration parameters include the intrinsic parameter matrix and extrinsic parameter matrix of each view camera. S2. Sketch Interactive Input Step: Receive two-dimensional sketch outlines drawn by the user from at least two different viewpoints in the image interactive interface, and convert the two-dimensional sketch outlines into two-dimensional reference mask images of the corresponding reference viewpoints. S3. Three-dimensional editing region determination steps: For each pixel in the image corresponding to each acquisition viewpoint, the depth value corresponding to the pixel is obtained using the depth map rendered by the three-dimensional Gaussian splash model. Based on the intrinsic and extrinsic parameter matrices in the corresponding camera calibration parameters, the two-dimensional pixel is back-projected into the three-dimensional scene space to obtain its corresponding three-dimensional spatial coordinates. For each back-projected three-dimensional spatial coordinate point, its normalized observation direction vector relative to the camera optical center is calculated. A dense three-dimensional sampling point set is uniformly generated along the normalized observation direction vector within the depth range at a preset sampling density. The dense three-dimensional sampling point set is projected onto the image plane of each reference viewpoint camera and matched with the two-dimensional reference mask image under the corresponding reference viewpoint to determine the target three-dimensional editing space region and the transformation mask image corresponding to each viewpoint. S4. Mask image denoising and edge enhancement steps: Perform noise point removal and edge information enhancement processing based on convolution filtering on the transformed mask image obtained in step S3 to obtain an enhanced mask image, and extract edge feature maps from the enhanced mask image as guiding information; S5. Multi-view consistent image editing steps: Input the edge feature map and text description information into the pre-trained conditional diffusion generation model, and perform joint denoising generation processing on the multi-view image through a cross-view attention control mechanism to obtain a multi-view consistent edited image; S6. Three-dimensional model optimization and reconstruction steps: Initialize new three-dimensional Gaussian primitives within the target three-dimensional editing space region. Using the multi-view consistent edited image as the supervision target signal, minimize the combined loss function between the rendered image and the edited image, and iteratively optimize the attribute parameters of the three-dimensional Gaussian primitives using the stochastic gradient descent algorithm to obtain the edited three-dimensional scene model.

2. The sketch-guided 3D Gaussian-splatting scene image controllable editing method of claim 1, wherein, Step S3 specifically includes: S31. Three-dimensional spatial back projection of image pixels: For each pixel in the image corresponding to each acquisition viewpoint, the depth value corresponding to the pixel is obtained by using the depth map rendered by the three-dimensional Gaussian splash model. Based on the intrinsic and extrinsic parameter matrices of the camera at the acquisition viewpoint, the two-dimensional pixel is back projected onto the three-dimensional scene space to obtain its corresponding three-dimensional spatial coordinates. S32. Sampling of dense 3D point set along the observation direction: For each 3D spatial coordinate point obtained by back projection, calculate its normalized observation direction vector relative to the optical center of the camera, and uniformly generate a dense 3D sampling point set along the normalized observation direction vector within the depth range at a preset sampling density N. S33. Reprojection of dense 3D points to reference viewpoint: Project the dense 3D sampling point set onto the image plane of each reference viewpoint camera to obtain the two-dimensional projection coordinates on the reference viewpoint; S34. Three-dimensional region determination based on multi-view mask matching: For each pixel in the image corresponding to the acquisition viewpoint, determine whether there is at least one three-dimensional sampling point in the dense three-dimensional sampling point set associated with it whose projection falls within the corresponding two-dimensional reference mask area in all reference viewpoints. If the condition is met, it is determined that the pixel is located within the projection range of the target three-dimensional editing space. The three-dimensional sampling points that meet the condition constitute the geometric representation of the target three-dimensional editing space. S35. Based on the determination result of step S34, generate a transformation mask image corresponding to each acquisition viewpoint. The transformation mask image is used to identify all pixels that are determined to be located within the projection range of the target three-dimensional editing space under that acquisition viewpoint.

3. The sketch-guided, controllable editing method for a 3D Gaussian splash scene image according to claim 2, characterized in that, In step S32, the preset sampling density N is set to 500, and the depth offset is set to 0.

1. In step S5, the multi-view editing uses 48 uniform sampling views, and the image resolution of each view is 512×512 pixels. In step S6, the weight coefficients of the combined loss function are set to L1 loss weight of 100, perceptual loss weight of 1, and anchor loss weight of 5, and the maximum number of optimization iterations is set to 7500. In the denoising process, the convolution kernel size is 5×5 and the counting threshold Tc is 9. In the edge enhancement process, the convolution kernel size is 3×3 and the threshold is 8. The inference step range of cross-view attention control is from step 20 to step 50, and the range of U-Net attention control layers is from layer 10 to layer 15.

4. The sketch-guided 3D Gaussian-splatting scene image controllable editing method of claim 1, wherein, The noise reduction process includes: The transformation mask image tensor is convolved using a k×k convolution kernel. The number of pixels in the k×k local neighborhood of each pixel that are equal to the preset mask mark value T is counted. Pixels whose count exceeds the counting threshold Tc are determined to be valid mask pixels, and their values ​​are set to the preset mask mark value T. The remaining pixels remain unchanged. The edge enhancement process includes: creating a color mask M, setting the pixel position of color T to 1, performing a convolution operation on the color mask M using a convolution kernel of size k×k, normalizing and thresholding the convolution result, and generating a binarized enhancement mask B as the enhancement mask image.

5. The method for controllable editing of a 3D Gaussian splash scene image based on sketch guidance according to claim 1, characterized in that, The cross-perspective attention control mechanism described in step S5 specifically includes: (a) Cross-view unified attention calculation: In the inference denoising process of the conditional diffusion generative model, image features from different perspectives are treated as a unified whole for cross-view attention calculation. The feature representation of the source perspective image is used as the key and value, and the feature representation of the target perspective image is used as the query. Cross-view information interaction is achieved through attention mapping. (b) Enhanced batch consistency guided by reference images: A pre-selected reference edit image is inserted at the first input position of each processing batch, which serves as a global guiding signal in the attention calculation of that batch.

6. The method for controllable editing of a 3D Gaussian splash scene image based on sketch guidance according to claim 1, characterized in that, Step S6 specifically includes: S61. Region constraint initialization: Randomly sample several points from the three-dimensional point set of the target three-dimensional editing space region as the position coordinates of the new three-dimensional Gaussian primitives. The other attributes of each new three-dimensional Gaussian primitive, except for the position, are initialized by randomly sampling from the original three-dimensional Gaussian primitives and reusing their attribute values. S62. Iterative optimization: A combined loss function is used to measure the difference between the current 3D model rendered image and the target edited image. The combined loss function includes pixel-level L1 loss, perceptual loss based on a pre-trained visual feature network, and anchor point loss. The attribute parameters are iteratively updated using a stochastic gradient descent algorithm. S63. Periodic Reset: During the optimization iteration process, the current position of all new 3D Gaussian elements is detected every preset number of steps, and 3D Gaussian elements that exceed the boundary of the target 3D editing space region are randomly reset to the inside of the region; S64. Adaptive Density Adjustment and Pruning: During the optimization process, the distribution density of 3D Gaussian primitives in the current editing area is periodically evaluated. New 3D Gaussian primitives are added to areas with insufficient density, while 3D Gaussian primitives that contribute little to the rendering result are removed.

7. The method for controllable editing of a 3D Gaussian splash scene image based on sketch guidance according to claim 1, characterized in that, The multi-view consistent image editing step in step S5 further includes: After editing, the mask region of the multi-view consistent edited image output by the conditional diffusion generation model is extracted and superimposed on the original multi-view acquired image to complete the local image editing fusion. The enhanced mask image generated in step S4 is used to distinguish between the edited area and the non-edited area, thereby ensuring that the edited content only applies to the area specified by the two-dimensional sketch outline and does not affect other parts of the multi-view acquired image.

8. A controllable editing device for a 3D Gaussian splash scene image based on sketch guidance, characterized in that, include: The 3D scene data acquisition module is used to acquire the 3D Gaussian splash model data, multi-view acquired images, and camera calibration parameters of the 3D scene to be edited; The sketch interactive input module is used to receive two-dimensional sketch outlines drawn by the user from at least two different perspectives through the image interactive interface and convert them into reference mask images; The 3D editing region determination module is used to perform 3D spatial back-projection, dense 3D point set sampling and reference viewpoint reprojection on pixels according to camera calibration parameters, and determine the target 3D editing space region and the transformation mask of each viewpoint through multi-view mask matching. The mask image preprocessing module is used to perform noise point removal and edge information enhancement processing on the transformed mask image based on convolution filtering; The multi-view consistent editing module, including an edge feature extraction submodule and a cross-view attention control submodule, is used to generate multi-view consistent edited images through a pre-trained conditional diffusion generation model. The 3D model optimization and reconstruction module is used to initialize 3D Gaussian primitives within the 3D editing area and reconstruct the edited 3D scene model through iterative optimization based on multi-view image supervision.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method as described in any one of claims 1 to 7.

10. An electronic device comprising a memory and a processor, wherein the memory stores a computer program executable on the processor, characterized in that, When the processor executes the computer program, it implements the sketch-guided three-dimensional Gaussian splash scene image controllable editing method as described in any one of claims 1 to 7; the processor includes a graphics processing unit (GPU), and the rendering and optimization process of the three-dimensional Gaussian splash model is accelerated by GPU parallel computing.

Images