Sparse-view based spatial 3D editing method, device and storage medium

Abstract

The application discloses a sparse view-based space 3D editing method, device and storage medium, relates to the technical field of 3D modeling, and discloses a sparse view-based space 3D editing method, which comprises the following steps: acquiring a plurality of sparse views containing a to-be-modeled object and a text editing instruction for specifying editing content in the sparse views; inputting the spliced sparse views and the text editing instruction into an image editing model trained in cross-view consistency to obtain a spliced image with global semantic consistency and local editing area alignment, wherein the image editing model is obtained through self-supervised training of a generated model of an insertion parameter fine-tuning module; dividing the spliced image into a plurality of multi-views with editing consistency, and inputting the multi-views into a three-dimensional Gaussian reconstruction model to obtain a three-dimensional model corresponding to the to-be-modeled object. Based on the above, the operation process and the calculation complexity of 3D space editing are simplified by relying on a feedforward processing architecture, and the three-dimensional editing effect and the editing efficiency are improved.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, and in particular to spatial 3D editing methods, devices and storage media based on sparse views. Background Technology

[0002] With the development of immersive media, extended reality, and robotics simulation technologies, semantic control and visually consistent editing capabilities for 3D scenes have become core industry requirements.

[0003] In related spatial 3D editing methods, the SDS (Score Distillation Sampling) method is used for processing. This method optimizes the 3D scene by using a pre-trained diffusion model to provide gradients. However, the SDS method only calculates gradients for a single viewpoint at a time, and the gradient directions given by the diffusion model are inconsistent or even contradictory under different viewpoints, affecting the realism of the constructed 3D model.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main purpose of this application is to provide a spatial 3D editing method, device and storage medium based on sparse views, which aims to solve the technical problem of poor realism in 3D modeling.

[0006] To achieve the above objectives, this application proposes a spatial 3D editing method based on sparse views, the method comprising: Obtain multiple sparse views containing the object to be modeled, and text editing instructions in the sparse views for specifying the content to be edited; The stitched sparse view and the text editing instructions are input into the cross-view consistency trained image editing model to obtain a stitched image with global semantic consistency and local editing region alignment. The image editing model is obtained through self-supervised training of the generative model of the insertion parameter fine-tuning module. The stitched image is divided into multiple edit-consistent multi-view images and input into a 3D Gaussian reconstruction model to obtain the 3D model corresponding to the object to be modeled.

[0007] In one embodiment, the parameter fine-tuning module is a LoRA module. Before the step of acquiring multiple sparse views containing the object to be modeled and text editing instructions in the sparse views for specifying the editing content, the spatial 3D editing method based on sparse views further includes: Construct a training dataset, wherein each training sample contains a preset number of sparse views from different perspectives in the same scene, and a corresponding training text editing instruction; The training samples are input into the generative model of the LoRA module to generate a training stitched image with a preset layout pattern, wherein the preset layout pattern is associated with the preset quantity. The global diffusion feature loss and local editing feature loss of the training stitched image are determined as the regularization term loss of the LoRA module; The parameters of the LoRA module are iteratively updated until the global diffusion feature loss and the local editing feature loss converge together to obtain the image editing model.

[0008] In one embodiment, before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: Gaussian noise is added to the training stitched image based on a preset time step to obtain a noisy stitched image; The noisy stitched image is input into the UNet network of the generation model to obtain the first diffusion features corresponding to the left and right views and the second diffusion features corresponding to the top and bottom views in the noisy stitched image. The first diffusion feature and the second diffusion feature are subjected to feature alignment processing to obtain the global diffusion feature loss.

[0009] In one embodiment, before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: Noise is added to the sparse view in the training samples based on the preset time step to obtain a noisy image; Determine the original diffusion features obtained after the noisy image is processed by the UNet network of the generation model; Calculate the L1 loss between the original diffusion feature and the global diffusion feature, and multiply the L1 loss by -1 so that the training stitched image deviates from the sparse view corresponding to the training sample.

[0010] In one embodiment, before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: The trained stitched image is divided into multiple non-overlapping image blocks; An image encoder based on a preset vector embedding model extracts image features from each image patch, and a text encoder based on a preset vector embedding model extracts text features from the training text editing instructions; Calculate the cosine similarity between the features of each image patch and the text features, and locate the local editing region of the training stitched image based on the image patches whose cosine similarity is greater than a preset similarity. The dense features of the local editing region are extracted based on the dense feature extraction model; The dense features are subjected to feature alignment processing to obtain the local editing feature loss.

[0011] In one embodiment, the step of inputting the multiple views into a 3D Gaussian reconstruction model to obtain a 3D model corresponding to the object to be modeled includes: Determine the editable area of ​​the multi-view; Extract the multi-view features of the region to be edited; The multi-view features are mapped to the center coordinates and Gaussian parameters of a three-dimensional Gaussian based on the Gaussian parameter prediction head. Based on the center coordinates and the Gaussian parameters, an edited 3D Gaussian sputtering scene is generated, resulting in the 3D model.

[0012] In one embodiment, the step of extracting the multi-view features of the region to be edited includes: The multi-view blocks are embedded as visual lexical units, and the camera intrinsic parameters corresponding to each view are projected as learnable embeddings and concatenated with the visual lexical units of the corresponding views to obtain the target visual lexical units. The ViT encoder based on shared weights extracts single-view visual features of target visual words in each view. The multi-view features are obtained by performing cross-view correlation modeling on all single-view visual features based on the cross-view decoder.

[0013] In one embodiment, the step of inputting the stitched sparse view and the text editing instructions into an image editing model trained on cross-view consistency to obtain a stitched image with global semantic consistency and local editing region alignment, wherein the image editing model is obtained through self-supervised training of the generative model of the insertion parameter fine-tuning module includes: The stitched sparse view and the text editing instructions are input into an image editing model trained on cross-view consistency, wherein the image editing model determines the editing module based on the number of stitches in the sparse view; Obtain the stitched image output by the editing module, which corresponds to the number of stitches in the sparse view and is semantically consistent globally and aligned with the local editing regions.

[0014] In addition, to achieve the above objectives, this application also proposes a 3D editing device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the spatial 3D editing method based on sparse views as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the spatial 3D editing method based on sparse views as described above.

[0016] One or more technical solutions proposed in this application have at least the following technical effects: The process involves acquiring multiple sparse views containing the object to be modeled, along with text editing instructions within those sparse views specifying the editing content. The stitched sparse views and text editing instructions are then input into an image editing model trained under self-supervised supervision by a generative model trained on cross-view consistency and fine-tuned by an insertion parameter module. This generates a stitched image with global semantic consistency and local editing region alignment. Finally, the stitched image is segmented into multiple edit-consistent multi-views, which are then input into a 3D Gaussian reconstruction model to obtain the 3D model corresponding to the object to be modeled. This feedforward processing architecture simplifies the 3D spatial editing process and reduces computational complexity. Cross-view consistency training eliminates semantic biases and spatial misalignments in the edited content from different viewpoints, avoiding defects such as texture breaks, viewpoint contradictions, and abrupt editing traces during 3D reconstruction. Meanwhile, by leveraging the self-supervised training method of the generative model through the insertion parameter fine-tuning module, the precise mapping and flexible adaptation of text editing instructions are achieved without compromising the original visual generation capabilities of the generative model. This allows the final generated 3D model to fully preserve the spatial structural features of the original scene, enabling the edited content and the original scene to achieve natural unity in visual style, spatial scale, and lighting logic, presenting a more realistic and immersive 3D editing effect. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating the first embodiment of the spatial 3D editing method based on sparse views provided in this application; Figure 2 This is a four-view image stitched together using the spatial 3D editing method based on sparse views, as described in this application. Figure 3 This is a schematic diagram of the stitched image obtained after model processing of the four views of the spatial 3D editing method based on sparse views in this application; Figure 4 A schematic diagram showing the image after processing using other 3D modeling methods; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the spatial 3D editing method based on sparse views in the embodiments of this application.

[0020] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0022] With the development of immersive media, extended reality, and robotics simulation technologies, semantic control and visually consistent editing capabilities for 3D scenes have become core industry requirements.

[0023] In related spatial 3D editing methods, the SDS-guided approach is used, which optimizes the 3D scene by using a pre-trained diffusion model to provide gradients. However, the SDS-guided approach only calculates gradients for a single viewpoint at a time, and the gradient directions provided by the diffusion model are inconsistent or even contradictory across different viewpoints, affecting the realism of the constructed 3D model.

[0024] Based on this, this application provides a solution that, with the help of the self-supervised training method of the generative model through the insertion parameter fine-tuning module, achieves accurate mapping and flexible adaptation of text editing instructions without destroying the original visual generation capabilities of the generative model. This allows the final generated 3D model to fully retain the spatial structural features of the original scene, and enables the edited content and the original scene to achieve natural unity in visual style, spatial scale, and lighting logic, presenting a more realistic and immersive 3D editing effect.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or 3D editing device capable of performing the above functions. The following description uses a 3D editing device as an example to illustrate this embodiment and the subsequent embodiments.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] This application provides a spatial 3D editing method based on sparse views, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the spatial 3D editing method based on sparse views in this application.

[0028] In this embodiment, the spatial 3D editing method based on sparse views includes steps S10 to S30: Step S10: Obtain multiple sparse views containing the object to be modeled, and text editing instructions in the sparse views for specifying the editing content.

[0029] A sparse view is a two-dimensional image acquired from different spatial perspectives of the same scene / object. Text editing instructions are natural language descriptions that typically contain editing information or attribute adjustment information for the object to be modeled in the image, such as: "Turn object A in the image black." The object to be modeled is the object information appearing in the sparse view; it can be all entity objects in the entire scene of the image, or a local target object specified in the text editing instruction. The number of sparse views can be 2, 4, 6, 8, etc. To reduce the computational cost of loss during model iteration training, 2 or 4 views are usually chosen for training, resulting in 2 or 4 sparse views. In this embodiment, the preferred number of sparse views is 4.

[0030] In this embodiment, multiple sparse views containing the object to be modeled can be directly acquired from the client, or multiple sparse views containing the object to be modeled can be directly acquired from the external camera of the 3D editing device. The sparse views uploaded by the client typically include corresponding text editing instructions; that is, when the user uploads multiple sparse views from the client, they also upload the corresponding text editing instructions. Similarly, after acquiring the multiple sparse views acquired from the external camera, the text editing instructions input by the user are received through a graphical user interface.

[0031] For example, a user points their smartphone at an inverted cup on their desk and takes a clear image from four different perspectives. Then, when uploading the images, they enter the text editing command "Change the red area in the cup to yellow" in the text editing interface. After the data is uploaded, the 3D device associates and stores the sparse view of the cup with the corresponding text editing command.

[0032] Step S20: Input the stitched sparse view and text editing instructions into the image editing model trained on cross-view consistency to obtain a stitched image with global semantic consistency and local editing region alignment.

[0033] Image stitching is a process of combining multiple sparse views into a single large image according to a preset spatial arrangement rule. This involves converting the input multi-view data into single-image data to adapt to the input format requirements of current mainstream models while preserving the relative spatial relationships between the views. Stitching methods can include grid stitching, such as stitching two images side-by-side, stitching four images in a 2×2 grid, or stitching six images in a preset 2×3 or 3×2 grid, and so on for other numbers. The parameter fine-tuning module is a lightweight, trainable sub-module inserted into the middle layer of the pre-trained generative model. This module can be a LoRA low-rank adaptation module, an adapter module, a hint tuning module, a prefix tuning module, etc. This parameter fine-tuning module only trains its parameters while freezing the backbone parameters of the generative model, enabling rapid adaptation to specific cross-view editing tasks while retaining the general generative capabilities of the pre-trained model. Cross-view consistency training introduces cross-view semantic consistency loss and spatial alignment loss during the training phase of the image editing model, forcing the model to generate editing results with the same semantics, matching spatial position, and consistent visual attributes for the same three-dimensional spatial object from different viewpoints.

[0034] In this embodiment, the image editing model is self-supervised trained using the LoRA fine-tuning module, obtaining four sparse views. Therefore, after obtaining these four sparse views, they are stitched together using a fixed 2×2 grid to generate a single stitched input image. This avoids designing complex multi-input model branches and reduces inference latency. Next, the stitched image and the structured semantic vector of the text editing instructions are input into an image editing model, such as the FLUX model, which is based on open-source or commercial closed-source technology and incorporates the LoRA fine-tuning module. Finally, during inference, the model extracts the semantic features of the edited object in different view regions and calculates the similarity of these features. Simultaneously, it calculates the spatial coordinate deviation of the object to be modeled in each view, and uses the weighted sum of these two values ​​as the loss term for backpropagation optimization to generate the final result. This eliminates semantic bias and positional misalignment from different perspectives. The similarity can be calculated using CLIP, other open-source models such as Qwen3-VL-Embedding-8B, or by replacing the LoRA fine-tuning module with an Adapter fine-tuning module.

[0035] For example, please refer to Figure 2The 3D equipment stitches together four upside-down cups from an office workstation into a 2×2 grid image in the order of left front, front, right front, and top. Simultaneously, a structured semantic vector indicating "turning the red areas in the cups yellow" is input into the model. During the generation process, the model calculates the feature similarity and spatial center coordinate deviation of the "cup" areas in the four views in real time, ensuring that the cups maintain consistency in size, position, and light reflection direction across the four perspectives, ultimately generating the desired image. Figure 3 The image shown is a stitched image with global semantic consistency and aligned editing regions.

[0036] It should be noted that the common editors DGE (Direct Gaussian Editor), EditSplat (editable Gaussian splash), and ViP3DE (generative Video Prior based 3D Editing) are different. Figure 1 The processing result of the pixel-level 3D editor is as follows: Figure 4 As shown. DGE modifies existing 2D editors using a training-free approach, achieving cross-view functionality solely through spatiotemporal attention mechanisms and epipolar constraints. Figure 1 Consistency. This post-processing constraint can only achieve coarse alignment at the feature level. For fine-grained semantic control tasks such as modifying only the color while preserving the text content, it is prone to the problem of changing the color while destroying the original details. For EditSplat, EditSplat ensures consistency by fusing editing information from multiple views, but this fusion process can produce an over-smoothing effect on high-frequency details (such as text strokes). In order to eliminate the differences between different views, the model tends to generate an averaged result, causing fine text details to be smoothed out, while the text color is not adjusted. For ViP3DE, this model introduces video priors to improve cross-view consistency. Figure 1 In contrast, video priors focus more on the overall movement and appearance consistency of objects, neglecting the preservation of local details. Therefore, for fine-grained editing tasks such as changing only the color of text content, video priors tend to generate a unified overall appearance, thus erasing all local details and adjusting the color of the project to yellow.

[0037] Please continue to refer to Figure 3 In this embodiment, the red part of the cup is accurately turned into yellow, the text on the cup is also clearly visible and the color is correct, and the pattern at the bottom of the cup is completely preserved. The overall lighting and shadow effects are consistent with the original image, and there is no loss of detail or color deviation.

[0038] Therefore, in this embodiment, instead of directly modifying the existing 2D editor for multi-view editing tasks, self-supervised cross-view consistency training is performed on the generative model with inserted parameter fine-tuning modules. By inserting lightweight parameter fine-tuning modules such as LoRA or Adapter, the powerful general generation capabilities of the pre-trained generative model are retained while learning specific mapping relationships for cross-view editing. This enables the model to accurately understand text editing instructions and generate high-quality, detailed editing results, avoiding the problems of decreased model generalization ability and loss of detail caused by full fine-tuning. This provides high-quality input data for subsequent 3D reconstruction and improves the realism of 3D modeling.

[0039] Optionally, multiple corresponding sub-models can be trained during model training for different numbers of sparse views. Then, during use, the corresponding sub-model within the model is selected for processing based on the number of sparse views. Therefore, when the stitched sparse views and text editing instructions are input into the cross-view consistency-trained image editing model, this model determines the editing module based on the number of stitched sparse views. It then directly obtains the stitched image output by the editing module, which corresponds to the number of stitched sparse views and is globally semantically consistent and has aligned local editing regions. For example, when the number of sparse views is 4, the editing module with training process 4 is selected to output the corresponding stitched image.

[0040] Step S30: Divide the stitched image into multiple edit-consistent multi-views and input the multi-views into the 3D Gaussian reconstruction model to obtain the 3D model corresponding to the object to be modeled.

[0041] Edit consistency multiview is obtained by segmenting from stitched edit images and is consistent with the original sparse view. Figure 1 For a single edited image, the difference between the original view and the current view from the same perspective stems from text editing commands, such as... Figure 2 and Figure 3 In this model, the difference between the segmented view and the original view lies in the color replacement; other information, including spatial location and visual attributes, remains highly consistent. The 3D Gaussian reconstruction model is a feedforward 3D Gaussian model. In this embodiment, this model differs from the traditional iterative optimization 3D Gaussian sputtering (3DGS) model, meaning it does not require multiple iterations to optimize the Gaussian parameters. It can predict all the complete parameters of the 3D Gaussian model from the input multiple views at once based on the image editing model, achieving millisecond-level 3D scene reconstruction, thereby realizing feedforward spatial 3D editing.

[0042] In this embodiment, the image data of the output 2×2 grid-stitched image can be uniformly divided into four sub-images in the exact reverse order of the stitching process, with each sub-image corresponding to one original sparse view. The sub-images are then subjected to edge smoothing to eliminate any edge stitching artifacts that may have occurred during the stitching process. The edit-consistent multi-view is then input into a feedforward 3D Gaussian reconstruction model to obtain the 3D model output by the model. In the model reconstruction process, each edited image can be segmented into non-overlapping image blocks. Each image block is converted into a preset visual token through an embedding layer, while preserving the local spatial features of the image. Then, the camera intrinsic parameters (focal length, principal point) and extrinsic parameters (rotation matrix, translation vector) are projected into learnable embedding vectors of the same dimension as the visual tokens, and concatenated with the visual tokens of the corresponding views, allowing the model to understand the spatial relationship between different views. Finally, the local features of a single view are extracted through a ViT encoder with shared weights, and the feature information of all views is fused through a cross-view decoder to establish the pixel correspondence between different views. Then, Gaussian parameters are output through two parallel prediction heads, and all the predicted 3D Gaussian parameters are combined into a standard 3DGS scene representation format for output.

[0043] This embodiment is based on a pre-trained end-to-end feedforward inference architecture, which eliminates the time-consuming process of traditional 3DGS requiring hundreds to thousands of iterations of optimization for each scene, thus eliminating the need for scene-specific optimization. At the same time, it compresses the reconstruction time in the 3D editing process from several minutes to several hours to seconds, improving efficiency by more than an order of magnitude and significantly improving the overall response speed of 3D editing, achieving near real-time editing feedback.

[0044] Alternatively, sparse view 3D reconstruction frameworks based on other multi-view geometry can be used, such as Gaussian, voxel, and point cloud methods, to replace feedforward 3D reconstruction networks for 3D Gaussian reconstruction. That is, the architecture only needs to be able to recover the 3D structure of the scene from the sparse view.

[0045] This embodiment provides a spatial 3D editing method based on sparse views. It requires only a small number of sparse views centered on the object to be modeled to achieve precise semantic text-driven editing of specific entities, eliminating the need for expensive multi-view acquisition equipment and professional 3D modeling software, thus lowering the technical threshold and hardware cost of 3D editing. Simultaneously, by combining a cross-view consistency-trained image editing model with semantically constrained 3D Gaussian reconstruction technology, it ensures the multi-view semantic consistency and spatial realism of the edited content from the 2D editing source to the 3D reconstruction output. This solves the defects of traditional single-view independent editing methods, such as cross-view content misalignment, texture discontinuity, and lighting contradictions, thereby enhancing the realism of the 3D model. Based on a lightweight parameter fine-tuning module and an end-to-end feedforward processing architecture, it avoids the catastrophic forgetting problem caused by full-scale model fine-tuning and eliminates the time-consuming process of scene-by-scene iterative optimization in traditional 3D Gaussian reconstruction, reducing the computational cost of model training and inference and improving the execution efficiency of 3D editing.

[0046] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to the first embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, the parameter fine-tuning module is a LoRA module; therefore, before step S10, the spatial 3D editing method based on sparse views further includes steps S01~S04: Step S01: Construct the training dataset.

[0047] Understandably, each training sample contains a preset number of sparse views from different perspectives within the same scene, along with a corresponding training text editing instruction. The preset number can be 2, 4, 6, 8, or other numbers. To reduce the computational cost of loss while improving the accuracy of the feedforward input results, this embodiment preferably uses 4.

[0048] In this embodiment, for LoRa fine-tuning of the image editing model, the open-source re10k (RealEstate10K) dataset can be used as the training dataset. After selecting the training dataset, four views from different perspectives are extracted from each scene as sparse views of that scene. At the same time, combined with the editing instructions given by the user for that scene, the four views of each scene are stitched together and combined with their corresponding editing instructions to form the smallest data unit tuple, i.e., the training sample (also known as an image instruction pair).

[0049] Optionally, training text can also be generated through automatic annotation, such as by fine-tuning the model to automatically generate corresponding training text editing instructions for each scene.

[0050] Step S02: Input the training samples into the generative model of the LoRA module to generate a training stitched image with a preset layout.

[0051] In this embodiment, the preset layout style is associated with the preset quantity. For example, when the number of training views is 2, the layout style is left and right splicing, while when the number of training views is 4, the layout style is a 2×2 grid pattern.

[0052] Specifically, the training sample has 4 views. During training, the horizontally and vertically stitched left, right, top, and bottom views, along with editing instructions, are input into the LoRA-inserted generative model. Based on single-step denoising inference at a normalized time step t=1, a single-step Euler update method is used to complete diffusion sampling, generating the corresponding stitched format of the left, right, top, and bottom views, such as... Figure 2 The four views shown.

[0053] Step S03: Determine the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module.

[0054] Global diffusion feature loss is a loss function used to constrain the overall semantic and visual style consistency of the generated image, preserving the general image generation capability of the pre-trained generative model and avoiding the LoRA module from overfitting to a specific editing task, which would lead to a decrease in the quality of the generated image. Local editing feature loss, on the other hand, is a loss function used to constrain the accuracy of the editing effect in the editing region, ensuring that the model can accurately execute text editing instructions and generate edited content that meets the requirements.

[0055] In this embodiment, to ensure global semantic and geometric consistency, a global diffusion feature loss is introduced as a regularization term for fine-tuning training. Similarly, to ensure consistency in local edited regions, a local edit feature loss is introduced as a regularization term for fine-tuning training. This can be achieved by calculating the global diffusion feature loss for each sub-image in the training stitched image, or by calculating the global diffusion feature loss for the corresponding region after region division. When calculating the local edit feature loss, the block to be calculated can be selected, and the image features of the block can be aligned to obtain the local edit feature loss.

[0056] Step S04: Iteratively update the parameters of the LoRA module until the global diffusion feature loss and the local editing feature loss converge together to obtain the image editing model.

[0057] In this embodiment, training hyperparameters are typically set during module iterative updates, such as batch size, learning rate, weight decay, and maximum number of iterations. Using global diffusion feature loss and local editing feature loss, only the parameters of LoRA are updated while the backbone weights remain frozen.

[0058] For example, the batch size for training hyperparameters is 4, the learning rate is 1e-4, the maximum number of iterations is 10000, and each iteration uses 4 training samples to generate a stitched training image and calculate the total loss. The parameters of the LoRA module are then updated via backpropagation. After every 100 iterations, the total loss is calculated on 1000 validation samples, with an initial validation set total loss of 0.52. As training progresses, the total loss gradually decreases. When the iteration reaches 6200, the validation set total loss reaches its minimum of 0.11. After 500 consecutive iterations, the loss no longer decreases, and training is terminated early. The LoRA module parameters at this point are saved, resulting in the final image editing model that can be used for image editing.

[0059] It should be noted that during training, the information of each training sample is usually the same; for example, each training sample contains four sparse views and a corresponding training edit text. To improve the model's specificity, the generative model can be configured with multiple different sub-models, each trained on different training samples. This allows for processing different numbers of sparse views using different sub-models.

[0060] This embodiment provides a spatial 3D editing method based on sparse views. By inserting a LoRA module for lightweight fine-tuning and using global and local joint loss function constraints, it retains the general generation capability of the pre-trained model while ensuring the accuracy and cross-view consistency of the editing effect. This allows the trained image editing model to be directly applied to subsequent 3D editing processes, providing a feedforward processing model for generating high-fidelity 3D models.

[0061] Based on the second embodiment of this application, in the third embodiment of this application, the same or similar content as the second embodiment can be referred to the above description, and will not be repeated hereafter. Furthermore, before using the global diffusion feature loss and local editing feature loss as regularization terms, it is necessary to first determine the global diffusion feature loss and local editing feature loss corresponding to the training stitched image.

[0062] Therefore, before step S03, the global diffusion feature loss can be calculated first. During the calculation, noise can be added to the horizontally and vertically stitched four views at a specific time step to create noisy images, which are then input into the model. The corresponding intermediate diffusion features are extracted from a specific upsampling layer of the UNet network in the model as global diffusion features. In this process, the global diffusion features of the horizontally and vertically stitched left, right, top, and bottom views are aligned to ensure consistency in global editing of the four views.

[0063] Specifically, Gaussian noise is added to the training stitched image at preset time steps to obtain a noisy stitched image. After obtaining the noisy stitched image, it is input into the UNet network of the generative model to obtain the first diffusion features corresponding to the left and right views and the second diffusion features corresponding to the top and bottom views in the noisy stitched image. Feature alignment processing is performed on the first and second diffusion features to obtain the global diffusion feature loss. Here, the left and right views are the left and right side views after vertical cutting of the four views, and the top and bottom views are the top and bottom side views after horizontal cutting of the four views.

[0064] It is understandable that when extracting the corresponding intermediate diffusion features from specific upsampling layers of the UNet network of the model as global diffusion features using noisy images, the left and right views and the top and bottom views correspond to the horizontal rotation view group and the vertical pitch view group of the object to be modeled, respectively. The spatial geometric transformation rules and feature continuity constraints of the two are fundamentally different. By calculating the global diffusion features corresponding to the left and right views and the top and bottom views in the stitched four views, we can avoid the weakening of the consistency of a certain dimension caused by the averaging of a single global loss. This ensures that the model learns the editing mapping rules in both the horizontal and vertical directions at the same time, providing accurate spatial constraints in two orthogonal dimensions for 3D reconstruction, and ultimately improving the overall geometric integrity and visual realism of the 3D model.

[0065] Therefore, with a preset quantity of 4, when inputting the noisy stitched image into the UNet network of the generative model, the noisy stitched image can be segmented horizontally and vertically through two parallel paths to obtain the left and right views and top and bottom views of the noisy stitched image (four views). These views are then input into the UNet network of the generative model to obtain the first diffusion feature corresponding to the left and right views and the second diffusion feature corresponding to the top and bottom views. After obtaining the two diffusion features, feature alignment is performed on the first and second diffusion features, i.e., the global features are added together to obtain the global diffusion feature loss.

[0066] As an alternative implementation for calculating the global diffusion feature loss of the training stitched images, when the number of training stitched images is two, noise can be added to the two horizontally stitched views at a specific time step to create noisy images, which are then input into the diffusion model. Corresponding intermediate diffusion features are extracted from a specific upsampling layer of the model's UNet network. Subsequently, the global diffusion features of the horizontally stitched left and right views are aligned, i.e., the global features on the left and right sides are aligned. Compared to calculating the left, right, top, and bottom views, the 3D model reconstructed from two views can only be rotated and viewed in a limited manner in one direction, and significant distortion may occur when rotating in other directions. However, by inputting the left, right, top, and bottom views of four views into the UNet network of the generator model and calculating the global loss, multiple spatial constraints in different directions can be provided simultaneously, thereby improving the geometric integrity and scene information richness of the input data, and further enhancing the spatial accuracy and visual realism of the subsequent 3D reconstruction model.

[0067] Optionally, when the number of stitches in the training image is 6 or more, the corresponding global loss can be calculated based on a preset calculation method. For example, in a 2×3 arrangement, the losses for the top, bottom, left, middle and right halves can be calculated, and then feature alignment is performed to obtain the global diffusion feature loss. The same applies to other numbers.

[0068] This embodiment provides a spatial 3D editing method based on sparse views. When calculating the global loss, the method first divides the stitched image into left and right views and top and bottom views. It then performs feature alignment processing on the first diffusion features corresponding to the left and right views and the second diffusion features corresponding to the top and bottom views. This targeted matching of spatial geometric transformation rules for each viewpoint simultaneously constrains the semantic consistency and spatial continuity of the edited content. This avoids the weakening of consistency in a certain dimension or interference between different dimension losses caused by averaging a single global loss. Simultaneously, it strengthens the correspondence between 2D editing features and 3D spatial structure, providing precise two-dimensional spatial constraints for subsequent 3D Gaussian reconstruction, and improving the geometric integrity and visual realism of the 3D model. Based on the second embodiment of this application, in the fourth embodiment of this application, the same or similar content as the second embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, when calculating the local editing feature loss, the local area to be edited can be located first, and then the image features can be extracted from the four views through a dense feature extraction model. The extracted features include the features of each specific image patch in the four different views, and finally, local loss alignment processing is performed.

[0069] Specifically, when locating the local region to be edited, the training stitched image can first be segmented into multiple non-overlapping image patches. Then, an image encoder based on a pre-defined vector embedding model extracts the image features of each image patch, and a text encoder based on the same model extracts the text features of the training text editing instructions. Next, the cosine similarity between the features of each image patch and the text features is calculated. Based on image patches with a cosine similarity greater than a pre-defined similarity, the local editing region of the training stitched image is located. The pre-defined vector embedding model can be a CLIP (Contrastive Language-Image Pre-training) similarity model or a Qwen3-VL-Embedding-8B model.

[0070] After determining the local area to be edited, the dense features of the local editing area are extracted according to the dense feature extraction model. Finally, the dense features are aligned to obtain the local editing feature loss.

[0071] For example, taking the text editing task of a ceramic mug as an example, the training stitched image is a 2048×2048 pixel image with a 2×2 grid arrangement, containing four original views of the mug from the left, right, top, and bottom. The corresponding training text editing instruction is "change the mug to a gradient blue and add a white cat pattern". The preset vector embedding model is Qwen3-VL-Embedding-8B, and the dense feature extraction model is DINOv2-L. When obtaining the local editing feature loss, the 2048×2048 training stitched image is first uniformly divided into 32×32 pixel non-overlapping image blocks, resulting in a total of 64×64=4096 image blocks. Each image block corresponds to an independent local region in the stitched image. Then, the Qwen3-VL-Embedding-8B model image encoder is called to extract the 4096-dimensional visual feature vector of each 32×32 image block in turn. Based on the model, the text encoder converts the training text editing instruction "Change the mug to gradient blue and add a white cat pattern" into a text feature vector of the same dimension.

[0072] Next, the cosine similarity between the visual features and text features of 4096 image patches was calculated. The similarity value ranged from -1 to 1, with higher values ​​indicating a stronger semantic relevance between the image patch and the editing instructions. The image patch similarity in the front view of the mug's body was 0.87, in the left view it was 0.83, in the right view it was 0.81, and in the top view the rim was 0.76. The similarity of the image patches in the background / tablet area was generally below 0.2. At this point, 128 image patches with a similarity greater than 0.6 were selected as the local editing area. These image patches were distributed across the mug's body in the left, right, top, and bottom views, covering all the parts that needed editing.

[0073] After determining the local editing regions, 128 image blocks of the located local editing regions are input into the DINOv2-L dense feature extraction model. A 16×16×768-dimensional dense feature map is extracted for each 32×32 image block, meaning each pixel corresponds to a 768-dimensional feature vector. Finally, cross-view dense feature alignment is performed to calculate the local loss. Based on the spatial correspondence of the local editing regions in the four views and the pre-calculated camera pose matrices for each view, the two-dimensional coordinates of each image block are projected into three-dimensional space, and then projected back into the other three views to obtain the corresponding regions of the image block in the other views. For example, the image block containing the cat's head in the front view corresponds to the image block of the left half of the cat in the left view, the image block of the right half of the cat in the right view, and the image block on top of the cat's head in the top view. Next, the dense features of each corresponding region are aligned, and the pairwise cosine similarity between the feature vectors of corresponding pixels at the same spatial location in the four views is calculated. The average value is taken as the feature consistency score for that location, and then 1 is subtracted from this score to obtain the local loss value for that location. For example, the feature similarity scores of the cat's left eye position in the four views are 0.93, 0.91, 0.89, and 0.92, respectively, with an average score of 0.9125 and a corresponding local loss of 0.0875. Finally, the loss values ​​of all pixels within the local editing region are summed and averaged to obtain the final local editing feature loss.

[0074] Optionally, a self-supervised learning feature extraction model, including but not limited to mask autoencoders and contrastive learning models, can be used to replace dense feature extraction networks for local feature alignment in order to achieve feature consistency across modalities or views.

[0075] This embodiment provides a spatial 3D editing method based on sparse views. It automatically locates the editing region through multimodal embedding, eliminating the need for manual annotation of editing masks and reducing the annotation cost of training data. Simultaneously, through cross-view dense feature alignment, it constrains the consistency of the edited content from the pixel-level semantic feature dimension, ensuring uniformity in texture, color, and geometry across different viewpoints and avoiding texture breaks and semantic misalignment issues in the editing region.

[0076] Based on the second embodiment of this application, in the fifth embodiment of this application, the same or similar content as the second embodiment can be referred to the above description, and will not be repeated hereafter. On this basis, in order to force the model to deviate from the feature distribution of the original input during the editing process and avoid the generated result directly copying the original content due to laziness, after calculating the local editing feature loss and the global diffusion feature loss, before using them as the regularization term loss of the LoRA module, it is also necessary to calculate the feature difference (L1 loss) between the original four views and the generated four views in the intermediate layer of the diffusion model and take a negative value, that is, to maximize the feature distance between the two. This ensures that when the model optimizes the LoRA module, in addition to satisfying the local and global editing constraints, it must also actively generate perceptible feature changes, thereby preventing the mode from collapsing to the identity mapping and ensuring that the generated four views do indeed contain the expected editing effect, rather than simply remaining the same.

[0077] Therefore, before step S03, noise can be added to the four views generated by the editing generation model and the original input four views at a specific time step to form a noisy image, which is then input into the model. The corresponding intermediate diffusion features are extracted from a specific upsampling layer of the UNet network of the model. Finally, the L1 loss of the intermediate features of the original input four views and the intermediate features of the generated four views is calculated and multiplied by -1, so that the model can ensure that the generated four views are edited, rather than remaining the same as the original input.

[0078] Specifically, noise is added to the sparse view in the training samples based on a preset time step to obtain a noisy image. Then, the original diffusion features obtained after the noisy image passes through the UNet network of the generative model are determined. The L1 loss between the original diffusion features and the global diffusion features is calculated, and the L1 loss is multiplied by -1 so that the training stitched image deviates from the sparse view corresponding to the training samples.

[0079] For example, the model currently being trained edits a sparse view (e.g., a front view) of a regular chair into a chair with armrests. To prevent the model from being lazy and directly outputting the original image while ignoring editing instructions, a suitable amount of noise is added to the original chair view at a preset time step (e.g., t=500) during training, resulting in a noisy image. This noisy image is then input into a UNet, and intermediate features are extracted from an upsampling layer. The L1 distance between the two features is then calculated using the existing global diffusion feature loss and multiplied by -1. If the model generates an image almost identical to the original chair, or if it successfully generates a chair with armrests, the feature differences between the two become significant, and the loss changes considerably. In this case, to minimize the loss (tending towards a smaller negative number), the optimizer guides the model to actively amplify the feature differences, forcing the final generated view to deviate significantly from the original sparse view. This ensures that the armrest editing effect is truly reflected, rather than simply copying the input image.

[0080] Based on the first embodiment of this application, in the sixth embodiment of this application, the content that is the same as or similar to the first embodiment described above can be referred to the above description and will not be repeated hereafter. Based on this, the process of inputting multiple views into the 3D Gaussian reconstruction model in step S30 to obtain the 3D model corresponding to the object to be modeled includes steps S31 to S34: Step S31: Determine the area to be edited in the multi-view.

[0081] In this embodiment, as an optional implementation, the local region to be edited, i.e., the region to be edited, can be located first using Qwen3-VL-Embedding-8B similarity. This enables refined prediction of the editable region and rapid reuse of the non-editable region. For example, the multi-view image can be divided into 32×32 non-overlapping image blocks. The visual embedding vector of each image block can be extracted using the Qwen3-VL-Embedding-8B model. The cosine similarity between the visual embedding and the text embedding can be calculated, and image blocks with a similarity higher than 0.7 can be marked as editable regions.

[0082] Alternatively, other similarity calculation methods can be used to locate the area that needs to be edited, and this application does not limit this to any particular method.

[0083] Step S32: Extract multi-view features of the area to be edited.

[0084] In this embodiment, multi-view blocks can be embedded as visual words, and the camera intrinsic parameters corresponding to each view can be projected as learnable embeddings and concatenated with the visual words of the corresponding view to obtain target visual words. Then, based on the shared weight ViT encoder, the single-view visual features of the target visual words of each view are extracted. Finally, cross-view correlation modeling is performed on all single-view visual features according to the cross-view decoder to obtain multi-view features.

[0085] Specifically, the edited image can be divided into blocks, i.e., the region to be edited, and embedded as a visual token. At the same time, the camera's intrinsic and extrinsic parameters are encoded into learnable embedding vectors and concatenated with the image tokens. Then, these tokens are used to extract multi-view features through a ViT encoder with shared weights, and then information from different perspectives is fused through a cross-view decoder.

[0086] Step S33: Map the multi-view features to the center coordinates and Gaussian parameters of a 3D Gaussian based on the Gaussian parameter prediction head.

[0087] In this embodiment, the Gaussian parameter prediction head may include a Gaussian center head and a Gaussian parameter head. The Gaussian center head μ can be predicted by the Gaussian center head, and the Gaussian parameter head can predict Gaussian parameters such as opacity α, rotation r, scaling s, and spherical harmonic coefficient c.

[0088] Specifically, the Gaussian center head predicts the three-dimensional coordinates of each Gaussian point based on multi-view features. For example, if the Gaussian point is distributed on the handrail protrusion, the Gaussian parameter head predicts the color, opacity, and shape of the Gaussian point at that location.

[0089] Step S34: Based on the three-dimensional Gaussian center and Gaussian parameters, generate the edited three-dimensional Gaussian sputtering scene to obtain the three-dimensional model.

[0090] In this embodiment, the edited 3D Gaussian sputtering scene can be directly generated based on the center coordinates and Gaussian parameters to obtain a 3D model. This involves fusing the predicted 3D Gaussian of the edited region with the pre-stored original 3D Gaussian of the non-edited region to generate a complete edited 3DGS scene representation.

[0091] This embodiment uses camera intrinsic parameters as learnable embeddings and concatenates them with visual lexical units, enabling the model to perceive the projection relationship between different viewpoints. At the same time, based on the ViT encoder with shared weights, it ensures the consistency of feature extraction for each view, reduces the number of parameters and the risk of overfitting, thereby improving the geometric structure consistency of the editing area under multiple viewpoints and achieving high-quality 3D scene representation.

[0092] This application provides a 3D editing device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the spatial 3D editing method based on sparse views in the first embodiment described above.

[0093] The following is for reference. Figure 5 It shows a structural schematic diagram of a 3D editing device suitable for implementing the embodiments of this application. Figure 5 The 3D editing device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0094] like Figure 5 As shown, the 3D editing device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the 3D editing device. The processing unit 1001, the ROM 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the input / output interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the 3D editing device to communicate wirelessly or wiredly with other devices to exchange data. Although a 3D editing device with various systems is shown in the figure, it should be understood that it is not required to implement or possess all the systems shown. More or fewer systems may be implemented alternatively.

[0095] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0096] The 3D editing device provided in this application employs the sparse view-based spatial 3D editing method described in the above embodiments, which can solve the technical problem of poor realism in 3D modeling. Compared with the prior art, the beneficial effects of the 3D editing device provided in this application are the same as those of the sparse view-based spatial 3D editing method provided in the above embodiments, and other technical features of this 3D editing device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0097] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0098] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0099] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the spatial 3D editing method based on sparse views in the above embodiments.

[0100] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory (EPROM, or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0101] The aforementioned computer-readable storage medium may be included in the 3D editing device; or it may exist independently and not be assembled into the 3D editing device.

[0102] The aforementioned computer-readable storage medium carries one or more programs that, when executed by a 3D editing device, cause the 3D editing device to: Obtain multiple sparse views containing the object to be modeled, and text editing instructions in the sparse views for specifying the content to be edited; The stitched sparse view and the text editing instructions are input into the cross-view consistency trained image editing model to obtain a stitched image with global semantic consistency and local editing region alignment. The image editing model is obtained through self-supervised training of the generative model of the insertion parameter fine-tuning module. The stitched image is divided into multiple edit-consistent multi-view images, and the multi-view images are input into a 3D Gaussian reconstruction model to obtain the 3D model corresponding to the object to be modeled.

[0103] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0104] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0105] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0106] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described spatial 3D editing method based on sparse views, which can solve the technical problem of poor realism in 3D modeling. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the spatial 3D editing method based on sparse views provided in the above embodiments, and will not be repeated here.

[0107] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A spatial 3D editing method based on sparse views, characterized in that, The spatial 3D editing method based on sparse views includes: Obtain multiple sparse views containing the object to be modeled, and text editing instructions in the sparse views for specifying the content to be edited; The stitched sparse view and the text editing instructions are input into the cross-view consistency trained image editing model to obtain a stitched image with global semantic consistency and local editing region alignment. The image editing model is obtained through self-supervised training of the generative model of the insertion parameter fine-tuning module. The stitched image is divided into multiple edit-consistent multi-view images, and the multi-view images are input into a 3D Gaussian reconstruction model to obtain the 3D model corresponding to the object to be modeled.

2. The spatial 3D editing method based on sparse views as described in claim 1, characterized in that, The parameter fine-tuning module is a LoRA module. Before the step of obtaining multiple sparse views containing the object to be modeled, and the text editing instructions in the sparse views for specifying the editing content, the spatial 3D editing method based on sparse views further includes: Construct a training dataset, wherein each training sample contains a preset number of sparse views from different perspectives in the same scene, and a corresponding training text editing instruction; The training samples are input into the generative model of the LoRA module to generate a training stitched image with a preset layout pattern, wherein the preset layout pattern is associated with the preset quantity. The global diffusion feature loss and local editing feature loss of the training stitched image are determined as the regularization term loss of the LoRA module; The parameters of the LoRA module are iteratively updated until the global diffusion feature loss and the local editing feature loss converge together to obtain the image editing model.

3. The spatial 3D editing method based on sparse views as described in claim 2, characterized in that, Before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: Gaussian noise is added to the training stitched image based on a preset time step to obtain a noisy stitched image; The noisy stitched image is input into the UNet network of the generation model to obtain the first diffusion features corresponding to the left and right views and the second diffusion features corresponding to the top and bottom views in the noisy stitched image. The first diffusion feature and the second diffusion feature are subjected to feature alignment processing to obtain the global diffusion feature loss.

4. The spatial 3D editing method based on sparse views as described in claim 3, characterized in that, Before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: Noise is added to the sparse view in the training samples based on the preset time step to obtain a noisy image; Determine the original diffusion features obtained after the noisy image is processed by the UNet network of the generation model; Calculate the L1 loss between the original diffusion feature and the global diffusion feature, and multiply the L1 loss by -1 so that the training stitched image deviates from the sparse view corresponding to the training sample.

5. The spatial 3D editing method based on sparse views as described in claim 2, characterized in that, Before the step of determining the global diffusion feature loss and local editing feature loss of the training stitched image as the regularization term loss of the LoRA module, the spatial 3D editing method based on sparse views further includes: The trained stitched image is divided into multiple non-overlapping image blocks; An image encoder based on a preset vector embedding model extracts image features from each image patch, and a text encoder based on a preset vector embedding model extracts text features from the training text editing instructions; Calculate the cosine similarity between the features of each image patch and the text features, and locate the local editing region of the training stitched image based on the image patches whose cosine similarity is greater than a preset similarity. The dense features of the local editing region are extracted based on the dense feature extraction model; The dense features are subjected to feature alignment processing to obtain the local editing feature loss.

6. The spatial 3D editing method based on sparse views as described in claim 1, characterized in that, The steps of inputting the multi-view data into a 3D Gaussian reconstruction model to obtain the 3D model of the object to be modeled include: Determine the editable area of ​​the multi-view; Extract the multi-view features of the region to be edited; The multi-view features are mapped to the center coordinates and Gaussian parameters of a three-dimensional Gaussian based on the Gaussian parameter prediction head. Based on the center coordinates and the Gaussian parameters, an edited 3D Gaussian sputtering scene is generated, resulting in the 3D model.

7. The spatial 3D editing method based on sparse views as described in claim 6, characterized in that, The step of extracting the multi-view features of the region to be edited includes: The multi-view blocks are embedded as visual lexical units, and the camera intrinsic parameters corresponding to each view are projected as learnable embeddings and concatenated with the visual lexical units of the corresponding views to obtain the target visual lexical units. The ViT encoder based on shared weights extracts single-view visual features of target visual words in each view. The multi-view features are obtained by performing cross-view correlation modeling on all single-view visual features based on the cross-view decoder.

8. The spatial 3D editing method based on sparse views as described in claim 1, characterized in that, The step of inputting the stitched sparse view and the text editing instructions into an image editing model trained on cross-view consistency to obtain a stitched image with global semantic consistency and local editing region alignment, wherein the image editing model is obtained through self-supervised training of the generative model by the insertion parameter fine-tuning module, includes: The stitched sparse view and the text editing instructions are input into an image editing model trained on cross-view consistency, wherein the image editing model determines the editing module based on the number of stitches in the sparse view; Obtain the stitched image output by the editing module, which corresponds to the number of stitches in the sparse view and is semantically consistent globally and aligned with the local editing regions.

9. A 3D editing device, characterized in that, The 3D editing device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the spatial 3D editing method based on sparse views as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the spatial 3D editing method based on sparse views as described in any one of claims 1 to 8.

Images