Cross-domain three-dimensional scene combination editing method with consistent appearance characterization

Abstract

The invention provides a cross-domain three-dimensional scene combination editing method with consistent appearance characterization, and the method comprises the steps: training an appearance consistency alignment network through employing multi-view image data with appearance disturbance, enabling the network to extract the appearance characterization for an input target domain scene, converting the appearance characterization into color parameter adjustment for a source domain object, and enabling the color characterization to be more uniform. And the appearance characterization between the source domain target and the target domain scene is coordinated and consistent. And combining the 3D Gaussian model of the source domain object with the 3D Gaussian model of the target domain scene after the color parameters are adjusted to form a cross-domain combined 3D Gaussian model with consistent appearance characterization, generating a 3D combined scene rendering image with consistent appearance characterization through rendering, and completing a cross-domain three-dimensional scene combination editing task. Compared with a traditional three-dimensional scene editing method, the appearance representation consistency and the generalization ability of a cross-domain scene are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical fields of computer vision, computer graphics, and 3D editing, and particularly relates to a cross-domain 3D scene composition editing method with consistent appearance representation. Background Art

[0002] With the rapid development of virtual reality (VR), augmented reality (AR), and digital twin technologies, 3D scene modeling and editing have gradually become an important research direction in the fields of computer vision and graphics. The construction of 3D scenes has extensive application requirements in fields such as game design, film and television production, architectural visualization, and intelligent driving simulation. Especially under the promotion of the metaverse concept, how to quickly generate high-quality and realistic 3D scenes has become a key challenge.

[0003] Traditional 3D scene generation and editing methods mostly rely on manual modeling, which not only consumes a large amount of human and time resources but also limits the efficiency of large-scale scene generation. In recent years, with the development of deep learning and generative modeling technologies, 3D scene generation methods based on neural networks have gradually emerged, such as methods based on 3D point clouds, voxels, and implicit scene representations. These methods have improved the automation level of 3D scene generation and the ability to represent scene details through data-driven means.

[0004] Existing 3D scene composition editing methods mainly focus on the following categories:

[0005] 1. Scene editing based on geometric operations

[0006] This type of method completes object adjustment in the scene by directly operating on the geometric shapes of 3D objects, such as rotation, translation, scaling, etc. However, this method requires high professional skills from users and lacks the ability to automatically maintain global scene consistency, such as lighting and consistent material styles.

[0007] 2. Rule-driven scene generation and editing

[0008] Use domain-specific rules or templates for 3D scene composition editing, such as building generation rules or furniture layout algorithms. This method has good effects in specific scenarios, but its generality is poor and it is difficult to meet the requirements of open scenarios.

[0009] 3. Data-driven neural network methods

[0010] In recent years, generative adversarial networks and conditional generation technologies have been widely used in 3D scene generation tasks. Implicit expression methods such as neural radiance fields have demonstrated the ability to reconstruct and edit highly complex scenes. However, these methods often face problems such as the loss of consistent appearance representation (such as lighting and texture) during scene editing, and their ability to precisely control complex objects is also relatively limited.

[0011] 4. Scenario Composition Editing Based on Hybrid Method

[0012] Combining rule-driven and data-driven methods, the hybrid strategy attempts to balance editing efficiency and performance capabilities. For example, semantic-guided scenario editing allows users to adjust objects in the scenario through natural language descriptions, but there are still matching deviations between the generated content and the editing instructions. Summary of the Invention

[0013] In view of this, the present invention provides a cross-domain three-dimensional scenario composition editing method with consistent appearance representation. After being trained, the composition editing architecture designed by this method can be directly applied to various 3D scenario editing tasks, achieving global appearance consistency maintenance, multi-target editing accuracy improvement, and efficient 3D scenario generation and optimization with real-time interaction.

[0014] To solve the above technical problems, the present invention is implemented as follows.

[0015] A cross-domain three-dimensional scenario composition editing method with consistent appearance representation, comprising:

[0016] Step 1: Obtain multi-view image data of source domain objects, use an appearance perturbation mechanism to dynamically transform the source domain object images to generate multi-view image data with appearance perturbation; based on the multi-view image data with appearance perturbation, construct a 3D Gaussian model of the source domain objects;

[0017] Step 2: Construct an appearance consistency alignment network; this appearance consistency alignment network extracts the appearance feature latent variables of the multi-view image data with appearance perturbation, fuses the appearance feature latent variables with the color parameters of the 3D Gaussian model of the source domain objects to obtain weighted color parameters, and provides them to the 3D Gaussian model of the source domain objects;

[0018] Step 3: Train the appearance consistency alignment network and the 3D Gaussian model of the source domain objects simultaneously to obtain an appearance consistency alignment network and a 3D Gaussian model of the source domain objects with appearance self-adaptive capabilities;

[0019] Step 4: When cross-domain combining the source domain objects and the target domain scenario, use the appearance consistency alignment network to extract the appearance feature latent variables of the rendered image of the target domain scenario, fuse them with the color parameters of the 3D Gaussian model of the source domain objects trained in Step 3 to obtain the target domain feature weighted color parameters, adjust the appearance representation of the 3D Gaussian model of the source domain objects, and combine it with the 3D Gaussian model of the target domain scenario to form a cross-domain combined 3D Gaussian model with consistent appearance representation;

[0020] Step 5: According to the cross-domain combined 3D Gaussian model, render and generate a 3D combined scene rendered image with consistent appearance representation to complete the cross-domain three-dimensional scenario composition editing task.

[0021] Preferably, in step 1, when obtaining multi-view image data of the source domain object, any visible light imaging device is used for acquisition.

[0022] Preferably, in step 1, the appearance perturbation mechanism is implemented by matching a histogram feature database, which is composed of color histogram features extracted in advance from a large dataset and includes different illumination, hue, and contrast change patterns.

[0023] Preferably, the appearance consistency alignment network includes an appearance encoder and a feature decoder; the input of the appearance encoder is the multi-view image data with appearance perturbation and the rendered image of the target domain scene to be cross-domain combined; the output of the appearance encoder is the appearance feature latent variable; the input of the feature decoder is the concatenated vector of the appearance feature latent variable and the color parameters of the 3D Gaussian model of the source domain object, and the output is the weighted color parameters.

[0024] Preferably, the appearance encoder includes three feature extraction layers, each feature extraction layer includes a convolutional layer, an average pooling layer, and a batch normalization layer, and ReLU is used as the activation function; the dimensions of the three convolutional layers are 32, 64, and 128 respectively; the end of the appearance encoder is two linear layers with dimensions of 128 and 32 respectively, which are used to convert the extracted features into appearance latent variables with a dimension of 32.

[0025] Preferably, the feature decoder is a multi-layer perceptron network, which consists of 5 hidden layers with dimensions of 128, 256, 256, 128, and 6 respectively, and each hidden layer uses the ReLU activation function.

[0026] Preferably, when training the appearance consistency alignment network and the 3D Gaussian model of the source domain object simultaneously in step 3, the loss function is:

[0027]

[0028] In the formula, DSSIM represents the structural similarity metric difference loss function, |||| 1 represents the L1 loss function, represents the multi-view image data with appearance perturbation input to the appearance consistency alignment network, represents the rendered image of the 3D Gaussian model of the source domain object, λ dssim represents the trade-off coefficient.

[0029] Preferably, in step 4, the cross-domain combined 3D Gaussian model with consistent appearance representation is obtained by directly concatenating the parameters of the 3D Gaussian model of the source domain target and the 3D scene Gaussian model of the target domain scene.

[0030] Preferably, in step 5, according to the cross-domain combined 3D Gaussian model, the rendering of the 3D combined scene rendering image with consistent appearance representation is as follows: Use a rasterizer to render the cross-domain combined 3D Gaussian model with consistent appearance representation to generate a 3D scene rendering image with consistent appearance.

[0031] Beneficial effects:

[0032] (1) The architecture provided by the present invention realizes the consistency adjustment of appearance representations between the source domain target and the target domain scene by introducing an appearance perturbation mechanism and an appearance consistency alignment network based on an appearance encoder-decoder. Compared with traditional scene editing methods, this architecture can adapt to the dynamic changes of different lighting, hue, and texture conditions, significantly improving the generalization ability and visual consistency of cross-domain scene combination.

[0033] (2) The present invention realizes efficient 3D scene editing and rendering through the combination of 3D Gaussian hierarchical representation and a rasterizer. Using hierarchical Gaussian representation reduces the computational complexity of scene generation while maintaining the high precision and detail restoration ability of the model. Compared with the prior art, the present invention has a faster response speed in complex scene editing tasks and can better meet the application requirements of real-time interaction.

[0034] (3) The dynamic appearance perturbation mechanism proposed by the present invention dynamically transforms the source domain image by online matching a pre-constructed histogram feature database, providing rich appearance perturbation priors for the model. This mechanism effectively enhances the appearance adaptation ability of the source domain 3D Gaussian model and at the same time solves the appearance limitation problem caused by a single lighting condition in traditional 3D scene editing.

[0035] (4) The architecture of the present invention is applicable to a variety of 3D scene editing tasks, including virtual reality, game design, film and television production, and architectural visualization, etc. The cross-domain 3D scene combination with appearance consistency not only improves the visual effect of scene generation but also significantly simplifies the complexity of manual adjustment, providing an efficient solution for 3D content generation in multiple fields. Brief Description of the Drawings

[0036] Figure 1 is a flowchart of the architecture provided by the present invention;

[0037] Figure 2 is a schematic diagram of the training process of the source domain target 3D Gaussian model of the present invention;

[0038] Figure 3 is a schematic diagram of the cross-domain 3D Gaussian model combination process of the present invention;

[0039] Figure 4 is an example diagram of the rendering result of the cross-domain three-dimensional combined model of the present invention. Detailed Embodiments

[0040] The present invention will be described in detail below with reference to the accompanying drawings and by way of examples.

[0041] The application scenario of the present invention is to insert a certain target domain image in the source domain image into the target domain scene. For example Figure 2 , insert the vase on the table in the house in the source domain image onto the table outdoors in the target domain image. However, due to the differences in appearance characteristics such as brightness and color saturation between the source domain image and the target domain image, the combined effect is not good. Therefore, it is necessary to adjust the color of the inserted object to make it adapt to the scene of the target domain image and obtain a combined image with consistent appearance characteristics.

[0042] To this end, the present invention provides a cross-domain three-dimensional scene combination editing method with consistent appearance characteristics. The core idea is: by training an appearance consistency alignment network, it can extract the appearance characteristics of the input target domain scene and transform them into color parameter adjustments for the source domain object, so as to make the appearance characteristics between the source domain target and the target domain scene coordinated and consistent.

[0043] Figure 1 The flowchart of the cross-domain three-dimensional scene combination editing method with consistent appearance characteristics of the present invention is shown. As shown in the figure, the method includes the following steps:

[0044] Step 1: Obtain multi-view image data of the source domain object, use the appearance perturbation mechanism to dynamically transform the source domain object image, and generate multi-view image data with appearance perturbation. Based on the multi-view image data with appearance perturbation, construct a 3D Gaussian model of the source domain object.

[0045] In this step, when obtaining the multi-view image data of the source domain object, any visible light imaging device can be used to ensure that the complete geometric appearance of the object is covered.

[0046] In a preferred solution, the dynamic appearance perturbation mechanism uses a pre-constructed histogram feature database to dynamically perturb the multi-view image data to generate appearance perturbation images. Among them, the histogram feature database is composed of color histogram features extracted from a large dataset, containing rich illumination, hue, and contrast change patterns. The input image is online transformed by dynamically matching the histogram features in the database to generate multi-view image data with appearance perturbation.

[0047] Step 2: Construct an appearance consistency alignment network. The appearance consistency alignment network extracts the appearance feature latent variables of the multi-view image data with appearance perturbation, fuses the appearance feature latent variables with the color parameters of the 3D Gaussian model of the source domain object, obtains weighted color parameters, and provides them to the 3D Gaussian model of the source domain object.

[0048] See Figure 2, the appearance consistency alignment network consists of an appearance encoder and a feature decoder. Among them,

[0049] The input of the appearance encoder is the multi-view image data of appearance perturbation, and the output is the appearance feature latent variable, which reflects the appearance representation in the multi-view image data of appearance perturbation;

[0050] The input of the feature decoder is the concatenated vector of the appearance feature latent variable and the color parameter SH of the 3D Gaussian model of the source domain object src and the output is the adjusted weighted color parameter SH tgt :

[0051] SH tgt = D(E(I tgt ), SH src )

[0052] In the formula, I tgt represents the multi-view image data of appearance perturbation, E represents the encoding process of the appearance encoder, and E(I tgt ) represents the appearance feature latent variable output by the appearance encoder; D represents the processing of the feature decoder.

[0053] Among them, the input color parameter vector is extracted from the 3D Gaussian model of the source domain object constructed based on the multi-view image data of appearance perturbation; during training, the initial value of the color parameter vector is set to 0 and changes continuously during training until it is fixed as the color parameter vector used in actual combined editing after training.

[0054] In a preferred solution, the appearance encoder is a shallow feature extraction network, including three feature extraction layers, each layer containing a convolutional layer, an average pooling layer and a batch normalization layer; the dimensions of the three convolutional layers are 32, 64 and 128 respectively, the convolutional kernel size is 3×3, the stride is 1, the padding is 1, and the activation function is ReLU; two linear layers are connected to the end of the encoder, with dimensions of 128 and 32 respectively, and the features are converted into appearance latent feature variables with a dimension of 32. The feature decoder is a multi-layer perceptron network, including 5 hidden layers, with dimensions of 128, 256, 256, 128 and 6 respectively, and the ReLU activation function is used for all hidden layers.

[0055] Step 3: Train the appearance consistency alignment network and the 3D Gaussian model of the source domain object simultaneously to obtain an appearance consistency alignment network and a 3D Gaussian model of the source domain object with appearance adaptation ability.

[0056] See Figure 2 , and use the multi-view image data of appearance perturbation generated in Step 1 As the input of the appearance consistency alignment network, the appearance feature latent variable is extracted through the appearance encoder in the appearance consistency alignment network; the color parameter vector structure is extracted from the 3D Gaussian model of the source domain object constructed from the multi-view image data based on appearance perturbation, and the vector value is initialized to 0, that is, the initial color parameter. After being concatenated with the appearance feature latent variable, it is input into the feature encoder, and the feature encoder outputs the weighted color parameter. The color parameter of the 3D Gaussian model of the source domain object is adjusted by using the weighted color parameter, and then the rendered image is obtained by using the Gaussian renderer of the source domain object Using the rendered image and the multi-view image data of appearance perturbation Calculate the loss function value, perform backpropagation, adjust the parameters of the appearance consistency alignment network and the 3D Gaussian model of the source domain object, and complete one round of training. After multiple rounds of training, an appearance consistency alignment network and a 3D Gaussian model of the source domain object with appearance adaptation ability are obtained

[0057] In a preferred embodiment, the loss function is:

[0058]

[0059] In the formula, DSSIM (Difference of Structural Similarity Index) represents the structural similarity loss function, |||| 1 represents the L1 loss function, represents the multi-view image data of appearance perturbation input into the appearance consistency alignment network, represents the rendered image of the 3D Gaussian model of the source domain object, λ dssim represents the trade-off coefficient

[0060] Step 4: When cross-domain combining the source domain object and the target domain scene, the appearance feature latent variable of the rendered image of the target domain scene is extracted by using the appearance consistency alignment network, and is fused with the color parameter of the 3D Gaussian model of the source domain object trained in Step 3 to obtain the target domain feature weighted color parameter, adjust the appearance representation of the 3D Gaussian model of the source domain object, and combine it with the 3D Gaussian model of the target domain scene to form a cross-domain combined 3D Gaussian model with consistent appearance representation

[0061] See Figure 3, a target domain scene 3D Gaussian model is constructed based on the target domain scene image, and rendered to obtain a target domain scene rendered image. This image is input into the appearance encoder of the trained appearance consistency alignment network to obtain an appearance feature latent variable; this appearance feature latent variable is concatenated with the color parameters (source domain color parameters) of the source domain target 3D Gaussian model trained in step 3 and input into the feature decoder to obtain weighted color parameters, that is, the target domain feature weighted color parameters. The source domain object and the target domain scene are combined according to the user-defined combination instruction (3D coordinates or 3D target box) to obtain a cross-domain combined 3D Gaussian model. The color parameters of the source domain object part model in this cross-domain combined 3D Gaussian model adopt the target domain feature weighted color parameters output by the feature decoder.

[0062] Among them, a cross-domain combined 3D Gaussian model with consistent appearance representation can be obtained by directly concatenating the parameters of the source domain target 3D Gaussian model and the target domain scene 3D scene Gaussian model.

[0063] Step 5: Render and generate a 3D combined scene rendered image with consistent appearance representation according to the cross-domain combined 3D Gaussian model with consistent appearance representation, and complete the cross-domain three-dimensional scene combination editing task.

[0064] In this step, a rasterizer is used to render the cross-domain combined 3D Gaussian model with consistent appearance representation to generate a 3D scene rendered image with consistent appearance.

[0065] Example

[0066] In this example, an indoor room scene is selected as the target domain scene, and a pine tree building block is selected as the source domain object for experiments. The target domain scene is an indoor room scene containing basic furniture such as tables, chairs, and table lamps. The light source is a central ceiling chandelier with uniform brightness. The target domain scene is collected by a high-definition RGB camera surrounding the scene, with a shooting angle covering 360°, a total of 311 images, and an image resolution of 779×519. The target domain 3D Gaussian model is constructed based on the above data using the basic 3D Gaussian hierarchical method. The source domain object is a pine tree building block with a volume of approximately 10cm×10cm×15cm, mainly white and red in color, and brown and black at the bottom. The source domain object is collected by a high-definition RGB camera surrounding the scene, with a shooting angle covering 360°, a total of 292 images, and an image resolution of 780×520. The experiment uses the appearance perturbation mechanism described in step 1 to perform image perturbation. The histogram feature database is composed of histogram features extracted in advance from the Photo Tourism dataset, which can meet the extraction and learning requirements of the appearance consistency alignment network for perturbed image features.

[0067] Construct the appearance consistency alignment network as described in step 2, and train it using the training method described in step 3. In this example, the model is trained for 30,000 steps in total, and an appearance consistency alignment network with appearance self-adaptation capability and a source domain target 3D Gaussian model are obtained.

[0068] The cross-domain 3D Gaussian model combination is performed according to the method described in step 4. The cross-domain scene combination and rendering includes three stages: appearance feature adjustment, model combination, and scene rendering. First, the appearance feature E(I tgt ), and with the source domain color parameter SH src Combined, the decoder generates the adjusted weighted color parameter SH tgt ;

[0069] According to the user-defined 3D combination position, the adjusted pine tree block 3D Gaussian model is directly spliced ​​with the target domain scene 3D Gaussian model to generate a cross-domain combined 3D Gaussian model. Finally, the combined model is rendered using the target domain scene’s rasterizer to generate the final 3D scene rendered image.

[0070] The rendering results of the cross-domain combined 3D Gaussian model are as follows Figure 4 As shown in the figure. The visualization of the rendering results shows that the appearance characteristics of the source domain object (pine tree building blocks) in the target domain scene are consistent with the lighting and color tone of the target domain scene, which fully proves the effectiveness of appearance consistency adjustment. In terms of visual effects, the branches of the pine tree building blocks show a soft reflection effect under the indoor lighting conditions of the target domain scene, which matches the overall light intensity and color tone of the room; the texture of the brown part at the bottom is also consistent with the wood grain style of the desktop in the target domain scene, avoiding the phenomenon of abrupt colors or inconsistent materials. In addition, the edge transition area of ​​the source domain object (such as the junction of the leaves and the background) does not show unnatural silhouettes or reflection anomalies under the target domain lighting, indicating that appearance consistency adjustment can effectively handle light and shadow changes and ensure the fusion effect of the source domain object and the target domain scene. Especially in the shadow details of the local area, such as the contact between the bottom of the pine tree and the desktop, the shadow effect is soft and the transition between light and dark is natural, which further proves that the adjusted color parameters can adapt to the light and shadow characteristics of the target domain scene.

[0071] In summary, this embodiment verifies the effectiveness of the present invention in the task of adjusting the consistency of appearance representation and editing cross-domain scenes through the cross-domain combination experiment of indoor room scenes and pine tree blocks. The experimental results show that the rendering results fully verify the appearance consistency adjustment capability of the present invention in the cross-domain scene combination from multiple aspects such as lighting, color tone, material coordination, and multi-view consistency, providing effective technical support for the seamless integration of objects in the target domain and the source domain.

[0072] It should be understood that the above are only the preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A cross-domain three-dimensional scene combination editing method with consistent appearance representation, characterized in that: include: Step 1: Obtain multi-view image data of the source domain object, use the appearance perturbation mechanism to dynamically transform the source domain object image, and generate appearance perturbed multi-view image data; based on the appearance perturbed multi-view image data, construct a 3D Gaussian model of the source domain object; Step 2: Construct an appearance consistency alignment network; the appearance consistency alignment network extracts appearance feature latent variables of multi-view image data with appearance perturbations, fuses the appearance feature latent variables with the color parameters of the 3D Gaussian model of the source domain object, obtains weighted color parameters, and provides them to the 3D Gaussian model of the source domain object; Step 3: Train the appearance consistency alignment network and the 3D Gaussian model of the source domain object simultaneously to obtain the appearance consistency alignment network and the 3D Gaussian model of the source domain object with appearance adaptive capabilities; Step 4: When combining source domain objects and target domain scenes across domains, use the appearance consistency alignment network to extract the appearance feature latent variables of the target domain scene rendering image, and fuse them with the color parameters of the source domain object 3D Gaussian model trained in step 3 to obtain the target domain feature weighted color parameters, adjust the appearance representation of the source domain object 3D Gaussian model, and combine it with the target domain scene 3D Gaussian model to form a cross-domain combined 3D Gaussian model with consistent appearance representation; Step 5: Based on the cross-domain combined 3D Gaussian model, a 3D combined scene rendering image with consistent appearance representation is rendered to complete the cross-domain three-dimensional scene combination editing task.

2. The method according to claim 1, characterized in that In the step 1, when acquiring the multi-view image data of the source domain object, any visible light imaging device is used to acquire the data.

3. The method according to claim 1, characterized in that In step 1, the appearance perturbation mechanism is implemented by matching a histogram feature database, which is composed of color histogram features extracted in advance from a large data set, including different illumination, hue, and contrast change patterns.

4. The method according to claim 1, characterized in that The appearance consistency alignment network includes an appearance encoder and a feature decoder; the input of the appearance encoder is multi-view image data with appearance perturbations and a target domain scene rendering image that needs to be cross-domain combined; the output of the appearance encoder is an appearance feature latent variable; The input of the feature decoder is the concatenation vector of the appearance feature latent variable and the color parameters of the 3D Gaussian model of the source domain object, and the output is the weighted color parameters.

5. The method according to claim 4, characterized in that The appearance encoder includes three feature extraction layers, each of which includes a convolution layer, an average pooling layer, and a batch normalization layer, and uses ReLU as the activation function; the dimensions of the three convolution layers are 32, 64, and 128, respectively; the end of the appearance encoder is two linear layers with dimensions of 128 and 32, respectively, which are used to convert the extracted features into appearance latent variables with a dimension of 32.

6. The method according to claim 4, characterized in that The feature decoder is a multi-layer perceptron network consisting of 5 hidden layers with dimensions of 128, 256, 256, 128 and 6 respectively, and each hidden layer uses a ReLU activation function.

7. The method according to claim 1, characterized in that When the step 3 simultaneously trains the appearance consistency alignment network and the source domain object 3D Gaussian model, the loss function is: In the formula, DSSIM represents the structural similarity metric difference loss function, ||||1 represents the L1 loss function, Multi-view image data representing the appearance perturbations that are input to the appearance consistency alignment network, Rendered image representing the 3D Gaussian model of the source domain object, λ dssim Represents the trade-off coefficient.

8. The method according to claim 1, characterized in that In step 4, the cross-domain combined 3D Gaussian model with consistent appearance representation is obtained by directly splicing the parameters of the source domain target 3D Gaussian model and the target domain scene 3D scene Gaussian model.

9. The method according to claim 1, characterized in that In the step 5, rendering and generating a 3D combined scene rendering image with consistent appearance representation according to the cross-domain combined 3D Gaussian model is as follows: using a rasterizer to render the cross-domain combined 3D Gaussian model with consistent appearance representation to generate a 3D scene rendering image with consistent appearance.

Images