3D scene-based style cross-domain migration method, computer equipment and program product

Abstract

The invention relates to a style cross-domain migration method based on a 3D scene, computer equipment and a program product, and the method comprises the steps: obtaining a real image, and obtaining a three-dimensional scene comprising a three-dimensional voxel grid and a three-dimensional coloring grid through the reconstruction of the three-dimensional scene; extracting to obtain content features and color features, and inputting the color features into a source domain coloring module to obtain source domain style features; performing volume rendering to obtain a source domain rendering image; comparing loss training to optimize the three-dimensional scene, and freezing the three-dimensional scene until the expectation is met; inputting the random noise and the color features into a target domain coloring module to obtain target domain style features; performing volume rendering to obtain a target domain rendering image; performing style migration on the real image and injecting random noise to obtain a target domain stylized image; comparing loss training to optimize a target domain coloring module, and freezing the target domain coloring module until expectation is met; and receiving a camera pose specified by a user, and generating a new target domain rendering image under the camera pose.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning technology, and in particular to a method, computer device and program product for style cross-domain transfer based on 3D scenes. Background Technology

[0002] For 3D scene style transfer, an obvious approach is to first render an image from a specific viewpoint of the scene, and then input this 2D image into a 2D style transfer method for processing. However, because 2D methods lack prior knowledge of the 3D scene, they cannot guarantee consistency across multiple viewpoints during image generation. Similarly, performing cross-domain style transfer on the real-world scene image used for reconstruction before 3D reconstruction is also infeasible. This is because 3D reconstruction requires capturing images with high consistency across different viewpoints, a requirement that images processed by 2D cross-domain style transfer typically fail to meet. Therefore, cross-domain style transfer methods for 3D scenes require targeted design.

[0003] In recent years, with the development of neural rendering technology, various 3D scene transfer methods based on neural representations have emerged. StyleNeRF (A Style-based 3D-Aware Generator for High-Resolution Image Synthesis) proposes an explicit modeling method for multi-view consistency style transfer, simultaneously optimizing a Neural Radiance Field (NeRF) and a stylized image generation module to generate images that possess both the target style and maintain geometric consistency. TeSTNeRF (Text-Driven 3D Style Transfer via Cross-Modal Learning) introduces textual guidance information to guide NeRF in transferring 3D scenes to a specified style domain. However, most of these methods extract style features from single images or text descriptions, typically focusing on artistic images. This means they cannot extract shared style features among multiple images in the target domain and rely on specific style images to provide style information.

[0004] In conclusion, maintaining stylistic consistency across different perspectives while achieving cross-domain 3D migration remains a challenging task. Summary of the Invention

[0005] Therefore, it is necessary to provide a method for cross-domain style transfer based on 3D scenes to address the aforementioned technical problems.

[0006] This application presents a method for cross-domain style transfer based on 3D scenes, including: Obtain realistic images from multiple perspectives of the same scene, and reconstruct a 3D scene including a 3D voxel mesh and a 3D shaded mesh; Content features and color features are extracted from the 3D voxel mesh and the 3D shaded mesh, respectively. The color features are input into the provided source domain shading module to obtain source domain style features. Volume rendering is performed on the content features and the source domain style features to obtain a source domain rendered image. The loss of the source domain rendered image and the real image are compared to train and optimize the 3D scene until the expectation is met, and then the 3D scene is frozen. The random noise and the color features are input into the target domain coloring module to obtain target domain style features; volume rendering is performed on the content features and the target domain style features to obtain a target domain rendered image; style transfer is performed on the real image using a pre-trained two-dimensional image cross-domain style transfer model, and the random noise is injected during the style transfer process to output a target domain stylized image; the loss of the target domain rendered image and the target domain stylized image are compared to train and optimize the target domain coloring module until the expected result is met, and then the target domain coloring module is frozen. Receive the camera pose specified by the user for the 3D scene, and generate a new target domain rendering image under the camera pose.

[0007] Optionally, the color features are input into the provided source domain shading module to obtain source domain style features, specifically including: obtaining the spatial coordinates of the three-dimensional scene, concatenating the spatial coordinates with the color features, and then inputting the concatenated spatial coordinates into the provided source domain shading module to obtain source domain style features.

[0008] Optionally, random noise and the color features are input into the provided target domain coloring module to obtain target domain style features. Specifically, this includes: obtaining the spatial coordinates of the three-dimensional scene, and concatenating the spatial coordinates, the color features, and the random noise before inputting them into the provided target domain coloring module to obtain target domain style features.

[0009] Optionally, the spatial coordinates, the color features, and the random noise are concatenated and then input into the provided target domain coloring module. Specifically, this includes: concatenating the spatial coordinates, the color features, and the position-encoded random noise and then inputting them into the provided target domain coloring module.

[0010] Optionally, the random noise is injected during the style transfer process, specifically including: the pre-trained two-dimensional image cross-domain style transfer model includes a sequentially connected encoder and decoder, and the decoder is input with the random noise.

[0011] Optionally, the input of the random noise specifically includes: inputting the random noise multiplied by the diffusion coefficient.

[0012] Optionally, the pre-trained two-dimensional image cross-domain style transfer model includes content encoding connected between the encoder and the decoder, wherein the content encoding is scaled by scaling factors to change the degree of stylization.

[0013] Optionally, the style cross-domain transfer method includes: receiving a camera pose specified by the user for the 3D scene, and generating a new source domain rendered image under the camera pose; Both the source domain coloring module and the target domain coloring module are constructed using a multilayer perceptron. Both the source domain style feature and the target domain style feature are 3D RGB color vectors.

[0014] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the style cross-domain transfer method based on 3D scenes described in this application.

[0015] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the style cross-domain transfer method based on 3D scenes described in this application.

[0016] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the style cross-domain transfer method based on a 3D scene described in this application.

[0017] The method for cross-domain style transfer based on 3D scenes in this application has at least the following effects: After freezing the target domain shading module, this application generates a target domain rendering image from a new perspective of the 3D scene by receiving the specified camera pose during the application phase.

[0018] This application uses a pre-trained 2D image cross-domain style transfer model as a teacher network to guide the training of the target domain shading module of a 3D scene, so that the target domain rendered image under the new 3D perspective has stylistic consistency.

[0019] This application uses random noise to establish a correspondence between diverse target domain stylized images output by the teacher network and target domain rendered images, solving the problem of single style output images under specific style input conditions. In the application stage, for a specified camera pose, multiple target domain rendered images with diverse styles can be further generated. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a style cross-domain transfer method based on a 3D scene in one embodiment of this application; Figure 2 for Figure 1 Detailed flowchart; Figure 3 This is a model architecture diagram of the geometric reconstruction stage of a style cross-domain transfer method based on 3D scenes in one embodiment of this application; Figure 4 This is a model architecture diagram of the recoloring stage of a style cross-domain transfer method based on 3D scenes in one embodiment of this application; Figure 5 For application Figure 2 The diagram illustrates the results of the style cross-domain transfer method performed on the LLFF dataset to the styles of Ukiyo-e and Van Gogh, two artists. Figure 6 For application Figure 2 The diagram illustrates the results of the cross-domain style transfer method performed on the Tanks and Temples dataset to the styles of Ukiyo-e and Van Gogh. Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0022] See Figure 1 and Figure 2 One embodiment of this application provides a style cross-domain transfer method based on 3D scenes, including steps S100 to S400.

[0023] Step S100 (3D scene reconstruction stage): Obtain real images from multiple perspectives in the same scene, and obtain a 3D scene including a 3D voxel mesh and a 3D shaded mesh through 3D scene reconstruction.

[0024] Step S200 (Geometric Reconstruction Stage) specifically includes steps S210 to S230. Step S210 extracts content features and color features from the 3D voxel mesh and the 3D shaded mesh, respectively. Step S220 inputs the color features into the provided source domain shading module to obtain source domain style features. Step S230 performs volume rendering on the content features and source domain style features to obtain the source domain rendered image. Step S240 compares the loss between the source domain rendered image and the real image, trains and optimizes the 3D scene until the expected results are met, and then freezes the 3D scene.

[0025] Step S300 (Recoloring Stage) specifically includes steps S310 to S330. Step S310: Content features and color features are extracted from the 3D voxel mesh and the 3D shading mesh, respectively. Step S320: Random noise and color features are input into the provided target domain shading module to obtain target domain style features. Step S330: Volume rendering is performed on the content features and target domain style features to obtain the target domain rendered image. Step S340: Style transfer is performed on the real image using a pre-trained 2D image cross-domain style transfer model, and random noise is injected during the style transfer process to output a target domain stylized image. Step S350: The loss of the target domain rendered image and the target domain stylized image are compared, and the target domain shading module is trained and optimized until the expected result is met, after which the target domain shading module is frozen.

[0026] Step S400 (Application Stage): Receive the camera pose specified by the user for the 3D scene and generate a new target domain rendering image under the camera pose.

[0027] In this embodiment, the source domain is the image domain provided by the real image, and the target domain is the target image domain after cross-domain migration. Step S310 is the same as step S210, and is repeated for ease of understanding in this embodiment. After freezing the target domain shading module, this embodiment receives the specified camera pose to generate a target domain rendering image from a new perspective of the 3D scene.

[0028] This embodiment uses a pre-trained 2D image cross-domain style transfer model as a teacher network to guide the training of the target domain shading module of the 3D scene, so that the target domain rendered image under the new 3D perspective has stylistic consistency.

[0029] Furthermore, by using random noise to establish the correspondence between the diverse target domain stylized images output by the teacher network and the target domain rendered images, the problem of the output images having a single style under specific style input conditions is solved. In the application stage, for a specified camera pose, multiple target domain rendered images with diverse styles can be generated.

[0030] Step S100 belongs to the 3D scene reconstruction stage. 3D scene reconstruction can be based on DVGO (DirectVoxGO) to construct the underlying 3D scene, or on TensorRF (Tensor Radiance Field) to represent the 3D scene. These are described in detail below: The underlying 3D scene is constructed using DVGO, drawing from real images of the same scene from different perspectives, rather than relying solely on a simple MLP (Multi-Layer Perceptron) as the primary structure used in previous Nerf implementations. Building the underlying 3D scene using DVGO involves constructing a DVGO-based 3D scene representation from several real images of the same scene taken from different angles. This 3D scene representation includes a 3D voxel mesh. 3D shaded mesh Two 3D meshes and As a scene representation, in Represents a vector. These represent N points in each of the x, y, and z dimensions, respectively. (3D voxel mesh) 3D shaded mesh Store the volume density features (i.e., content features) and color features (specifically, color feature vectors) of each point in space. The 3D scene representation accepts spatial coordinate x, and uses trilinear interpolation to interpolate the volume density at coordinate x in the mesh. and color feature vector The volume density is processed through the softplus activation function to obtain the final volume density features used in volume rendering. : (1) This represents the volume density features that ultimately participate in volume rendering; , representing the interpolated volume density; , represents the offset, which is a hyperparameter, for example, 0.00001 to 0.001.

[0031] The 3D scene representation based on TensorRF also represents the 3D scene as a 3D voxel multichannel feature mesh G, and divides it into a volume density mesh in the channel dimension. (Equivalent to a 3D voxel mesh) and color feature mesh (Equivalent to a three-dimensional color grid), where Represents a vector. Let P represent the resolution in three spatial dimensions, where P represents the dimension of the color feature vector. Using the Vector-Matrix Decomposition (VM) proposed by TensorRF, the mesh is represented as a decomposable tensor. Through trilinear interpolation, the volume density features at any coordinate point in the scene can be obtained. (i.e., content features) and color feature vectors (i.e., color characteristics).

[0032] Combining examples of two 3D scene construction methods, in step S100, whether constructing the underlying 3D scene based on DVGO or the 3D scene representation based on TensorRF, both can provide spatial coordinates x and color features (color feature vectors). ), and content features (i.e., volume density features) ).

[0033] See Figure 3 Step S200 belongs to the geometric reconstruction stage. Step S220 involves inputting color features into the provided source domain shading module to obtain source domain style features. Specifically, this includes: obtaining the spatial coordinates x of the 3D scene, and then combining the spatial coordinates x with the color features (color feature vector). After splicing, input the provided source domain coloring module to obtain the source domain style features.

[0034] In the geometric reconstruction phase, a 3D scene representation for representing geometric content is trained and optimized using real images captured from a real scene and camera poses provided by the 3D scene. The camera poses include camera position and camera viewpoint. During this process, the 3D scene representation is continuously optimized until the expected loss is met. Further, step S200 includes step S250, which includes: receiving the camera pose specified by the user for the 3D scene and generating a new source domain rendered image under that camera pose. It is understood that the real images provided in step S100 represent a limited number of viewpoints. After the 3D scene representation optimization is complete, source domain rendered images with other viewpoints and source domain styles can be generated using specified camera poses.

[0035] See Figure 3 Step S300 belongs to the recoloring stage. This stage provides and optimizes the target domain coloring module. After the target domain coloring module is completed, the recoloring stage ends. Step S320 inputs random noise and color features into the provided target domain coloring module to obtain target domain style features. Specifically, this includes: obtaining the spatial coordinates of the 3D scene, and concatenating the spatial coordinates, color features, and position-encoded random noise before inputting them into the provided target domain coloring module.

[0036] In step S340, random noise is injected during the style transfer process. Specifically, this includes: a pre-trained two-dimensional image cross-domain style transfer model comprising a sequentially connected encoder and decoder, with random noise input to the decoder. The decoder input of random noise is specifically achieved by multiplying the random noise by a diffusion coefficient.

[0037] It is understandable that the coloring module includes two types: a source domain coloring module and a target domain coloring module. Since step S200 is executed before step S300, if step S250 is not required, the source domain coloring module can continue to be used as the target domain coloring module. Alternatively, a target domain coloring module independent of the source domain coloring module can also be provided. The source domain coloring module and the target domain coloring module are applied in two different stages: geometric reconstruction and recoloring, respectively. Both the source domain coloring module and the target domain coloring module are constructed using a multilayer perceptron and are only responsible for mapping from the high-dimensional input feature space to the RGB color space. Therefore, their depth is relatively shallow and they converge more easily. Both the source domain style features and the target domain style features are 3D RGB color vectors.

[0038] Specifically, color features Next, the spatial coordinate x and the noise vector encoded by position are used. After concatenation, the input vector of the target domain coloring module is obtained. Noise vector This is obtained by positionally encoding random noise. Input vector The input dimension is 62-dimensional, including 3-dimensional spatial coordinates x and 32-dimensional noise coding. and 27-dimensional color feature vector The output is a 3D RGB color vector. ReLU (Rectified LinearUnit) is used as the activation function in both shading modules, while a Sigmoid function is used before the final output to map the 3D RGB vector to (0, 1). It can be understood that the source domain style features output in the geometric reconstruction stage and the target domain style features output in the reshaping stage are both 3D RGB color vectors.

[0039] See Figure 4 The pre-trained 2D image cross-domain style transfer model serves as a teacher network. This network can transfer the style of a content image to the target domain and uses the style transfer results from different perspectives obtained by the teacher network as style references to pass to the underlying 3D scene constructed in step one. In the recoloring stage, the target domain coloring module is trained using the diverse transfer images generated by the teacher network, and diverse effects are produced through noise injection.

[0040] The teacher network employs a generator network structure based on a two-dimensional diversity image generation model, enabling it to simultaneously provide artistic styles from different works by the same artist. This allows the recoloring module to incorporate artistic styles from various works by the same artist. The teacher network uses an encoder-decoder structure, consisting of an encoder E(·) and a decoder D(·). The input features of each residual convolutional layer... After passing through this residual layer, the following is obtained: It needs to be added This content image is then used as input for the next residual layer. The encoder receives a 2D content image as input, performs convolution-regularization operations layer by layer, and outputs a content feature. This content feature, along with random noise z∼N(0,1) sampled from a standard normal distribution, is input into the decoder. Except for the first residual layer, which accepts the content feature as input, all other modules in the decoder concatenate the output feature of the previous layer with the 8-dimensional random noise z before inputting it into the next layer. Therefore, after noise concatenation, the input dimension of the next layer is 8 dimensions greater than the output dimension of the previous layer. In summary, the encoder consists of multiple residual layers, each inputting random noise z∼N(0,1). This hierarchical connection of noise structure enhances the impact of noise on the generator, thereby reducing the mode collapse problem to some extent.

[0041] Further, in step S340, the pre-trained 2D image cross-domain style transfer model includes content encoding connected between the encoder and decoder. This content encoding is scaled using scaling factors to change the degree of stylization. During the distillation process, the teacher network uses a method with controllable stylization to adjust the diversity of the images generated, thereby augmenting the distillation learning process by generating style transfer images with different degrees of stylization. This is mainly achieved through the content encoding between the teacher network's encoder and decoder. This is achieved through scaling. The scaling strategy is based on the input stylization level. , This indicates a uniform distribution, which is then transformed by a non-linear transformation module and activated by a softplus function. The result is then used as a scaling factor for content encoding. The images are scaled down to fine-tune the results generated by the teacher network, and then distilled to produce teacher images with different degrees of stylization.

[0042] In some implementations, a style encoder is provided. The method for calculating the loss between two images. Style encoder. To compute style features during the cross-domain style transfer stage, a shallow portion (e.g., the first four layers) of a pre-trained VGG network can be used as a style encoder. The output of the four convolutional modules of the style encoder is then used as the style features. The mean and variance of these features are calculated and concatenated as the style loss. By extracting features from different network layers, multi-scale feature representations can be obtained, which helps to capture the style of the image more comprehensively. Features from shallower layers help capture image details, while features from deeper layers help preserve the overall information of the image. Multi-level feature extraction can also enhance the non-linearity of the network to capture the style information of the image more finely. Using the mean and variance of the features can provide statistical information about the distribution of image features, maintaining the stability of the transfer results.

[0043] The training process includes training in two phases: geometric reconstruction and reshaping. In the reconstruction phase, images captured from the real scene and computed camera poses are used to train a 3D mesh representing the geometric information. The camera poses include camera position and viewpoint. In the reshaping phase, the target domain's coloring module is trained using diverse transfer images generated by the teacher network, and diverse effects are produced by injecting random noise.

[0044] In the geometric reconstruction phase, both the 3D scene representation methods DVGO and TensorRF can be represented using meshes, therefore... Represents a three-dimensional color grid. Represents a 3D voxel mesh. The source domain shading module uses... This indicates that the target domain coloring module uses In the geometric reconstruction of the 3D scene, a two-stage importance sampling strategy is employed to sample coordinate points in space along the ray direction, and then integrates them for volume rendering. This two-stage sampling strategy skips portions with near-zero volume density during geometric reconstruction, further shortening the training time.

[0045] In each round of training, the camera pose of a viewpoint from one of the training sets is first randomly selected. and the corresponding real image According to the camera pose Obtain the set of rays R, and sample the spatial coordinates x on the ray r. Then from the mesh and The volume density and color features of the eight vertices surrounding coordinate point x are queried separately, and the volume density feature of spatial coordinate point x is obtained by trilinear interpolation. and color features Then through the source domain coloring module ,according to The source domain's coloring result is obtained, resulting in the source domain's rendered image.

[0046] When using TensorRF as a 3D scene representation, the loss function uses the pixel-space MSE loss. The optimization goal is to make the rendered image closer to the real image. (2) , representing MSE loss (MSE, Mean Squared Error); This indicates that ray r belongs to the ray set R; , represents the color result of the volume rendering; , representing the color of the pixel in the target domain stylized image output by the teacher network from the real image.

[0047] During the recoloring stage, when using DVGO as the 3D scene representation, the loss function includes the pixel space MSE loss. There is also the loss of upsampling points on the same light beam. and background loss : (3) , indicating the upsampling point loss; , indicating that light passes through the first The probability of not being absorbed at each sampling point; , indicating the first Voxels at each sampling point; , Indicates the distance between adjacent sampling points. Indicates the sampling point; coordinates The resulting color; , representing the color of the pixel in the target domain stylized image output by the teacher network from the real image.

[0048] (4) , indicating background loss; , which represents the probability that light hits the background.

[0049] The aim is to make the results of sampling points on the same ray close to the final integral result, in order to help the network avoid getting trapped in local optima. This distinguishes the background from the foreground, preventing them from blending together. The overall loss function is: (5) , representing the total loss, , , For example, take 2, 1, and 1 respectively.

[0050] During the recoloring stage, the reconstructed 3D mesh , The parameters are frozen, and the parameters of the target domain coloring module are trained. In this stage, the target domain rendered image is obtained using volume rendering, and the target domain stylized image is obtained using a teacher network. First, an 8-dimensional noise vector z∼N(0,1) is sampled from a standard normal distribution. Then, the target domain stylized image is generated. and target domain rendering image At the same time, the noise vector z is simultaneously fed into the decoder of the teacher network and the target domain colorization module. To generate more diverse results from noise sampled from the same distribution, the noise z is multiplied by a diffusion coefficient before being input into the teacher network to expand its distribution range. This process can be represented as a target domain stylized image. .in Represents a real image. Representing random noise, the target domain renders the image. Where λ represents the diffusion coefficient, for example, it can be 2; D(·) represents the decoder of the teacher network; E(·) represents the encoder of the teacher network; VolmeRender(·) represents the volume rendering process.

[0051] The style cross-domain transfer method based on 3D scenes provided in the embodiments of this application can generate diverse target domain stylized images. Random noise Render image to target domain The correspondence can be obtained from the target domain coloring module, and is represented as follows: By Bayes' theorem The conditional probability of noise z is expressed as: : (6) This indicates random noise as defined by Bayes' theorem. Stylized images of the target domain The conditional probability; This indicates that the target domain stylized image is derived from the Bayesian formula. The conditional probability of random noise z; , representing the target domain stylized image The probability of; , representing random noise The probability of.

[0052] so, right The conditional probability can be obtained from the law of total probability: (7) , representing the target domain stylized image Render the image to the target domain The conditional probability; Render image to target domain The correspondence can be obtained from the target domain coloring module. This indicates that the Bayes formula is used to express the idea that Bayes' theorem is correct. Stylized images of the target domain The conditional probability.

[0053] This application targets stylized images. and target domain rendering image A correspondence was established between them, and the addition of random noise affected the rendering of the target domain image. It possesses a certain degree of diversity within the framework of the same artist's style. In the application phase of inference, the diversity transfer results can be obtained simply by randomly sampling noise. Specifically, for example... Figure 5 and Figure 6 The image shown is a 3x3 matrix of nine diverse target domain rendering images.

[0054] The loss function in the recoloring stage consists of two parts: the target domain rendered image and the loss function. and target domain stylized images MSE loss between and its style feature loss The MSE loss is the difference between two images in pixel space, as shown in Equation 8.

[0055] (8) , indicating MSE loss; This indicates that ray r belongs to the ray set R; , representing the light ray r and random noise The true color obtained through volume rendering specifically represents the color of the pixel corresponding to ray r after integration. , representing the target domain stylized image The color value of the pixel at ray r.

[0056] The loss in step S350, specifically the style feature loss, is the difference in features extracted by the pre-trained style feature extractor (i.e., the aforementioned style encoder), as shown in Equation 9, where µ(·) and σ(·) represent the mean and variance of the style features. Both losses use L2 distance as the metric.

[0057] (9) , indicating style feature loss , indicating the style encoder's first layer; , representing the target domain stylized image output by the teacher network; This indicates the target domain for rendering the image; , represents the mean; , representing variance; The overall loss function is: (10) in and The hyperparameter is the weighted value of the two losses, for example, 2 and 1 respectively.

[0058] This application provides various embodiments of a style cross-domain transfer method based on 3D scenes, which can generate 3D scenes with diverse styles while maintaining the consistency of the target domain. Specifically, it includes: (a) proposing a NeRF-based network structure, modifying the target domain coloring module in the radiation field to accept random noise as input, thereby supporting the generation of 3D scene images with diverse styles while maintaining the consistency of the appearance features of the target domain. (b) proposing a pre-trained 2D image cross-domain style transfer model as a teacher encoder network, which encodes the content image and the style transfer image respectively. By guiding the target domain coloring module to learn style mapping from random noise, it achieves the mapping from spatial coordinate features and random noise vectors to point color values, effectively supporting cross-domain style transfer tasks. (c) utilizing volume rendering technology, it integrates the sampled point colors into pixel colors to achieve the generation of stylized rendered images. Figure 5 and Figure 6 As shown, this method is applicable to the commonly used LLFF forward dataset (Local Light Field Fusion) and Tanks and Temples dataset, and it has good style fidelity and content consistency in 3D scene transfer tasks of art styles such as Ukiyo-e and Van Gogh.

[0059] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 7As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for style cross-domain transfer based on a 3D scene. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0060] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: Step S100: Obtain real images from multiple perspectives of the same scene, and obtain a 3D scene including a 3D voxel mesh and a 3D shading mesh through 3D scene reconstruction. Step S200: Specifically includes steps S210 to S230. Step S210: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S220: Input the color features into a provided source domain shading module to obtain source domain style features. Step S230: Perform volume rendering on the content features and source domain style features to obtain a source domain rendered image. Step S240: Compare the loss between the source domain rendered image and the real image, train and optimize the 3D scene until the expected result is met, and then freeze the 3D scene. Step S300: Specifically includes steps S310 to S330. Step S310: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S320: Input random noise and color features into the provided target domain coloring module to obtain target domain style features. Step S330: Perform volume rendering on the content features and target domain style features to obtain a target domain rendered image. Step S340: Perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer model, injecting random noise during the style transfer process to output a target domain stylized image. Step S350: Compare the losses of the target domain rendered image and the target domain stylized image, train and optimize the target domain coloring module until the expected results are met, then freeze the target domain coloring module. Step S400: Receive the camera pose specified by the user for the 3D scene, and generate a new target domain rendered image under the camera pose.

[0061] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: Step S100: Obtain real images from multiple perspectives of the same scene, and obtain a 3D scene including a 3D voxel mesh and a 3D shading mesh through 3D scene reconstruction. Step S200: Specifically includes steps S210 to S230. Step S210: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S220: Input the color features into a provided source domain shading module to obtain source domain style features. Step S230: Perform volume rendering on the content features and source domain style features to obtain a source domain rendered image. Step S240: Compare the loss between the source domain rendered image and the real image, train and optimize the 3D scene until the expected result is met, and then freeze the 3D scene. Step S300: Specifically includes steps S310 to S330. Step S310: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S320: Input random noise and color features into the provided target domain coloring module to obtain target domain style features. Step S330: Perform volume rendering on the content features and target domain style features to obtain a target domain rendered image. Step S340: Perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer model, injecting random noise during the style transfer process to output a target domain stylized image. Step S350: Compare the losses of the target domain rendered image and the target domain stylized image, train and optimize the target domain coloring module until the expected results are met, then freeze the target domain coloring module. Step S400: Receive the camera pose specified by the user for the 3D scene, and generate a new target domain rendered image under the camera pose.

[0062] In one embodiment, a computer program product is provided, including computer instructions that, when executed by a processor, implement the following steps: Step S100: Obtain real images from multiple perspectives within the same scene, and reconstruct a 3D scene including a 3D voxel mesh and a 3D shading mesh through 3D scene reconstruction. Step S200: Specifically includes steps S210 to S230. Step S210: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S220: Input the color features into a provided source domain shading module to obtain source domain style features. Step S230: Perform volume rendering on the content features and source domain style features to obtain a source domain rendered image. Step S240: Compare the loss between the source domain rendered image and the real image, train and optimize the 3D scene until the expected result is met, and then freeze the 3D scene. Step S300: Specifically includes steps S310 to S330. Step S310: Extract content features and color features from the 3D voxel mesh and the 3D shading mesh, respectively. Step S320: Input random noise and color features into the provided target domain coloring module to obtain target domain style features. Step S330: Perform volume rendering on the content features and target domain style features to obtain a target domain rendered image. Step S340: Perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer model, injecting random noise during the style transfer process to output a target domain stylized image. Step S350: Compare the losses of the target domain rendered image and the target domain stylized image, train and optimize the target domain coloring module until the expected results are met, then freeze the target domain coloring module. Step S400: Receive the camera pose specified by the user for the 3D scene, and generate a new target domain rendered image under the camera pose.

[0063] In this embodiment, the computer program product includes program code portions for performing the steps of the style cross-domain transfer method based on a 3D scene as described in the embodiments of this application, when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. It may also be provided for download via a data network (e.g., via a RAN, via the Internet, and / or via an RBS). Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or its functionality may be provided for download by means of a hardware description language.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0065] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. When technical features of different embodiments are embodied in the same drawing, it can be regarded as the drawing also disclosing examples of combinations of the various embodiments involved.

[0066] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-domain style transfer based on 3D scenes, characterized in that, include: Obtain realistic images from multiple perspectives of the same scene, and reconstruct a 3D scene including a 3D voxel mesh and a 3D shaded mesh; Content features and color features are extracted from the three-dimensional voxel mesh and the three-dimensional shaded mesh, respectively. The color features are then input into the provided source domain shader module to obtain source domain style features. Volume rendering is performed on the content features and the source domain style features to obtain a source domain rendered image; the loss of the source domain rendered image and the real image are compared to train and optimize the 3D scene until the expected result is met, and then the 3D scene is frozen. The random noise and the color features are input into the provided target domain coloring module to obtain the target domain style features; Volume rendering is performed on the content features and the target domain style features to obtain a target domain rendered image; The real image is style transferred using a pre-trained two-dimensional image cross-domain style transfer model, and the random noise is injected during the style transfer process to output a target domain stylized image. By comparing the loss of the target domain rendered image and the target domain stylized image, the target domain coloring module is trained and optimized until the expected result is met, and then the target domain coloring module is frozen. Receive the camera pose specified by the user for the 3D scene, and generate a new target domain rendering image under the camera pose.

2. The style cross-domain transfer method as described in claim 1, characterized in that, The color features are input into the provided source domain shading module to obtain source domain style features. Specifically, this includes: obtaining the spatial coordinates of the three-dimensional scene, concatenating the spatial coordinates with the color features, and then inputting the concatenated spatial coordinates into the provided source domain shading module to obtain source domain style features.

3. The style cross-domain transfer method as described in claim 1, characterized in that, The random noise and the color features are input into the provided target domain coloring module to obtain target domain style features. Specifically, this includes: obtaining the spatial coordinates of the three-dimensional scene, and concatenating the spatial coordinates, the color features, and the random noise before inputting them into the provided target domain coloring module to obtain target domain style features.

4. The style cross-domain transfer method as described in claim 3, characterized in that, The process of concatenating the spatial coordinates, the color features, and the random noise and then inputting them into the provided target domain coloring module specifically includes: concatenating the spatial coordinates, the color features, and the position-encoded random noise and then inputting them into the provided target domain coloring module.

5. The style cross-domain transfer method as described in claim 1, characterized in that, The random noise is injected during the style transfer process, specifically including: the pre-trained two-dimensional image cross-domain style transfer model includes a sequentially connected encoder and decoder, and the decoder is input with the random noise.

6. The style cross-domain transfer method as described in claim 5, characterized in that, The input random noise specifically includes: input random noise multiplied by a diffusion coefficient.

7. The style cross-domain transfer method as described in claim 5, characterized in that, The pre-trained 2D image cross-domain style transfer model includes content encoding connected between the encoder and decoder, which is scaled by scaling factors to change the degree of stylization.

8. The style cross-domain transfer method as described in claim 1, characterized in that, The style cross-domain transfer method includes: receiving a camera pose specified by the user for the 3D scene, and generating a new source domain rendered image under the camera pose; Both the source domain coloring module and the target domain coloring module are constructed using a multilayer perceptron. Both the source domain style feature and the target domain style feature are 3D RGB color vectors.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the style cross-domain transfer method based on any one of claims 1 to 8.

10. A computer program product, including computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the style cross-domain transfer method based on a 3D scene as described in any one of claims 1 to 8.

Images