Three-dimensional scene cross-domain migration method based on neural radiation field representation, computer equipment, readable storage medium and program product

Abstract

The invention relates to a three-dimensional scene cross-domain migration method based on neural radiation field representation, computer equipment, a readable storage medium and a program product. The three-dimensional scene cross-domain migration method comprises the steps that a real image is acquired for three-dimensional scene reconstruction; inputting the three-dimensional scene information into a geometric module to extract content features; inputting the three-dimensional scene information into a source domain coloring module to extract source domain style features; performing volume rendering to obtain a first rendered image, training and optimizing the geometric module and the source domain coloring module until the expectation is met, and freezing the geometric module; inputting the three-dimensional scene information into a target domain coloring module to extract target domain style features; performing volume rendering to obtain a second rendered image; performing style migration on the real image to obtain a stylized image; comparing loss training to optimize a target domain coloring module, and freezing the target domain coloring module until expectation is met; and receiving a specified view angle direction, respectively outputting features by using a geometric module and a target domain coloring module, and obtaining a target domain image through volume rendering.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning technology, and in particular to a method for cross-domain transfer of three-dimensional scenes based on neural radiation field representation, a computer device, a readable storage medium, and a program product. Background Technology

[0002] Currently, cross-domain transfer methods for 2D images can generate high-quality transfer results. CycleGAN (Cycle-Consistent Generative Adversarial Networks) proposes a recurrent generative adversarial network for cross-domain transfer of 2D images. CUT (Contrastive Learning for Unpaired Image-to-Image Translation) employs the NCE (Noise Contrastive Estimation) framework, which improves the quality of style-transferred images by maximizing the mutual information between the style-transferred image and the input image.

[0003] However, many challenges remain in style transfer tasks for 3D scenes. First, we cannot simply transfer real-world images to the target domain using 2D methods and then use these transferred images to reconstruct the 3D scene. This is because 2D methods do not consider style consistency between images from different viewpoints within a 3D scene, leading to a lack of consistent optimization objectives during 3D reconstruction and consequently, blurry reconstruction results. Similarly, we cannot apply 2D cross-domain transfer methods frame-by-frame to a series of rendered 3D scene images, as this would result in significant inconsistencies between the style images generated from different viewpoints.

[0004] In recent years, with the development of neural rendering technology, various 3D scene transfer methods based on neural representations have emerged. StyleNeRF (A Style-based 3D-Aware Generator for High-Resolution Image Synthesis) proposes an explicit modeling method for multi-view consistency style transfer, simultaneously optimizing a Neural Radiance Field (NeRF) and a stylized image generation module to generate images that possess both the target style and maintain geometric consistency. TeSTNeRF (Text-Driven 3D Style Transfer via Cross-Modal Learning) introduces textual guidance information to guide NeRF in transferring 3D scenes to a specified style domain. However, most of these methods extract style features from single images or text descriptions, typically focusing on artistic images. This means they cannot extract shared style features among multiple images in the target domain and rely on specific style images to provide style information.

[0005] In conclusion, maintaining stylistic consistency across different perspectives while achieving cross-domain 3D migration remains a challenging task. Summary of the Invention

[0006] Therefore, it is necessary to provide a method for cross-domain migration of three-dimensional scenes based on neural radiation field representation to address the above-mentioned technical problems.

[0007] This application presents a method for cross-domain migration of 3D scenes based on neural radiation field representation, including: Obtain real images from multiple perspectives of the same scene, perform 3D scene reconstruction, and obtain the reconstructed 3D scene based on neural radiation field representation. The 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction. A geometry module is provided, into which the 3D scene information is input to extract the content features of the 3D scene from the perspective of the real image; a source domain shading module is provided, into which the 3D scene information is input to extract the source domain style features of the 3D scene from the perspective of the real image; volume rendering is performed on the content features and the source domain style features to obtain a first rendered image; the loss of the first rendered image is compared with that of the real image; the geometry module and the source domain shading module are trained and optimized until the expected result is met, and then the geometry module is frozen. A target domain coloring module is provided. The 3D scene information is input into the target domain coloring module to extract the target domain style features of the 3D scene under the viewpoint direction of the real image. The content features and target domain style features are volume rendered to obtain a second rendered image. The real image is style transferred using a pre-trained 2D image cross-domain style transfer network to obtain a stylized image. The loss of the second rendered image and the stylized image are compared to train and optimize the target domain coloring module until the expected result is met, and then the target domain coloring module is frozen. The system receives the user's specified viewpoint direction for the 3D scene, and uses the frozen geometry module and the frozen target domain coloring module to output features and obtain the target domain image through volume rendering.

[0008] Optionally, a three-dimensional scene reconstruction is performed to obtain a reconstructed three-dimensional scene based on neural radiation field representation. Specifically, this includes: using the COLMAP tool to reconstruct the three-dimensional scene and obtaining the camera pose of the real image, thereby obtaining the reconstructed three-dimensional scene based on neural radiation field representation.

[0009] Optionally, a main module is provided, wherein the input of the main module is the three-dimensional scene information, and the output of the main module is simultaneously connected to the geometry module, the source domain shading module, and the target domain shading module.

[0010] Optionally, the geometry module, the source domain coloring module, the target domain coloring module, and the main module are each constructed using a multilayer perceptron.

[0011] Optionally, the main module has the output feature of simultaneously outputting to the geometry module, the source domain coloring module, and the target domain coloring module; The output features are combined with the viewpoint direction and then input into the source domain coloring module. The output features are combined with the viewpoint direction and then input into the target domain coloring module.

[0012] Optionally, the loss between the first rendered image and the real image can be compared using the following formula:

[0013] In the formula: , representing the loss between the first rendered image and the real image; ， This represents the sampled light rays of the three-dimensional scene. Indicates all A set; In the coarse sampling stage, which involves sampling in two phases, the pixel color of the sampled points after rendering. In the subsequent fine sampling stage of the two-stage sampling, the pixel color of the sampled point is rendered. The true color of the pixels in the real image; , representing the difference loss between the rendered color of the first rendered image and the real color of the real image during the previous coarse sampling stage; , which represents the difference loss between the rendered color of the first rendered image and the real color of the real image during the subsequent fine sampling stage.

[0014] Optionally, the loss of the second rendered image and the stylized image can be compared using the following formula:

[0015] In the formula: , representing the loss of the second rendered image and the stylized image; , indicating expectation; ， This represents the sampled light rays of the three-dimensional scene. Indicates all A set; , which represents the pixel color rendered after sampling points are sampled in the first coarse sampling stage of the two-stage sampling process; , representing the stylized image; , representing the stylized image Corresponding light By the color of the pixel; In the subsequent fine sampling stage, which involves sampling in two phases, the pixel color of the sampled points is rendered.

[0016] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional scene cross-domain migration method based on neural radiation field representation described in this application.

[0017] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the three-dimensional scene cross-domain migration method based on neural radiation field representation described in this application.

[0018] This application also provides a computer program product, including computer instructions that, when executed by a processor, implement the steps of the three-dimensional scene cross-domain migration method based on neural radiation field representation described in this application.

[0019] The 3D scene cross-domain migration method based on neural radiation field representation proposed in this application has at least the following effects: The process preceding the freezing of the geometry module in this application belongs to the geometry reconstruction stage. In the geometry reconstruction stage, the geometry module is trained using real images from multiple perspectives, enabling the geometry module to describe the geometric information of the 3D scene.

[0020] The process from freezing the geometry module to freezing the target domain coloring module constitutes the recoloring stage. In this stage, a pre-trained 2D image cross-domain style transfer network is used to generate a stylized image after cross-domain transfer. This image is then used to train the target domain coloring module, enabling it to describe the style information of the target domain. At this point, the frozen geometry module and the target domain coloring module can extract content features from the corresponding viewpoint of the 3D scene, and provide style features derived from the pre-trained 2D image cross-domain style transfer network, respectively.

[0021] The process following the freezing of the target domain coloring module belongs to the application phase. From any number of user-specified viewpoints, target domain images with consistent styles from different viewpoints are generated, conforming to both the source domain image content and the target domain image style. At this point, the generated target domain images are no longer limited by the angle and number of the original real images. Attached Figure Description

[0022] Figure 1 This is a flowchart illustrating a three-dimensional scene cross-domain migration method based on neural radiation field representation in one embodiment of this application. Figure 2 This is a schematic diagram of the scene representation network structure of a three-dimensional scene cross-domain transfer method based on neural radiation field representation in one embodiment of this application; Figure 3 for Figure 1 A schematic diagram of the network structure of a pre-trained 2D image cross-domain style transfer network; Figure 4 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0024] See Figure 1 and Figure 2 One embodiment of this application provides a method for cross-domain migration of three-dimensional scenes based on neural radiation field representation, including steps S100 to S400. Wherein: Step S100: Obtain real images from multiple perspectives in the same scene, perform 3D scene reconstruction, and obtain the reconstructed 3D scene based on neural radiation field representation. The 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction.

[0025] Step S200 includes steps S210 to S240: Step S210: Provide a geometry module, inputting 3D scene information into the geometry module to extract content features of the 3D scene from the perspective of the real image. Step S220: Provide a source domain shading module, inputting 3D scene information into the source domain shading module to extract source domain style features of the 3D scene from the perspective of the real image. Step S230: Perform volume rendering on the content features and source domain style features to obtain a first rendered image. Step S240: Compare the loss between the first rendered image and the real image, train and optimize the geometry module and the source domain shading module until the expected result is met, then freeze the geometry module.

[0026] Step S300 includes steps S210 to S330: Step S310, a target domain coloring module is provided, and 3D scene information is input into the target domain coloring module to extract the target domain style features of the 3D scene in the real image viewpoint direction. Step S320, volume rendering is performed on the content features and target domain style features to obtain a second rendered image; style transfer is performed on the real image using a pre-trained 2D image cross-domain style transfer network to obtain a stylized image. Step S330, the loss of the second rendered image and the stylized image are compared, and the target domain coloring module is trained and optimized until the expected result is met, after which the target domain coloring module is frozen.

[0027] Step S400: Receive the viewpoint direction specified by the user for the 3D scene, and use the frozen geometry module and the frozen target domain shading module to output features and obtain the target domain image through volume rendering.

[0028] In this embodiment, step S100 completes 3D scene reconstruction based on real images, step S200 is the geometric reconstruction stage, and step S300 is the recoloring stage. In the geometric reconstruction stage, this embodiment uses real images captured from multiple perspectives to train the geometry module, enabling it to describe the geometric information of the 3D scene. In the recoloring stage, a pre-trained 2D image cross-domain style transfer network is used to generate a stylized image after cross-domain transfer, thereby training the target domain coloring module to describe the style information of the target domain. During the execution of step S330, the transfer from 2D style domain information to 3D coloring information is gradually realized. After training, the target domain coloring module learns all the 3D coloring information.

[0029] The technical effect of this embodiment is mainly reflected in step S400. Step S400, based on the three-dimensional scene obtained in step S100, can directly generate target domain images with consistent style from any number of viewpoints specified by the user. At this time, the generated target domain images are no longer limited by the angle and number of the original real images. It can be understood that the target domain is the target image domain after the cross-domain migration is completed, and the original image domain is before the cross-domain migration.

[0030] In some embodiments, several optional sub-steps are provided for steps S100 to S400.

[0031] In step S100, 3D scene reconstruction is performed to obtain the reconstructed 3D scene based on neural radiation field representation. Specifically, this includes using the COLMAP tool to reconstruct the 3D scene and obtain the camera pose of the real images, thereby obtaining the reconstructed 3D scene based on neural radiation field representation. Specifically, given a set of N real images... This represents real images captured from N different camera perspectives in a real-world scene, and their corresponding camera poses are estimated using COLMAP (COLLISION-MAPping). This includes camera position and camera viewpoint, i.e., obtaining a 3D scene based on neural radiation field representation.

[0032] See Figure 2 For the geometry module, source domain coloring module, and target domain coloring module in steps S200 and S300, the 3D scene cross-domain transfer method based on neural radiation field representation includes providing a subject module. The subject module receives 3D scene information as input and simultaneously outputs features to the geometry module, source domain coloring module, and target domain coloring module. These output features are combined with the viewpoint direction and then input to the source domain coloring module. Similarly, these output features are combined with the viewpoint direction and then input to the target domain coloring module. Specifically, the geometry module, source domain coloring module, target domain coloring module, and subject module are each constructed using a multilayer perceptron.

[0033] It is understandable that the geometric module Source domain coloring module and target domain coloring module and main modules The entire scene representation network consists of four parts. The main module uses an 8-layer MLP, composed of 8 linear layers and the ReLU (Rectified Linear Unit) activation function, with each layer labeled from 0 to 7. The geometry module is a single linear layer, and the source domain shading module... and target domain coloring module Use a common linear layer Linear_0, and then connect each individual linear layer-activation function layer. Main module Accepts 3D coordinates and spatial coordinates from 3D scene information. 10th-order positional coding As input, the main module outputs a 256-dimensional feature vector. 10th-order positional encoding and The 63-dimensional vector obtained by splicing = ⊕ The input to the sixth layer, Linear_5, is the vector output from the fifth layer, ReLU_4 (the ReLU activation function). and Input by concatenation.

[0034] Geometry module Source domain coloring module and target domain coloring module All accept 256 dimensions As input, the outputs are content features (one-dimensional scalar σ) and source domain style features (three-dimensional vector), respectively. ) and target domain style features (3D vectors) The three output features represent volume density, source domain color, and target domain color, respectively.

[0035] Target domain coloring module Linear_1 accepts the output of Linear_0. And a concatenated vector of a 27-dimensional view vector d representing the view direction, and the positional encoding γ(d) of d. Input indicating the viewpoint direction. In other implementations, As input representing the viewpoint direction, the viewpoint vector d, as mentioned earlier, is the coordinate system in spherical coordinates. First, we need to transform the viewpoint direction from the spherical coordinate system to the Cartesian coordinate system. , , Then, position encoding and splicing are performed.

[0036] In step S320, the pre-trained 2D image cross-domain style transfer network uses the CycleGAN generator, with a ResNet-9 architecture. It can be broadly divided into three parts: a downsampling module, a residual module, and an upsampling module. The downsampling module maps the RGB image to a feature map, which is then processed through 9 layers of residual modules and finally upsampling modules to obtain the RGB image. Generally, deeper networks perform better than shallower networks, but as depth accumulates, gradients can increase or decrease exponentially, leading to gradient explosion or gradient vanishing. From a macroscopic perspective, gradient explosion causes parameters in deeper network layers to overflow, while gradient vanishing causes parameters in deeper image layers to remain almost unchanged. Gradient explosion can be addressed by limiting the upper bound of the gradient, while gradient vanishing requires the residual module to resolve. The strategy of the residual block involves adding the input feature x to the feature F(x) processed by the residual block before passing it into the next module. This skip connection strategy allows shallow feature information to be simultaneously transmitted to deep layers during forward inference, thus solving the gradient vanishing problem in deep neural networks. This enables the network to grow to greater depths, thereby improving network performance. The structure of the residual block is as follows: Figure 3 As shown, x represents the input of the pre-trained 2D image cross-domain style transfer network, h represents the output of the pre-trained 2D image cross-domain style transfer network, ⊕ represents matrix addition, and F(·) represents the residual block. The parameters of the pre-trained 2D image cross-domain style transfer network are frozen in both the geometric reconstruction stage and the recoloring stage.

[0037] ———————————————————————————— For the geometric reconstruction stage in step S200, the goal is to represent the underlying 3D scene based on multi-view images. In this stage, the subject module is trained using real images. Geometric module Source domain coloring module It does not use the target domain coloring module.

[0038] During training, a real image is randomly selected in each training round. and its camera pose and randomly select We have M pixels, and then we emit M rays starting from the center of the camera model, passing through these M pixels, denoted as MM. =1. For a single ray r, we can use Let represent the direction of the emitted ray, where o represents the center of the camera model, d represents the direction of the emitted ray, and t represents the division of a single ray r into t sampling segments. Then, Np sampling points are sampled along the direction of the ray, with a small-scale random perturbation added to avoid overfitting caused by sampling the same data in each training iteration. The sampling points are then processed through the position space coordinates... 10th-order positional coding Send to the main module Obtain the feature vector Then, the feature vectors are respectively... Input geometry module In this process, the volume density σ = Fvol(f) is obtained. The eigenvectors are then... The image is concatenated with the position-encoded viewpoint γ(d) and then sent to the source domain coloring module. In the middle, the output features are combined and concatenated with the viewpoint direction, and then input into the source domain coloring module to obtain the color. .in, Let x and d represent the coordinates of a point in space, d = (θ, ϕ) be the view vector representing the direction of the ray projection at that point, γ(d) be the positional encoding of d, and c = (r, g, b) be the color of that point. Before inputting the model, the coordinates x and d are first positionally encoded to increase the input dimension from low to high, thereby increasing the differences between different sampling points and preventing similar sampling points in space from producing similar outputs. The specific formula for positional encoding is as follows:

[0039] , represents the input of three-dimensional scene information, including three-dimensional coordinates (x, y, z) and a view vector (θ, φ) representing the view direction; , indicating position code; , indicating the number of position codes.

[0040] Finally, the color of the sampled points is integrated using volume rendering to obtain the rendered color of the target pixel. The equation for volume rendering is: , in

[0041] This represents the image obtained through volume rendering. Indicates the sampling light; , , respectively representing the sampling light Near and far boundaries; Indicates from arrive The transmittance accumulated along the sampled light ray, i.e., the transmittance of the sampled light ray from... arrive The probability of not hitting other particles; Represents the voxels used for rendering, where Indicates the number of segments in the sampled light beam. The sampling light ray represents the first part; Indicates the rendered color, where The sampling light ray represents the first part, The view vector represents the view direction; , This represents the distance between two adjacent sampling points. This indicates the sampling segment of the sampling ray where the sampling point is located. Represents voxels.

[0042] Since the sampled points are discrete, Equation 2 needs to be discretized.

[0043]

[0044] , representing the color of the pixel after the coarse-stage rendering; , indicating the first The color of each sampling point; , indicating the first Voxels at each sampling point; , representing the distance between adjacent sampling points; , indicates the sampling point.

[0045] To further improve sampling efficiency, this stage also employs a two-stage resampling strategy, consisting of a coarse stage and a fine stage, for further resampling and rendering. In the coarse stage, direct sampling is used first. The color of each sample is obtained through the aforementioned module, representing the color of the nth sample. The color of each sampling point And represents the first voxels at each sampling point , The color of the pixel after integration in the coarse stage is calculated using the above rendering equation. :

[0046]

[0047]

[0048] , representing the color of the pixel after the coarse-stage rendering; , indicating the number of samples; , represents the weight; , indicating the first The color of each sampling point; , indicating the first Voxels at each sampling point; , Indicates the distance between adjacent sampling points. Indicates the sampling point.

[0049] Weight After normalization, a piecewise constant along the ray is obtained. In this stage, this distribution is used as the probability density, and the newly added data is sampled using an inverse transform. Each sampling point. And will include the coarse stage. Each sampling point together Calculate all sampling points together This stage involves detailed calculations. The optimized loss function is:

[0050] , representing the loss between the first rendered image and the real image; ， Represents the sampled light rays of a 3D scene. Indicates all A set; In the coarse sampling stage, which involves sampling in two phases, the pixel color of the sampled points after rendering. In the subsequent fine sampling stage of the two-stage sampling, the pixel color of the sampled point is rendered. The true color of a pixel in a real image; , which represents the difference loss between the rendered color of the first rendered image and the real color of the real image in the previous coarse sampling stage; , which represents the difference loss between the rendered color of the first rendered image and the real color of the real image in the subsequent fine sampling stage.

[0051] ———————————————————————————— In the recoloring phase of step S300, the coloring module of the scene representation network is retrained to render the target domain image. In this phase, the target domain coloring module is trained using a 2D cross-domain transfer image. Data no longer flows through the source domain coloring module. The parameters of the subject module and geometry module are frozen and not modified to maintain the geometric structure of the scene.

[0052] This stage uses a two-dimensional image cross-domain transfer method. By pre-training a two-dimensional image cross-domain style transfer network, stylized images are generated from real images through cross-domain style transfer. Then, stylized images were used. As a supervised and guided scene representation network, it renders a second rendered image that conforms to the shading style of the target domain. Two-dimensional cross-domain style transfer methods need to generate images that maintain broad consistency, rather than consistency in detail. For example, the image's tone and style should remain consistent, but the texture can vary. For this reason, this paper chooses the generative network CycleGAN as the pre-trained two-dimensional image cross-domain style transfer network. This paper uses real images... Input a pre-trained 2D image into a cross-domain style transfer network to obtain a stylized image. ,in This represents a pre-trained 2D image cross-domain style transfer network. On the other hand, the target domain coloring module of the scene representation network renders a second rendered image with the corresponding pose. .

[0053] Similar to the reconstruction phase, MSE loss is used. This is used to measure the distance between the teacher's image and the student's image. The formula is as follows:

[0054] , representing the loss between the second rendered image and the stylized image; , indicating expectation; ， Represents the sampled light rays of a 3D scene. Indicates all A set; , which represents the pixel color rendered after sampling points are sampled in the first coarse sampling stage of the two-stage sampling process; , indicating a stylized image; , representing stylized images Corresponding light By the color of the pixel; In the subsequent fine sampling stage, which involves sampling in two phases, the pixel color of the sampled points is rendered.

[0055] Furthermore, this application uses an image discriminator D as an examiner to determine whether the output image belongs to the target domain. The image discriminator D adopts the PatchGAN structure proposed in pixel-to-pixel translation (Image-to-Image Translation with Conditional Adversarial Networks), which learns a unified style representation of the target style domain through the discriminator and optimizes the target domain colorization module in the scene representation network. Here, LeakyReLU(0.2) represents the LeakyReLU activation function (Leaky Rectified LinearUnit) with a negative half-axis slope of 0.2. InstanceNorm2d represents the instance normalization module, which performs in-channel normalization. Each N-channel pixel of the normalized feature map can represent the features of a certain sub-image block of the input image. The discriminator uses a fully convolutional network structure, accepting the image as input and outputting an N×N matrix. Each output in the matrix represents a receptive field of the original image, corresponding to a sub-block of the original image. In this way, a score is output indicating whether each sub-block of the image conforms to the target domain.

[0056] Due to the high resource requirements of NeRF rendering, a sparse sub-mesh is sampled from the grid of the teacher image. NeRF then renders only the pixels of this sparse sub-mesh to reduce computational resource demands. Since the sparse sub-mesh covers a large area of ​​the image, it can reflect the overall image information to a certain extent. The sub-mesh is denoted as […]. Ultimately, this paper adopts adversarial loss. To measure whether the output of the target domain coloring module conforms to the style of the target domain:

[0057] , indicating generator The losses incurred in the confrontation; , indicating the discriminator The losses incurred in the confrontation; , Indicates the sampling light. Represents the set of all sampled rays. Sampling; This indicates that a portion of the light obtained from the sampling passes through... The reconstructed image patch; , indicating the discriminator Discrimination of image patches; , representing the sampling of the stylized image generated by the pre-trained 2D image cross-domain style transfer network. The final loss for NeRF training in this paper is . ,in To adjust the weight of adversarial loss in the total loss, for example, a value of 2 can be used.

[0058] This application provides a method for cross-domain transfer of 3D scenes based on neural radiation field representation. The neural radiation field is used as the underlying 3D scene representation to ensure that the scene's geometry remains unchanged under different viewpoints. Correct shading module parameters are trained to enable NeRF to render an image belonging to the target domain. Specifically, in step S300, a pre-trained 2D image cross-domain style transfer network is used as an appearance feature extractor for the real image. The target domain shading module is trained through this pre-trained 2D image cross-domain style transfer network to transfer 2D domain style information to 3D shading information, ensuring consistency of the transfer results across multiple viewpoints. Furthermore, adversarial training is introduced to ensure that the style of the output image conforms to the target domain, enabling the generated stylized 3D rendered image to have consistency across multiple viewpoints. In step S400, the final goal is to generate a target domain image that conforms to both the source domain image content (real image content) and the target domain image style, based on any given viewpoint direction.

[0059] See Figure 4 In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When executed by the processor, the computer program implements a method for cross-domain migration of three-dimensional scenes based on neural radiation field representation. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0060] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the following steps: Step S100: Obtain real images from multiple perspectives of the same scene, perform 3D scene reconstruction, and obtain a reconstructed 3D scene based on neural radiation field representation. The 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction. Step S200 includes steps S210 to S240: Step S210: Provide a geometry module, input the 3D scene information into the geometry module to extract the content features of the 3D scene under the viewpoint direction of the real image. Step S220: Provide a source domain shading module, input the 3D scene information into the source domain shading module to extract the source domain style features of the 3D scene under the viewpoint direction of the real image. Step S230: Perform volume rendering on the content features and source domain style features to obtain a first rendered image. Step S240: Compare the loss of the first rendered image with that of the real image, train and optimize the geometry module and the source domain shading module until the expected result is met, and then freeze the geometry module. Step S300 includes steps S210 to S330: Step S310: Provide a target domain coloring module, inputting 3D scene information into the target domain coloring module to extract target domain style features of the 3D scene under the real image viewpoint direction. Step S320: Perform volume rendering on the content features and target domain style features to obtain a second rendered image; perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer network to obtain a stylized image. Step S330: Compare the losses of the second rendered image and the stylized image, train and optimize the target domain coloring module until the expected result is met, then freeze the target domain coloring module. Step S400: Receive the viewpoint direction specified by the user for the 3D scene, and use the frozen geometry module and the frozen target domain coloring module to output features respectively and obtain the target domain image through volume rendering.

[0061] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, it performs the following steps: Step S100: Obtain real images from multiple perspectives of the same scene, perform 3D scene reconstruction, and obtain a reconstructed 3D scene based on neural radiation field representation. The 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction. Step S200 includes steps S210 to S240: Step S210: Provide a geometry module, input the 3D scene information into the geometry module to extract the content features of the 3D scene under the viewpoint direction of the real image. Step S220: Provide a source domain shading module, input the 3D scene information into the source domain shading module to extract the source domain style features of the 3D scene under the viewpoint direction of the real image. Step S230: Perform volume rendering on the content features and source domain style features to obtain a first rendered image. Step S240: Compare the loss of the first rendered image with that of the real image, train and optimize the geometry module and the source domain shading module until the expected result is met, and then freeze the geometry module. Step S300 includes steps S210 to S330: Step S310: Provide a target domain coloring module, inputting 3D scene information into the target domain coloring module to extract target domain style features of the 3D scene under the real image viewpoint direction. Step S320: Perform volume rendering on the content features and target domain style features to obtain a second rendered image; perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer network to obtain a stylized image. Step S330: Compare the losses of the second rendered image and the stylized image, train and optimize the target domain coloring module until the expected result is met, then freeze the target domain coloring module. Step S400: Receive the viewpoint direction specified by the user for the 3D scene, and use the frozen geometry module and the frozen target domain coloring module to output features respectively and obtain the target domain image through volume rendering.

[0062] In one embodiment, a computer program product is provided, including computer instructions that, when executed by a processor, implement the following steps: Step S100, obtaining real images from multiple perspectives within the same scene, performing 3D scene reconstruction, and obtaining a reconstructed 3D scene based on neural radiation field representation, wherein the 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction. Step S200 includes steps S210 to S240: Step S210, providing a geometry module, inputting 3D scene information into the geometry module to extract content features of the 3D scene under the viewpoint direction of the real images. Step S220, providing a source domain shading module, inputting 3D scene information into the source domain shading module to extract source domain style features of the 3D scene under the viewpoint direction of the real images. Step S230, performing volume rendering on the content features and source domain style features to obtain a first rendered image. Step S240, comparing the loss between the first rendered image and the real image, training and optimizing the geometry module and the source domain shading module until the expected result is met, and then freezing the geometry module. Step S300 includes steps S210 to S330: Step S310: Provide a target domain coloring module, inputting 3D scene information into the target domain coloring module to extract target domain style features of the 3D scene under the real image viewpoint direction. Step S320: Perform volume rendering on the content features and target domain style features to obtain a second rendered image; perform style transfer on the real image using a pre-trained 2D image cross-domain style transfer network to obtain a stylized image. Step S330: Compare the losses of the second rendered image and the stylized image, train and optimize the target domain coloring module until the expected result is met, then freeze the target domain coloring module. Step S400: Receive the viewpoint direction specified by the user for the 3D scene, and use the frozen geometry module and the frozen target domain coloring module to output features respectively and obtain the target domain image through volume rendering.

[0063] In this embodiment, the computer program product includes a program code portion for performing the steps of the three-dimensional scene cross-domain migration method in the various embodiments of this application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. The computer program product may also be provided for download via a data network (e.g., via RAN, via the Internet, and / or via RBS). Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or application-specific integrated circuit (ASIC), or the functionality may be provided for download by means of a hardware description language.

[0064] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0065] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. When technical features of different embodiments are embodied in the same drawing, it can be regarded as the drawing also disclosing examples of combinations of the various embodiments involved.

[0066] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for cross-domain migration of 3D scenes based on neural radiation field representation, characterized in that, include: Obtain real images from multiple perspectives of the same scene, perform 3D scene reconstruction, and obtain the reconstructed 3D scene based on neural radiation field representation. The 3D scene information provided by the 3D scene includes 3D coordinates and viewpoint direction. A geometry module is provided, into which the 3D scene information is input to extract the content features of the 3D scene under the viewpoint direction of the real image; A source domain coloring module is provided, and the three-dimensional scene information is input into the source domain coloring module to extract the source domain style features of the three-dimensional scene in the view direction of the real image; Volume rendering is performed on the content features and the source domain style features to obtain a first rendered image. The loss of the first rendered image is compared with that of the real image. The geometry module and the source domain shading module are trained and optimized until the expected result is met, and then the geometry module is frozen. A target domain coloring module is provided, and the three-dimensional scene information is input into the target domain coloring module to extract the target domain style features of the three-dimensional scene under the viewpoint direction of the real image; The content features and target domain style features are volumetrically rendered to obtain a second rendered image; The real image is used to perform style transfer using a pre-trained two-dimensional image cross-domain style transfer network to obtain a stylized image; By comparing the loss of the second rendered image and the stylized image, the target domain coloring module is trained and optimized until the expected result is met, and then the target domain coloring module is frozen. The system receives the user's specified viewpoint direction for the 3D scene, and uses the frozen geometry module and the frozen target domain coloring module to output features and obtain the target domain image through volume rendering.

2. The three-dimensional scene cross-domain migration method as described in claim 1, characterized in that, Performing 3D scene reconstruction to obtain a reconstructed 3D scene based on neural radiation field representation specifically includes: using the COLMAP tool to perform 3D scene reconstruction and obtaining the camera pose of the real image, thereby obtaining the reconstructed 3D scene based on neural radiation field representation.

3. The three-dimensional scene cross-domain migration method as described in claim 1, characterized in that, A main module is provided, the input of which is the 3D scene information, and the output of which is simultaneously connected to the geometry module, the source domain shading module, and the target domain shading module.

4. The three-dimensional scene cross-domain migration method as described in claim 3, characterized in that, The geometry module, the source domain coloring module, the target domain coloring module, and the main module are each constructed using a multilayer perceptron.

5. The three-dimensional scene cross-domain migration method as described in claim 3, characterized in that, The main module has the characteristic of simultaneously outputting to the geometry module, the source domain coloring module, and the target domain coloring module; The output features are combined with the viewpoint direction and then input into the source domain coloring module. The output features are combined with the viewpoint direction and then input into the target domain coloring module.

6. The three-dimensional scene cross-domain migration method as described in claim 1, characterized in that, The loss between the first rendered image and the real image is compared using the following formula: In the formula: , representing the loss between the first rendered image and the real image; ， This represents the sampled light rays of the three-dimensional scene. Indicates all A set; In the coarse sampling stage, which involves sampling in two phases, the pixel color of the sampled points after rendering. In the subsequent fine sampling stage of the two-stage sampling, the pixel color of the sampled point is rendered. The true color of the pixels in the real image; , representing the difference loss between the rendered color of the first rendered image and the real color of the real image during the previous coarse sampling stage; , which represents the difference loss between the rendered color of the first rendered image and the real color of the real image during the subsequent fine sampling stage.

7. The three-dimensional scene cross-domain migration method as described in claim 1, characterized in that, The loss between the second rendered image and the stylized image is compared using the following formula: In the formula: , representing the loss of the second rendered image and the stylized image; , indicating expectation; ， This represents the sampled light rays of the three-dimensional scene. Indicates all A set; , which represents the pixel color rendered after sampling points are sampled in the first coarse sampling stage of the two-stage sampling process; , representing the stylized image; , representing the stylized image Corresponding light By the color of the pixel; In the subsequent fine sampling stage, which involves sampling in two phases, the pixel color of the sampled points is rendered.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the three-dimensional scene cross-domain migration method as claimed in any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the three-dimensional scene cross-domain migration method as described in any one of claims 1 to 7.

10. A computer program product, including computer instructions, characterized in that, When executed by a processor, the computer instructions implement the steps of the three-dimensional scene cross-domain migration method as described in any one of claims 1 to 7.

Images