Three-dimensional scene style transfer method, three-dimensional scene style transfer system and computer device

Abstract

The application relates to a three-dimensional scene style transfer method, system and computer device, which is used for transferring the style of a style image to a three-dimensional scene and keeping the geometric structure of the three-dimensional scene. The three-dimensional scene style transfer method comprises the following steps: based on a neural radiation field three-dimensional scene, using a voxel grid to model a style mode space and a geometric space, obtaining a geometric body of the neural radiation field in the geometric space, obtaining a spherical coordinate of the neural radiation field in the style mode space and an appearance in a viewpoint direction; using a noise image to extract a style feature map of the style image, combining a prompt vector to correct the style feature map, decoding the corrected style feature map to obtain a style mode; corresponding the spherical coordinate to the style mode, coupling the color of the appearance and the color of the style mode to obtain a stylized color; transferring the stylized color to the geometric body, rendering to obtain a three-dimensional style transfer rendering image in the viewpoint direction.

Description

Technical Field

[0001] This application relates to the fields of computer vision and deep learning, and in particular to a 3D scene style transfer method, a 3D scene style transfer system, and a computer device. Background Technology

[0002] 3D scene stylization is an important editing task in the visual and graphics fields, enabling the creation of novel artistic scenes. Given a 3D scene and a style image, it generates a stylized image of the scene from an arbitrarily novel perspective, based on the artistic style of the style reference. Due to the lack of 3D information, naively applying image / video stylization methods to 3D scenes often yields inconsistent results. To address this inconsistency, existing methods have explored 3D scene stylization based on explicit 3D models (e.g., meshes, voxels, and point clouds). However, their discrete representation of the scene leads to a loss of geometric accuracy.

[0003] Neural Radiance Fields (NeRF) proposes using neural networks for continuous scene modeling. Due to its excellent performance in reconstructing geometry and appearance, it has become a superior scene representation. Because of its reliability, Stylizing-3D-Scene was first introduced for 3D scene stylization in "Stylizing 3D Scene via Implicit Representation and HyperNetwork" (2022, Pei-Ze Chiang, Meng-Shiun Tsai, Hung-Yu Tseng, Wei-Sheng Lai, Wei-Chen Chiu). It fixes the scene's geometry and only modifies the appearance by using a hypernetwork to predict parameters used to compute the artistic look. Several methods have been developed to further improve the visual quality of stylization. However, despite these valuable efforts, these methods still become somewhat entangled with geometry and appearance, thus requiring retraining on new scenes. Summary of the Invention

[0004] Therefore, it is necessary to provide a 3D scene style transfer method to address the aforementioned technical problems.

[0005] This application presents a 3D scene style transfer method for transferring the style of a style image to a 3D scene while preserving the geometric structure of the 3D scene, including:

[0006] Based on the 3D scene of the neural radiation field, voxel meshes are used to model the style mode space and geometric space to obtain the geometry of the neural radiation field in the geometric space, and to obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the view direction.

[0007] Style feature maps of style images are extracted from noisy images, and the style feature maps are corrected by combining cue vectors. The style pattern is obtained by decoding the corrected style feature maps.

[0008] Map the spherical coordinates to the style pattern, and couple the color of the appearance with the color of the style pattern to obtain stylized colors;

[0009] The stylized colors are transferred to the geometry, and a 3D style-transferred rendering image is obtained from the viewpoint.

[0010] Optionally, obtaining the spherical coordinates of the neural radiation field in style pattern space and its appearance in the viewpoint direction specifically includes:

[0011] The spherical coordinates of the neural radiation field in style pattern space are obtained by applying UV mapping;

[0012] By combining the three-dimensional scene of the neural radiation field in the viewpoint direction, a multilayer perceptron is applied to extract the appearance of the neural radiation field in the viewpoint direction.

[0013] Optionally, style feature maps of the style image can be extracted from the noisy image, specifically including:

[0014] The noisy image and the style image are sequentially input into the first encoder and the trained first converter to obtain the style feature map;

[0015] Decoding the corrected style feature map yields the style pattern, specifically including:

[0016] The style pattern is obtained by decoding the corrected style feature map using the first decoder.

[0017] Optionally, mapping the spherical coordinates to the style mode specifically includes:

[0018] Map the spherical coordinates to cube coordinates;

[0019] By using different noisy images, style feature maps of style images are extracted multiple times to obtain multiple style patterns;

[0020] Map the cube coordinates to the multiple style patterns.

[0021] Optionally, mapping the spherical coordinates to the style mode specifically includes:

[0022] Map the spherical coordinates to cube coordinates;

[0023] Using six different noisy images, style feature maps of the style images are extracted multiple times to obtain six style patterns;

[0024] The six faces of the cube coordinates are respectively mapped to the six style modes.

[0025] Optionally, the color of the appearance and the color of the style mode are coupled to obtain stylized colors, specifically including:

[0026] The stylized color is obtained by performing a multiplication operation on the color of the appearance and the color of the style mode.

[0027] Optionally, the cue vector is obtained through training and fine-tuning, and the training and fine-tuning process of the cue vector includes:

[0028] Obtain an initial viewpoint image of the neural radiation field in the viewpoint direction, the initial viewpoint image having geometry in the viewpoint direction and appearance in the viewpoint direction;

[0029] The style of the style image is transferred to the initial viewpoint image to obtain a two-dimensional style-transferred image;

[0030] The loss difference between the obtained 3D style transfer rendered image and the 2D style transfer image is compared, and the cue vector is fine-tuned using the loss difference.

[0031] Optionally, the style of the style image is transferred to the initial viewpoint image to obtain a two-dimensional style-transferred image, specifically including:

[0032] The initial viewpoint image and the style image are input sequentially into the second encoder, the trained second converter, and the second decoder to obtain a two-dimensional style transfer image.

[0033] This application provides a 3D scene style transfer system for transferring the style of a style image to a 3D scene while preserving the geometric structure of the 3D scene, including:

[0034] The neural radiation field extraction module is used to model the style mode space and geometric space of a 3D scene based on the neural radiation field using voxel meshes, obtain the geometry of the neural radiation field in the geometric space, and obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction.

[0035] The style pattern extraction module is used to extract style feature maps from the noisy image, correct the style feature maps by combining them with cue vectors, and decode the corrected style feature maps to obtain style patterns.

[0036] The appearance mapping module is used to map the spherical coordinates to the style pattern, and couple the color of the appearance with the color of the style pattern to obtain stylized colors;

[0037] The stylization mapping module is used to transfer the stylized colors to the geometry and render a 3D style-transferred rendering image in the viewpoint direction.

[0038] This application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional scene style transfer method described in this application.

[0039] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the three-dimensional scene style transfer method described in this application.

[0040] The 3D scene style transfer method of this application has at least the following effects:

[0041] This application uses voxel meshes to model the style pattern space and geometric space based on the three-dimensional coordinates of the neural radiation field and the viewpoint direction, thereby decoupling the appearance of the 3D scene from its geometry. This application maps spherical coordinates to the style pattern and obtains stylized colors for transfer to the geometry through a mapping retrieval of the spherical surface to the style pattern, thus achieving stylization of the 3D scene's appearance.

[0042] This application introduces cue vectors based on existing two-dimensional image style transfer methods to modify style feature maps, which can stylize the appearance more harmoniously. The cue vectors can be initialized with random values ​​and fine-tuned during training.

[0043] This application improves the style transfer effect by comparing the loss difference between 3D style transfer rendered images and 2D style transfer images under the same viewpoint direction, and fine-tuning the training cue vector. Attached Figure Description

[0044] Figure 1 This is an implementation model architecture diagram of a three-dimensional scene style transfer method in one embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating a three-dimensional scene style transfer method in one embodiment of this application;

[0046] Figure 3 This is a schematic diagram illustrating the application of UV mapping in one embodiment of this application, performing bijective mapping between real space (x, y, z) and style pattern space (u, v);

[0047] Figure 4This is a schematic diagram of the process of mapping the spherical coordinates to the cube coordinates in one embodiment of this application (illustration of sphere to cube texture retrieval);

[0048] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0049] Current style transfer methods have achieved good results in transferring content from two-dimensional images. However, for three-dimensional scene content, existing methods require training style transfer networks for different three-dimensional scenes one by one, resulting in poor transfer quality and often causing problems such as blurring and poor three-dimensional consistency in the transferred scenes.

[0050] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0051] See Figures 1-3 One embodiment of this application provides a three-dimensional scene style transfer method for transferring the style of a style image Is to a three-dimensional scene while preserving the geometric structure of the three-dimensional scene. That is, the style of any style image can be transferred to any three-dimensional scene while maintaining the content of the three-dimensional scene. The three-dimensional scene style transfer method includes steps S100 to S400.

[0052] Step S100: Based on the 3D scene of the neural radiation field, use voxel mesh to model the style mode space and geometric space to obtain the geometry of the neural radiation field in the geometric space, and obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction.

[0053] The 3D implicit representation model based on Neural Radiance Fields (NeRF) uses neural radiance fields to represent 3D scenes. However, the appearance and geometry of the neural radiance fields are intertwined and difficult to edit. The goal of 3D scene style transfer is to stylize the appearance of the scene while preserving the geometry. Therefore, a key requirement is to decouple the appearance from the geometry. This embodiment uses voxel meshes to model the style pattern space and geometric space instead of the traditional multilayer perceptron, which can significantly accelerate training while maintaining reconstruction quality.

[0054] like Figure 1 As shown, this application is based on the three-dimensional coordinates (x, y, z) of the neural radiation field and the viewpoint direction. Modeling is then performed. This involves using 3D coordinates and view orientation (viewpoint direction) as input to decouple the appearance of the 3D scene from its geometry. It can be understood that the neural radiation field in the viewpoint direction can be used to obtain the appearance of the 3D scene.

[0055] In step S100, the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction are obtained. Specifically, this includes: applying UV mapping to obtain the spherical coordinates of the neural radiation field in the style mode space, combining the three-dimensional scene of the neural radiation field in the viewpoint direction, and applying a multilayer perceptron (MLP) to extract the appearance of the neural radiation field in the viewpoint direction.

[0056] This embodiment uses a 2D unit sphere as the style pattern space, where u and v represent the azimuth and elevation angles within the unit sphere, respectively. This embodiment projects the real-world coordinates of the 3D scene onto a UV mapping module in the 2D (UV) style pattern space, separating the appearance from the geometry. Projecting the real-world 3D coordinates onto the 2D (UV) style pattern space further explicitly de-entangles the appearance from the geometry. Through this de-entanglement, the color of a point on a ray at its mapped style pattern coordinates can be obtained, and the appearance of the scene in the 2D style pattern space can be intuitively stylized.

[0057] Specifically, deentanglement is achieved through simple reconstruction training. A multilayer perceptron is applied to extract the appearance of the neural radiation field in the viewpoint direction, and UV-style pattern coordinates are mapped along the view direction to colors representing the original appearance of the scene (e.g., ...). Figure 1 The radiation color shown is Cr).

[0058] Step S200: Extract the style feature map f of the style image Is using the noisy image z, correct the style feature map f by combining it with the cue vector p, and decode the corrected style feature map f to obtain the style pattern S. By adding the cue vector, the appearance of the scene is stylized in the 2D style pattern space. It understands the geometric information of the scene and can stylize the appearance more harmoniously.

[0059] In step S200, the style feature map f of the style image Is is extracted using the noisy image z, specifically including: inputting the noisy image z and the style image Is into the first encoder in sequence ( Figure 1 The encoder A and the trained first converter A are used to obtain the style feature map f. In step S200, the style mode is obtained by decoding the corrected style feature map f, specifically by using the first decoder to decode the corrected style feature map f to obtain the style mode.

[0060] The method utilizes a noisy image z to extract a style feature map f from a style image Is, along with a decoded and corrected style feature map f, an encoder A, a converter A, and a decoder A. This can be found in existing techniques for style transfer of two-dimensional images. Existing techniques using pre-trained two-dimensional image style transfer methods can effectively extract texture information from style images to generate the desired style pattern, thus generating a 2D style pattern for the style image. Building upon this, this application introduces a cue vector to correct the style feature map; the cue vector can be initialized with random values.

[0061] In this embodiment, the style image Is and the noise image z are sequentially input into encoder A, converter A, and decoder A to obtain the corresponding style mode. This is used to execute step S300 and other steps to change the appearance of the scene in the style mode space. The style mode can be used to map the result of a secondary mapping to UV mapping, as detailed in step S300.

[0062] In step S300, the spherical coordinates are mapped to the style pattern, and the appearance color and the style pattern color are coupled to obtain the stylized color C. Specifically, in step S300, coupling the appearance color and the style pattern color to obtain the stylized color involves performing a multiplication operation on the appearance color and the style pattern color to obtain the stylized color. After de-entanglement and obtaining the appearance in the viewpoint direction, the retrieved style pattern color is multiplied by the original appearance color to obtain the final stylized color representing the new stylized appearance of the scene.

[0063] Step S300, based on the style pattern extracted and generated in step S200, uses a sphere-to-cube mapping retrieval to locate each UV coordinate in the style pattern space to the corresponding pixel in the 2D style pattern (S). Then, the color of the style pattern is extracted and coupled with the appearance color (radiant color Cr) obtained in step S100. This completes the stylization mapping, visually stylizing the scene's appearance in the 2D style pattern space.

[0064] In step S300, mapping the spherical coordinates to style modes specifically includes: step S310, mapping the spherical coordinates to cube coordinates; step S311, using different noise images, extracting style feature maps of the style images multiple times to obtain multiple style modes; and step S312, mapping the cube coordinates to multiple style modes.

[0065] In step S310, the sphere coordinates are mapped to cube coordinates, i.e., a sphere-to-cube mapping retrieval is performed. This embodiment obtains stylized colors for migration to the geometry through a sphere-to-cube mapping retrieval, thereby achieving stylization of the 3D scene's appearance.

[0066] To facilitate retrieval from style patterns, this application uses a 2D unit sphere as the style pattern space. u and v represent the azimuth and elevation angles within the unit sphere, respectively. Cube mapping is widely used for spherical mapping in visualization graphics; it consists of the six faces of a unit frame and contains all color information projected from a unit sphere, such as... Figure 4 As shown. Each pair of style mode coordinates represents a different point on the unit sphere. Given style mode coordinates, a ray from the origin passes through its corresponding point on the unit sphere, and as the ray extends, it intersects the unit box in the ray direction. The cubemap value at the intersection point is a value retrieved from the style mode. This method enables sphere-to-cube mapping retrieval.

[0067] In this embodiment, each UV style pattern coordinate in the style pattern space can be used to retrieve a specific point in the style pattern (S) through a sphere-cube graph retrieval operation. The style pattern can then be used to stylize the appearance of the style pattern space and obtain the final stylized color representing the new stylized appearance of the scene.

[0068] See Figure 4 Step S311 specifically includes: using six different noisy images, repeatedly extracting style feature maps from the style images to obtain six style patterns. Step S312 specifically includes: mapping the six faces of the cube coordinates to the six style patterns respectively.

[0069] To match the output style patterns to the size of the cube map, this embodiment sets the batch size of the image stylization network to 6. A batch of style images is created as style input by repeating a given style reference 6 times. Correspondingly, the input consists of different noise images with a batch size of 6. Therefore, the output style patterns have the same batch size, and the images in this batch represent the six faces of the cell box respectively.

[0070] Step S400: The stylized color is transferred to the geometry, and a 3D style-transferred rendered image Ics is obtained in the viewpoint direction. In this embodiment, the stylized color C obtained after coupling is mapped to the geometry of the 3D scene, for example, through... Figure 1 The volume rendering implementation is achieved in this way. Ultimately, the style mapping is independent of the 3D scene, thus enabling the stylization of any 3D scene.

[0071] The embodiments of this application achieve stylization of arbitrary 3D scenes based on neural radiation fields. This not only allows for the transfer of arbitrary styles but also enables the stylization of any 3D scene using a single stylization model. The 3D scene style transfer method provided in the embodiments of this application is used to perform style transfer on arbitrary 3D scenes, resulting in a 3D scene with good 3D consistency after transfer. See also... Figure 1The specific experimental operation functions include the following:

[0072]

[0073]

[0074] In the formula, r represents the ray of light in three-dimensional space corresponding to an image pixel. The predicted pixel color is N, where N is the number of sampling points on the corresponding ray, and σ is the number of sampling points on the ray. i It is the volume density corresponding to a point in geometric space, δ i =t i+1 -t i It is the distance between adjacent sampling points on the ray, and exp is an exponential function.

[0075] According to equation (1), the contribution of each sampling point to the final pixel color is considered as a weight:

[0076] ω i =T i (1-exp(-σ i δ i (2)

[0077] The cyclic mapping process and cyclic loss are as follows Figure 3 As shown, it is defined as:

[0078] (x,y,z)→(u,v)→(x′,y′,z′) (3)

[0079] L cycle =∑ i ω i ||(x,y,z)-(x′,y′,z′)|| (4)

[0080] The total loss function for scene reconstruction is:

[0081] L=λ rec L rec +λ cycle L cycle (5)

[0082] In our experiment, we used λ rec =λ cycle =1.

[0083] Referring to Equations (2) and (4), for each ray, this application focuses more on whether the sampling points that contribute more to the final pixel color have an accurate cyclic mapping, and these sampling points represent the surface of the scene.

[0084] See formula (3) and Figure 3For the periodic loss, this application performs a bijective mapping between real-world 3D coordinates and 2D style pattern coordinates. Specifically, a cyclic loss is used to ensure the reasonableness of the style pattern space, avoiding mapping multiple points in real space to the same point in the style pattern space. Another inverse mapping network is trained, which projects the coordinates of the style pattern space onto the real space.

[0085] For steps S100 to S400, in step S300, the appearance of the 3D scene has been stylized in the 2D pattern space, and the final stylized color C is obtained through coupling. However, due to the lack of consideration for the scene's geometry, this method may lead to chaotic results. A satisfactory stylized scene should not only exhibit a pleasing style pattern in appearance but also harmoniously integrate the style pattern with the scene geometry. Therefore, the perception of geometric information is crucial for 3D scene stylization and must be appropriately considered.

[0086] To integrate geometric information into the generated style pattern, this application fine-tunes the cue vector under the supervision of a geometry-aware stylization loss. The appearance in the 2D style pattern space is stylized based on the cue vector. As a cue-based 2D stylization algorithm, this embodiment maps the appearance of a 3D scene to the 2D style pattern space, improving its style transfer effect.

[0087] In one embodiment, the cue vector is obtained through training and fine-tuning. The training and fine-tuning process of the cue vector includes: step S510, obtaining an initial viewpoint image of the neural radiation field in the viewpoint direction, the initial viewpoint image having geometry and appearance in the viewpoint direction; step S520, transferring the style of the style image to the initial viewpoint image to obtain a two-dimensional style transfer image Ics'; step S530, comparing the loss difference Lp between the obtained three-dimensional style transfer rendering image Ics and the two-dimensional style transfer image Ics', and fine-tuning the cue vector p using the loss difference Lp. Step S520 specifically includes: inputting the initial viewpoint image and the style image sequentially into the second encoder, the trained second converter, and the second decoder to obtain the two-dimensional style transfer image Ics'.

[0088] This embodiment improves practical flexibility by introducing fast and flexible on-demand learning instead of retraining the decoder (decoder A) of the 2D stylized network, avoiding the tedious and time-consuming process of fine-tuning the decoder in a conventional manner.

[0089] Specifically, we add cue vectors to the output feature map of the converter in the 2D stylization network. Figure 1 In f), the cue vector acts as an additional, independent, learnable component, implicitly representing the geometric information of the scene. For example, L... gasTrain cue vectors to generate more harmonious style patterns for the scene during the stylization process.

[0090] In this embodiment, the 2D style transfer image Ics' and the 3D style transfer rendered image Ics share the same viewpoint direction. The 2D style transfer image obtained under the same viewpoint direction can be used to correct the cue vector to optimize the 3D style transfer rendered image.

[0091] The optimization process of the cue vector is a training and learning process. After training and learning are completed, steps S510 to S520 can be removed. Therefore, the 3D scene style transfer method at this point belongs to an arbitrary 3D scene style transfer method based on cue learning and neural style mapping.

[0092] In this embodiment, all parameters of the image stylization network are frozen, and the cue vector is the only parameter that needs to be trained during the stylization stage. It is understood that the cue vector approach used in this application is not limited to a specific 2D stylization network. The cue vector is plug-and-play and can be easily integrated into existing image style transfer methods. The cue vector can also be adjusted on a single scene. In this case, the cue vector can be viewed as a scene-dependent adapter that adapts to the style patterns generated by the image stylization network to more harmoniously transfer the appearance of the target scene.

[0093] Specifically, the hint vector uses L gas train: θ * =argmin θ L gas (6)

[0094] Where L gas It is a geometry-aware stylization loss, where R is the set of rays corresponding to all pixels from the training view, and This represents the pixel color obtained after the pixel has been processed by a 2D stylization network with fine-tuned cue vectors. This represents the pixel color of a stylized image that uses the training view as content input to a pre-trained 2D stylization network. Because the stylized training view is used for explicit supervision, the style patterns generated by the 2D stylization network are better adapted to the geometry of the scene.

[0095] It should be understood that, although Figure 2 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0096] This application also provides a 3D scene style transfer system for transferring the style of a style image to a 3D scene while preserving the geometric structure of the 3D scene, including:

[0097] The neural radiation field extraction module is used to model 3D scenes based on neural radiation fields. It uses voxel meshes to model style mode space and geometric space, obtains the geometry of neural radiation fields in geometric space, obtains the spherical coordinates of neural radiation fields in style mode space, and their appearance in the viewpoint direction.

[0098] The style pattern extraction module is used to extract style feature maps from noisy images, correct the style feature maps by combining cue vectors, and decode the corrected style feature maps to obtain style patterns.

[0099] The appearance mapping module is used to map spherical coordinates to style patterns, and couple the appearance color with the style pattern color to obtain stylized colors;

[0100] The Stylization Mapping module is used to transfer stylized colors to geometry and render a 3D style-transferred image from the viewpoint.

[0101] The 3D scene style transfer methods provided in the embodiments of this application, based on cue learning (fine-tuning of cue vectors) and neural style mapping, can transfer the style of any 3D scene, thereby achieving controllable color and texture transfer of 3D scenes. It is proposed to utilize the knowledge learned by the latest pre-trained visual models and leverage cue learning to deeply mine this knowledge, enabling the constructed style transfer model for 3D scenes to be trained on a small amount of 3D scene content, and then generalized to any scene. This allows the use of pre-trained model knowledge to improve the overall quality of color and texture editing (style transfer) of 3D scenes.

[0102] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, network interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a 3D scene style transfer method. The display screen can be an LCD screen or an e-ink screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0103] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0104] Step S100: Based on the 3D scene of the neural radiation field, use voxel mesh to model the style mode space and geometric space to obtain the geometry of the neural radiation field in the geometric space, and obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction.

[0105] Step S200: Extract style feature map of style image using noisy image, correct style feature map by combining cue vector, and decode style feature map to obtain style pattern.

[0106] Step S300: Map the sphere coordinates to the style mode, couple the appearance color and the style mode color to obtain the stylized color.

[0107] Step S400: Transfer the stylized colors to the geometry and render the 3D style-transferred rendering image in the viewpoint direction.

[0108] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0109] Step S100: Based on the 3D scene of the neural radiation field, use voxel mesh to model the style mode space and geometric space to obtain the geometry of the neural radiation field in the geometric space, and obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction.

[0110] Step S200: Extract style feature map of style image using noisy image, correct style feature map by combining cue vector, and decode style feature map to obtain style pattern.

[0111] Step S300: Map the sphere coordinates to the style mode, couple the appearance color and the style mode color to obtain the stylized color.

[0112] Step S400: Transfer the stylized colors to the geometry and render the 3D style-transferred rendering image in the viewpoint direction.

[0113] In this embodiment, the computer program product includes a program code portion for executing the steps of the 3D scene style transfer method in the embodiments of this application when the computer program product is executed by one or more computing devices. The computer program product may be stored on a computer-readable recording medium. It may also be provided via a data network (e.g., via RAN, via the Internet, and / or via RBS) for download. Alternatively or additionally, the method may be encoded in a field-programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or its functionality may be provided by means of a hardware description language for download. Those skilled in the art will understand that implementing all or part of the processes in the methods of the above embodiments can be accomplished by a computer program instructing related hardware. The computer program may be stored in a non-volatile computer-readable storage medium, and when executed, it may include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0114] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered to be within the scope of this specification. When technical features of different embodiments are embodied in the same drawing, it can be regarded as the drawing also disclosing examples of combinations of the various embodiments involved.

[0115] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A 3D scene style transfer method, used to transfer the style of a style image to a 3D scene while preserving the geometric structure of the 3D scene, characterized in that, include: Based on the 3D scene of the neural radiation field, voxel meshes are used to model the style mode space and geometric space to obtain the geometry of the neural radiation field in the geometric space, and to obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the view direction. Style feature maps of style images are extracted from noisy images, and the style feature maps are corrected by combining cue vectors. The style pattern is obtained by decoding the corrected style feature maps. Map the spherical coordinates to the style pattern, and couple the color of the appearance with the color of the style pattern to obtain stylized colors; The stylized colors are transferred to the geometry, and a 3D style-transferred rendering image is obtained from the viewpoint.

2. The three-dimensional scene style transfer method as described in claim 1, characterized in that, Obtaining the spherical coordinates of the neural radiation field in style pattern space and its appearance in the viewpoint direction specifically includes: The spherical coordinates of the neural radiation field in style pattern space are obtained by applying UV mapping; By combining the three-dimensional scene of the neural radiation field in the viewpoint direction, a multilayer perceptron is applied to extract the appearance of the neural radiation field in the viewpoint direction.

3. The 3D scene style transfer method as described in claim 1, characterized in that, Extracting style feature maps from style images using noisy images, specifically including: The noisy image and the style image are sequentially input into the first encoder and the trained first converter to obtain the style feature map; Decoding the corrected style feature map yields the style pattern, specifically including: The style pattern is obtained by decoding the corrected style feature map using the first decoder.

4. The three-dimensional scene style transfer method as described in claim 1, characterized in that, Mapping the spherical coordinates to the style mode specifically includes: Map the spherical coordinates to cube coordinates; By using different noisy images, style feature maps of style images are extracted multiple times to obtain multiple style patterns; Map the cube coordinates to the multiple style patterns.

5. The three-dimensional scene style transfer method as described in claim 4, characterized in that, Mapping the spherical coordinates to the style mode specifically includes: Map the spherical coordinates to cube coordinates; Using six different noisy images, style feature maps of the style images are extracted multiple times to obtain six style patterns; The six faces of the cube coordinates are respectively mapped to the six style modes.

6. The three-dimensional scene style transfer method as described in claim 1, characterized in that, Couple the color of the appearance with the color of the style mode to obtain stylized colors, specifically including: The stylized color is obtained by performing a multiplication operation on the color of the appearance and the color of the style mode.

7. The three-dimensional scene style transfer method as described in claim 1, characterized in that, The cue vector is obtained through training and fine-tuning. The training and fine-tuning process of the cue vector includes: Obtain an initial viewpoint image of the neural radiation field in the viewpoint direction, the initial viewpoint image having geometry in the viewpoint direction and appearance in the viewpoint direction; The style of the style image is transferred to the initial viewpoint image to obtain a two-dimensional style-transferred image; The loss difference between the obtained 3D style transfer rendered image and the 2D style transfer image is compared, and the cue vector is fine-tuned using the loss difference.

8. The three-dimensional scene style transfer method as described in claim 7, characterized in that, The style of the style image is transferred to the initial viewpoint image to obtain a two-dimensional style-transferred image, specifically including: The initial viewpoint image and the style image are input sequentially into the second encoder, the trained second converter, and the second decoder to obtain a two-dimensional style transfer image.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the three-dimensional scene style transfer method as claimed in any one of claims 1 to 8.

10. A 3D scene style transfer system, used to transfer the style of a style image to a 3D scene while preserving the geometric structure of the 3D scene, characterized in that, include: The neural radiation field extraction module is used to model the style mode space and geometric space of a 3D scene based on the neural radiation field using voxel meshes, obtain the geometry of the neural radiation field in the geometric space, and obtain the spherical coordinates of the neural radiation field in the style mode space and its appearance in the viewpoint direction. The style pattern extraction module is used to extract style feature maps from the noisy image, correct the style feature maps by combining them with cue vectors, and decode the corrected style feature maps to obtain style patterns. The appearance mapping module is used to map the spherical coordinates to the style pattern, and couple the color of the appearance with the color of the style pattern to obtain stylized colors; The stylization mapping module is used to transfer the stylized colors to the geometry and render a 3D style-transferred rendering image in the viewpoint direction.

Images