3D gaussian scene style transfer method based on 2D prior, computer device and computer program product

Abstract

The application relates to a 3D Gaussian scene style transfer method based on 2D prior, a computer device and a computer program product, which comprises the following steps: constructing a 3D Gaussian by using a multi-view original image set of a 3D scene; performing image rendering on the 3D Gaussian to obtain a rendering image set corresponding to the view of the original image set; performing style transfer on the rendering images in the rendering image set by using an expected style image to obtain stylized images with the same content structure as the rendering images; replacing the images of the corresponding view in the original image set by using the stylized images to form a supervised image set after the replacement; and comparing the difference loss of the images under the same view in the supervised image set and the rendering image set to optimize the 3D Gaussian. The application can obtain stylized images by using the prior knowledge of a two-dimensional style transfer method, and then optimize the 3D Gaussian through the difference loss to transfer the style information of the stylized images to the 3D Gaussian scene, so that the scene style transfer of the 3D Gaussian is realized.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and deep learning, and in particular to a 3D Gaussian scene style transfer method based on 2D prior, a computer device, and a computer program product. Background Art

[0002] 3D scene style transfer is an important editing task in the fields of vision and graphics, and a rapidly developing research area. It allows designers and artists to transfer a specific visual style from one 3D model to another, or to perform various style transformations on the same model to create new 3D scenes. Given a set of multi-view images of a 3D scene and an image that captures the target style, it generates stylized views of the 3D scene from arbitrary novel viewpoints based on the target style of the style image. This technology has important applications in many scenarios, such as film special effects, game development, virtual reality, and various artistic creations.

[0003] However, implementing 3D style transfer often faces challenges, such as how to maintain the shape and structure of the model while ensuring style consistency, how to handle complex 3D geometry and material information, and how to efficiently transfer style on large-scale 3D models. Due to the lack of 3D information, directly applying image / video stylization methods to 3D scenes often leads to inconsistent results across different views.

[0004] To deal with the inconsistency problem, existing methods have explored 3D scene stylization based on explicit 3D models (e.g., meshes, voxels, and point clouds). However, their discrete representation of the scene results in a loss of geometric accuracy. Among the most advanced style transfer techniques, NeRF (Neural Radiance Field) is generally used as a representation of complex 3D scenes. Compared with point clouds and meshes, Neural Radiance Field can be more reliably obtained from multi-view images and is continuous in 3D space, making learning easier. However, there are still limitations in using NeRF as a representation of 3D scenes in style transfer tasks. NeRF requires querying hundreds of sample points along a ray to render a single pixel. Memory limitations make it very difficult to draw an entire image or even a large enough image block at once, which is very important for calculating content and style losses. Therefore, directly training stylized NeRF with perceptual style and content losses on smaller training blocks will lead to poor stylization results.

[0005] Recently, 3D Gaussian (3DGS) emerges as a revolutionary technique in the field of explicit radiance field and computer graphics, and receives more and more attention due to its fast rendering capability and excellent rendering quality. It is quite different from the neural radiance field method which mainly uses an implicit coordinate-based model to map the spatial coordinates to the pixel value, and 3D Gaussian uses millions of 3D Gaussian functions to construct the scene. With its explicit scene representation and differentiable rendering algorithm, it not only has faster rendering speed than NeRF radiance field, but also introduces an unprecedented level of control and editability, overcoming the disadvantage of NeRF being difficult to edit, making it possible to achieve higher quality 3D style transfer. SUMMARY

[0006] Therefore, it is necessary to provide a 3D Gaussian scene style transfer method based on 2D priori in view of the above technical problems.

[0007] The 3D Gaussian scene style transfer method based on 2D priori provided in the present application comprises:

[0008] A 3D Gaussian is constructed using a multi-view original image set of a 3D scene.

[0009] Image rendering is performed on the 3D Gaussian to obtain a rendering image set corresponding to the view angles of the original image set.

[0010] Style transfer is performed on the rendering images in the rendering image set using a desired style image to obtain stylized images with the same content structure as the rendering images.

[0011] The stylized images are used to replace the images of the corresponding view angles in the original image set, and after the replacement, a supervised image set is formed.

[0012] The difference loss of the images under the same view angle in the supervised image set and the rendering image set is compared, and the 3D Gaussian is optimized.

[0013] Optionally, the 3D Gaussian is constructed using a multi-view original image set of a 3D scene, specifically comprising: constructing an initialized 3D Gaussian, and remodeling the initialized 3D Gaussian using a multi-view original image set of a 3D scene to obtain a reconstructed 3D Gaussian.

[0014] Optionally, the replacement of the images of the corresponding view angles in the original image set by the stylized images to form the supervised image set after the replacement is performed in multiple rounds, and the difference loss of the images under the same view angle in the supervised image set and the rendering image set is compared in each round to optimize the 3D Gaussian.

[0015] Optionally, the original images in the supervised image set are gradually replaced in each round;

[0016] The supervised image set includes stylized images replaced in this round, stylized images replaced in previous rounds, and original images that have not been replaced temporarily.

[0017] Optionally, optimizing the 3D Gaussian includes a first stage and a second stage performed sequentially:

[0018] In the first stage, the original image is not replaced, and all parameters of the 3D Gaussian are unlocked when optimizing the 3D Gaussian.

[0019] In the second stage, there is no unreplaced original image, and the 3D Gaussian color parameters are unlocked when optimizing the 3D Gaussian.

[0020] Optionally, in the first stage, the stylized image is used to replace the original image of the corresponding perspective in the original image set, and after the replacement is completed, a supervision image set is formed;

[0021] In the second stage, the stylized image is used to update and replace the stylized image of the corresponding perspective in the supervision image set, and the supervision image set is updated after the replacement is completed.

[0022] Optionally, the 3D Gaussian scene style transfer method also includes:

[0023] Use the optimized 3D Gaussian to render and obtain new perspective images;

[0024] Random noise is added to the new perspective image to obtain a noisy image, a pre-trained diffusion model is used to perform prediction on the noisy image and obtain predicted noise, a difference between the random noise and the predicted noise is compared, and the 3D Gaussian is supplemented and optimized.

[0025] Optionally, the 3D Gaussian represents a 3D scene as a set of 3D Gaussian primitives, where the 3D Gaussian primitives include a parameter mixture of multiple 3D Gaussians, and the parameters of the 3D Gaussian include center, shape and size, opacity, and color.

[0026] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the 2D prior-based 3D Gaussian scene style transfer method described in the present application.

[0027] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the 2D prior-based 3D Gaussian scene style transfer method described in the present application are implemented.

[0028] The application also provides a computer program product comprising computer instructions which, when executed by a processor, implement the steps of the 3D Gaussian scene style transfer method based on 2D prior art described in the application.

[0029] The 3D Gaussian scene style transfer method based on 2D prior art of the application has at least the following effects:

[0030] The application can use the prior art two-dimensional style transfer method in the prior art to obtain a stylized image. Then, the style information of the stylized image is transferred to the 3D Gaussian scene by optimizing the 3D Gaussian through the difference loss. By updating the supervised image set, the 3D Gaussian gradually has a style scene that matches the style image, and the artistic style information of the style image is transferred to the 3D Gaussian scene. At the same time of the transfer, the geometric structure of the three-dimensional scene is maintained, and the style transfer of the 3D Gaussian scene is completed.

[0031] The optimization of the 3D Gaussian of the application includes a first stage and a second stage executed in sequence. In the middle and later stages of the first stage, the proportion of stylized images in the supervised image set increases over time, the participation of stylized images used to optimize the 3D Gaussian gradually increases, the 3D Gaussian is constantly updated and repeatedly rendered and iterated, the 3D Gaussian begins to converge to a globally consistent stylized scene, and the three-dimensional consistency of the stylized result can be effectively guaranteed. In the whole process, the target style gradually penetrates into the three-dimensional scene and is merged into a globally consistent 3D representation. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 The flowchart of the 3D Gaussian scene style transfer method based on 2D prior art in an embodiment of the application is shown in the figure;

[0033] Figure 2 The structure block diagram of the 3D Gaussian scene style transfer method based on 2D prior art in an embodiment of the application is shown in the figure;

[0034] Figure 3 The structure block diagram of the 3D Gaussian scene style transfer method based on 2D prior art in an embodiment of the application is shown in the figure;

[0035] Figure 4 The structure block diagram of the 3D Gaussian scene style transfer method based on 2D prior art in an embodiment of the application is shown in the figure;

[0036] Figure 5 The internal structure diagram of the computer device in an embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0037] Current style transfer methods have achieved good results in migrating two-dimensional image content. However, for three-dimensional scene content, the migration quality of existing methods is poor, and the migrated scenes often have problems such as blurring and poor three-dimensional consistency.

[0038] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0039] See also Figure 1 and Figure 2 In view of this, an embodiment of the present application provides a 3D Gaussian scene style transfer method based on 2D prior, comprising:

[0040] Step S100, constructing a 3D Gaussian using a set of original images of a 3D scene from multiple perspectives;

[0041] Step S200 , performing image rendering on the 3D Gaussian to obtain a rendered image set corresponding to the viewing angle of the original image set;

[0042] Step S300 , performing style transfer on the rendered images in the rendered image set using the expected style image to obtain a stylized image having the same content structure as the rendered image;

[0043] Step S400: Using the stylized image to replace the image of the corresponding perspective in the original image set, and after the replacement is completed, a supervised image set is formed;

[0044] Step S500 , comparing the difference loss of images under the same viewing angle in the supervised image set and the rendered image set, and optimizing the 3D Gaussian.

[0045] This embodiment uses the two-dimensional style transfer method in the existing technology as prior knowledge to obtain a stylized image. Then, the 3D Gaussian is optimized through difference loss to transfer the style information of the stylized image to the 3D Gaussian scene. In specific implementation, steps S300 to S500 can be executed cyclically to update the supervision image set, gradually making the 3D Gaussian have a style scene that matches the style image, and migrating the artistic style information of the style image to the 3D Gaussian scene to complete the style transfer of the 3D Gaussian scene. Through the above steps, the style of the style image is finally transferred to the three-dimensional scene while maintaining the geometric structure of the three-dimensional scene, and the migrated three-dimensional scene has a high three-dimensional consistency.

[0046] See also Figure 3In some embodiments, the 3D Gaussian scene style transfer method based on 2D prior also includes step S600, specifically including: step S610, using the optimized 3D Gaussian to render and obtain a new perspective image; step S620, adding random noise to the new perspective image to obtain a noisy image, using a pre-trained diffusion model to perform prediction on the noisy image and obtain predicted noise, comparing the difference between the random noise and the predicted noise, and supplementing the optimized 3D Gaussian.

[0047] See also Figure 1 and Figure 4 In some embodiments, a 3D Gaussian scene style transfer method based on a 2D prior is provided to explain and illustrate in detail steps S100 to S600 and their optional sub-steps. The method includes steps S1 to S3. Step S1 includes 3D Gaussian-based 3D scene modeling; step 2 includes 3D Gaussian optimization based on iterative updates of a supervised dataset; and step 3 includes 3D Gaussian optimization based on a diffusion prior.

[0048] Step S1, three-dimensional scene modeling based on 3D Gaussian, is used to explain and correspond to step S100 and step S200. Step S100 specifically includes: constructing an initialized 3D Gaussian, using a set of original images of multiple perspectives of a 3D scene, remodeling the initialized 3D Gaussian, and obtaining a reconstructed 3D Gaussian. This embodiment mainly adopts a three-dimensional scene style migration method based on 3D Gaussian technology. In order to obtain the geometric structure and style pattern of the three-dimensional scene, a visual three-dimensional reconstruction framework (such as a colmap framework) is used to generate a sparse point cloud, and the 3D Gaussian initialization is completed. Subsequently, an original image set consisting of multi-perspective original images in a 3D scene is used, and the three-dimensional scene is remodeled using 3D Gaussian technology to obtain a reconstructed 3D Gaussian.

[0049] 3D Gaussian represents a 3D scene as a set of 3D Gaussian primitives, which are a mixture of parameters of multiple 3D Gaussian primitives. The parameters of a 3D Gaussian primitive include center, shape and size, opacity, and color. The specific representation of a 3D Gaussian primitive is Where:

[0050] represents a set of 3D Gaussian primitives;

[0051] g p , represents a 3D Gaussian, that is, a 3D Gaussian sphere;

[0052] μ p , represents the mean, used to specify a 3D Gaussian g p the center of

[0053] Σ p , represents the covariance, used to specify a 3D Gaussian g pshape and size;

[0054] σ p , indicating opacity;

[0055] c p , indicating color;

[0056] p, represents the number of each 3D Gaussian sphere;

[0057] P, represents the total number of Gaussian balls, which means the set Contains P Gaussian components.

[0058] Corresponding to step S200, when a 2D rendered image is obtained by image rendering, step S1 calculates the color C of the pixel and mixes N ordered 3D Gaussians overlapping the target pixel to effectively render the 3D Gaussian. The formula is as follows:

[0059]

[0060] Where:

[0061] C, represents the color of a pixel in the 2D rendered image;

[0062] c i , represents the color of the i-th Gaussian;

[0063] Δ i , represents the color contribution of the c-th Gaussian to the pixel;

[0064]

[0065] x i , represents the distance between the i-th Gaussian center point and the pixel point;

[0066] ∑ i , represents the covariance of a 3D Gaussian, which is used to represent the shape and size of a 3D Gaussian;

[0067] σ i , represents the opacity of a 3D Gaussian;

[0068] It can be understood that for a pixel point of a rendered image, when the pixel point corresponds to a three-dimensional space, it will be affected by the parameters of 3D Gaussians at different spatial positions, and orderly overlap and parameter mixing of each 3D Gaussian will occur in the corresponding viewing angle according to the viewing angle order.

[0069] Step 2, 3D Gaussian optimization based on iterative updates of the supervised dataset, is used to explain and correspond to steps S300 to S500.

[0070] In step S300, the style image is used to perform style transfer on the rendered image to obtain a stylized image. This is accomplished using the AdaIN algorithm. The AdaIN formula is as follows:

[0071]

[0072] Among them, I cs To stylize the image, I s is the style image, I r is the content image, which is specifically a 3D Gaussian rendering image in this embodiment;

[0073] E is the encoder, D is the decoder, σ is the variance, and μ is the mean. After the style transfer is completed, the stylized image I cs The input style image I will be stored in s style information, and step S300 is completed.

[0074] In step S400, the obtained stylized images are used to replace the original images corresponding to the camera viewpoint in the original image set to form a supervised image set. In step S500, the style information in these stylized images is integrated into the 3D Gaussian through 3D Gaussian training with partially frozen parameters.

[0075] In steps S400 and S500, the stylized image is used to replace the image of the corresponding viewpoint in the original image set. After the replacement is completed, the supervised image set is formed, which includes multiple rounds of execution. Each round compares the difference loss of images under the same viewpoint in the supervised image set and the rendered image set to optimize the 3D Gaussian. Step S400 edits the training image dataset (supervised image set) of the 3D Gaussian, and step S500 updates the 3D Gaussian representation. Steps S300 to S500 are executed in a loop to train and optimize the 3D Gaussian.

[0076] See also Figure 4 , specifically, the original image set consists of images from a series of viewpoints {v i}, which is represented as In each round, 2D image stylization is performed n times to obtain a set of stylized images. Then use the stylized image set {I cs}Update replaces the original image set After the replacement, the stylized supervised image set (i.e., supervised image dataset) is formed.

[0077] After completing a round of replacement, use the supervised image set During supervised learning, several iterations of 3D Gaussian optimization are performed within a single round to obtain an iterative stylized 3D Gaussian. A round can be understood as the process of replacing a supervised image set with a stylized image and applying it. A round involves multiple iterations of 3D Gaussian optimization and rendering. Specifically, the 3D Gaussian is optimized multiple times within a single round, and the rendered image from each iteration is compared against the supervised image set for that round.

[0078] Furthermore, in different rounds, for step S400, the original images in the supervised image set are gradually replaced in each round. The supervised image set includes the stylized images replaced in the current round, stylized images replaced in previous rounds, and original images that have not yet been replaced. The replacement process includes, before the first replacement, determining a random sorting table for the original images based on different camera viewpoints v; and performing each round of replacement sequentially according to this random sorting table. The number of images replaced in each round can be the same or different.

[0079] In this embodiment, optimizing the 3D Gaussian is a cyclic and step-by-step process, in which the 2D style transfer model is used to iteratively update the training dataset images, and then the 3D Gaussian is trained on these updated supervised image sets.

[0080] In step S500, optimizing the 3D Gaussian includes a first stage and a second stage that are executed in sequence. The first stage ends after all images in the original image set are replaced. In the first stage, there are original images that have not been replaced, and all parameters of the 3D Gaussian are unlocked when optimizing the 3D Gaussian; in the second stage, there are no original images that have not been replaced, and the color parameters of the 3D Gaussian are unlocked when optimizing the 3D Gaussian. In the first stage, the original images of the corresponding perspectives in the original image set are replaced with stylized images, and after the replacement is completed, a supervised image set is formed; in the second stage, the stylized images of the corresponding perspectives in the supervised image set are updated and replaced with stylized images, and after the replacement is completed, the supervised image set is updated.

[0081] In this embodiment, each round of supervised optimization of the 3D Gaussian uses the image set updated by the iterative stylization of this round, so that the supervisory signal of each round of optimization is a mixture of old information and the updated new style information. In the early stage of the first stage, since the original images in the original image set are suddenly replaced by stylized images, the supervised image set changes suddenly, and the difference loss between the supervised image set and the rendered image set causes the 3D Gaussian optimization to perform structurally inconsistent reconstruction. In the middle and late stages of the first stage, as time goes by, the proportion of stylized images in the supervised image set continues to increase, and the participation of stylized images used to optimize the 3D Gaussian gradually increases. The 3D Gaussian is constantly updated and repeatedly rendered and iterated. The 3D Gaussian begins to converge to a globally consistent stylized scene and can effectively ensure the three-dimensional consistency of the stylized results. During the whole process, the target style gradually penetrates into the three-dimensional scene and merges it into a globally consistent 3D representation.

[0082] In the second stage, the 3D Gaussian optimization releases the lock of the 3D Gaussian color parameters, and freezes the remaining parameters (covariance matrix, mean, etc.), and only the color parameters of the 3D Gaussian are updated during training. The loss function is:

[0083]

[0084] In the formula:

[0085] denotes the total loss of the difference between the rendered image set and the supervised image set;

[0086] denotes the image quality loss of the rendered image set and the supervised image set loss;

[0087] denotes the image quality loss of the rendered image set and the supervised image set

[0088] λ, the value range is 0.2-0.8.

[0089] Step 3, 3D Gaussian optimization based on diffusion prior, used to explain and correspond to step S600.

[0090] Since the diffusion model is pre-trained on large-scale data, the model contains a general image prior. In the stylization process of the 3D Gaussian scene, the camera pose of the new view has no corresponding supervised image, and the area with low coverage in the supervised view is easy to lose details. In order to make the stylization result of the 3D Gaussian scene have a more complete three-dimensional representation, a diffusion prior is introduced in the iterative optimization process of the 3D Gaussian.

[0091] Specifically, first, a series of new view camera poses {v i ′} are generated in the camera-to-world coordinate system. The camera pose of the new view does not exist in the original image set {v i}. Then the 3D Gaussian scene is rendered under the new camera pose to obtain a series of new view images {I′ r}, and random noise v is introduced in the new view image. The diffusion model is used to denoise the 3D Gaussian rendering image with noise, calculate the introduced noise, and obtain the predicted noise. By comparing the difference between the introduced random noise and the predicted noise, the score distillation sampling (SDS) loss is calculated, and the 3D Gaussian scene is optimized by back propagation. The score distillation sampling loss The formula is as follows:

[0092]

[0093] Where G represents the parameters of the 3D Gaussian model, Indicates the expectation, ∈ φ represents the prediction noise calculated by the diffusion model, Represents the noisy image after the new perspective image is noisy, t represents the time step, and ∈ represents the introduced random noise. Denotes the use of gradient backpropagation to supplement the optimization of 3D Gaussian. The fractional distillation sampling loss leverages the powerful prior of the diffusion model. The 3D Gaussian scene optimization after introducing this loss can supplement the missing details after the stylization of the 3D Gaussian scene from a new perspective.

[0094] Because the diffusion model is trained on a large-scale dataset, optimization based on the diffusion prior can interpolate missing details after the stylized 3D Gaussian scene. This embodiment introduces a diffusion prior and fractional distillation sampling loss to optimize the 3D Gaussian scene and supplement the missing details after the 3D Gaussian scene is stylized. The gradient update at each step of this process includes a random mixture of rays distributed across many viewpoints, which greatly improves the stability of training and can effectively transfer the learned style information to the 3D scene, resulting in a stylized 3D Gaussian that is consistent in three dimensions.

[0095] Each embodiment of the present application performs 3D Gaussian optimization based on iterative updates of a supervised image set. First, a 2D stylized model is used to stylize the rendered image of the 3D Gaussian, and the rendered image is used to replace the supervised image of the corresponding camera viewpoint, and the supervised image set of the 3D Gaussian is updated. By making full use of the 2D stylized model prior, style features are extracted from the style image, and the 3D Gaussian rendered image is updated through image style transfer in each iteration, and the supervised image set is semi-permanently updated, that is, it is gradually replaced and updated without regression, retaining the stylized image. The style information in the dataset is then integrated into the 3D Gaussian through 3D Gaussian training. The updating of the supervised image set and the 3D Gaussian optimization training are alternated, and the learned style information is transferred to the three-dimensional scene through iterative dataset updates while maintaining three-dimensional consistency.

[0096] It should be understood that although Figure 1 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. In addition, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0097] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a 3D Gaussian scene style migration method based on 2D prior is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad provided on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0098] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0099] Step S100, constructing a 3D Gaussian using a set of original images of a 3D scene from multiple perspectives;

[0100] Step S200 , performing image rendering on the 3D Gaussian to obtain a rendered image set corresponding to the viewing angle of the original image set;

[0101] Step S300 , performing style transfer on the rendered images in the rendered image set using the expected style image to obtain a stylized image having the same content structure as the rendered image;

[0102] Step S400: Using the stylized image to replace the image of the corresponding perspective in the original image set, and after the replacement is completed, a supervised image set is formed;

[0103] Step S500 , comparing the difference loss of images under the same viewing angle in the supervised image set and the rendered image set, and optimizing the 3D Gaussian.

[0104] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0105] Step S100, constructing a 3D Gaussian using a set of original images of a 3D scene from multiple perspectives;

[0106] Step S200 , performing image rendering on the 3D Gaussian to obtain a rendered image set corresponding to the viewing angle of the original image set;

[0107] Step S300 , performing style transfer on the rendered images in the rendered image set using the expected style image to obtain a stylized image having the same content structure as the rendered image;

[0108] Step S400: Using the stylized image to replace the image of the corresponding perspective in the original image set, and after the replacement is completed, a supervised image set is formed;

[0109] Step S500 , comparing the difference loss of images under the same viewing angle in the supervised image set and the rendered image set, and optimizing the 3D Gaussian.

[0110] In one embodiment, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the following steps:

[0111] Step S100, constructing a 3D Gaussian using a set of original images of a 3D scene from multiple perspectives;

[0112] Step S200 , performing image rendering on the 3D Gaussian to obtain a rendered image set corresponding to the viewing angle of the original image set;

[0113] Step S300 , performing style transfer on the rendered images in the rendered image set using the expected style image to obtain a stylized image having the same content structure as the rendered image;

[0114] Step S400: Using the stylized image to replace the image of the corresponding perspective in the original image set, and after the replacement is completed, a supervised image set is formed;

[0115] Step S500 , comparing the difference loss of images under the same viewing angle in the supervised image set and the rendered image set, and optimizing the 3D Gaussian.

[0116] In this embodiment, the computer program product includes a program code portion for executing the steps of the 2D prior-based 3D Gaussian scene style transfer method in each embodiment of the present application when the computer program product is executed by one or more computing devices. The computer program product can be stored on a computer-readable recording medium. The computer program product can also be provided for download via a data network (e.g., via a RAN, via the Internet, and / or via an RBS). Alternatively or additionally, the method can be encoded in a field programmable gate array (FPGA) and / or an application-specific integrated circuit (ASIC), or the functionality can be provided for download with the aid of a hardware description language.

[0117] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0118] The technical features of the above embodiments may be combined in any manner. To simplify the description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there are no conflicts in the combination of these technical features, they should be considered to be within the scope of this specification. When technical features in different embodiments are reflected in the same figure, it can be regarded as that figure also discloses the combination examples of the various embodiments involved.

[0119] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that a person skilled in the art could make various modifications and improvements without departing from the spirit of the present application, all of which fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. 3D Gaussian scene style transfer method based on 2D prior, characterized by: include: Construct a 3D Gaussian using a set of original images from multiple perspectives of a 3D scene; Performing image rendering on the 3D Gaussian to obtain a rendered image set corresponding to the viewing angle of the original image set; Using the expected style image, performing style transfer on the rendered images in the rendered image set to obtain a stylized image having the same content structure as the rendered image; The stylized image is used to replace the image of the corresponding perspective in the original image set, and after the replacement is completed, a supervision image set is formed; The difference loss of images under the same viewing angle in the supervised image set and the rendered image set is compared to optimize the 3D Gaussian.

2. The 3D Gaussian scene style transfer method based on 2D prior according to claim 1, characterized in that The method of constructing a 3D Gaussian using a set of original images of a 3D scene with multiple perspectives specifically includes: constructing an initialized 3D Gaussian, and remodeling the initialized 3D Gaussian using a set of original images of a 3D scene with multiple perspectives to obtain a reconstructed 3D Gaussian.

3. The 2D prior-based 3D Gaussian scene style transfer method according to claim 1, wherein: The stylized image is used to replace the image of the corresponding perspective in the original image set. After the replacement is completed, the supervised image set is formed, which includes multiple rounds of execution, and each round compares the difference loss of the images under the same perspective in the supervised image set and the rendered image set to optimize the 3D Gaussian.

4. The 3D Gaussian scene style transfer method based on 2D prior according to claim 3, characterized in that: The original images in the supervised image set are gradually replaced in each round; The supervised image set includes stylized images replaced in this round, stylized images replaced in previous rounds, and original images that have not been replaced temporarily.

5. The 3D Gaussian scene style transfer method based on 2D prior according to claim 1, characterized in that Optimizing the 3D Gaussian consists of a first and a second stage performed sequentially: In the first stage, the original image is not replaced, and all parameters of the 3D Gaussian are unlocked when optimizing the 3D Gaussian. In the second stage, there is no unreplaced original image, and the 3D Gaussian color parameters are unlocked when optimizing the 3D Gaussian.

6. The 3D Gaussian scene style transfer method based on 2D prior according to claim 5, characterized in that: In the first stage, the stylized image is used to replace the original image of the corresponding perspective in the original image set, and after the replacement is completed, a supervision image set is formed; In the second stage, the stylized image is used to update and replace the stylized image of the corresponding perspective in the supervision image set, and the supervision image set is updated after the replacement is completed.

7. The 3D Gaussian scene style transfer method based on 2D prior according to claim 1, characterized in that: 3D Gaussian scene style transfer methods also include: Use the optimized 3D Gaussian to render and obtain new perspective images; Random noise is added to the new perspective image to obtain a noisy image, a pre-trained diffusion model is used to perform prediction on the noisy image and obtain predicted noise, a difference between the random noise and the predicted noise is compared, and the 3D Gaussian is supplemented and optimized.

8. The 3D Gaussian scene style transfer method based on 2D priors according to claim 1, wherein: The 3D Gaussian represents a 3D scene as a set of 3D Gaussian primitives. The 3D Gaussian primitives include a parametric mixture of multiple 3D Gaussians. The parameters of the 3D Gaussian include center, shape and size, opacity, and color.

9. A computer device comprising a memory, a processor and a computer program stored in the memory, characterized in that The processor executes the computer program to implement the steps of the 2D prior-based 3D Gaussian scene style transfer method according to any one of claims 1 to 8.

10. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by a processor, the steps of the 2D prior-based 3D Gaussian scene style transfer method according to any one of claims 1 to 8 are implemented.

Images