Image stylization editing method, device, equipment and medium for three-dimensional scene

Abstract

The present invention relates to the field of computer technology, and more particularly to a method, apparatus, device, and medium for stylizing and editing images of three-dimensional scenes. The method comprises: obtaining a source three-dimensional model and a reference style image, as well as a first image stylization editing network and a second image stylization editing network; obtaining content features of the image to be edited and style features of the reference style image; utilizing image diffusion loss to optimize the two image stylization editing networks with added low-rank adaptation functions to construct a third image stylization editing network; inputting the image to be edited and the reference style image into the third image stylization editing network for processing, and outputting a stylized single-view image; utilizing an optimization strategy based on the combination of 3D content and style to iteratively update the stylized single-view image to obtain a stylized 3D model. This method improves the stylization effect of the reference style image and, under the premise of consistent style, ensures the fidelity and multi-view consistency of the 3D content.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for stylized editing of images of a three-dimensional scene. Background Art

[0002] 3D model stylization is an important method for editing the appearance of 3D models. It can achieve consistent 3D stylization given a single reference style image. This means that the style of any 2D style image can be consistently transferred to a 3D scene. The goal of 3D scene style transfer is to transfer the style of the reference style image to the 3D scene while maintaining style consistency, content fidelity, and multi-view consistency.

[0003] Currently, image editing is being promoted to three-dimensional space using iterative dataset update strategies, enabling edits to 3D models, such as color and texture. These edits are applied to the dataset images of the original generated scene. Recently, an efficient feature rendering strategy has been proposed that can quickly render high-dimensional VGG (Visual Geometry Group) features. However, this approach requires prior learning of a style image and integrating it into 3D Gaussian features. While this method achieves some success in 3D stylized editing, it still suffers from issues such as leakage of the reference style image content and excessive editing of the 3D scene geometry (e.g., deformation). Summary of the Invention

[0004] In view of this, the present invention provides a method, apparatus, device and medium for stylizing the image of a three-dimensional scene to improve the 3D stylization effect based on a reference style image, and ensure the fidelity and multi-perspective consistency of the 3D content while ensuring the consistency of the style.

[0005] In a first aspect, the present invention provides a method for stylizing an image of a three-dimensional scene, the method comprising:

[0006] Obtaining a source 3D model and a reference style image, and obtaining a first image stylization editing network and a second image stylization editing network;

[0007] Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image;

[0008] Using the content features and the style features as denoising information, and using image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network to which a low-rank adaptation function is added, a third image stylization editing network is constructed, and a low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network;

[0009] Inputting the image to be edited and the reference style image into the third image stylization editing network for processing, and outputting a stylized single-view image;

[0010] The stylized edited single-view image is iteratively updated using an optimization strategy based on a combination of 3D content and style to obtain a stylized edited 3D model.

[0011] In conjunction with the first aspect, in one possible implementation, decoupling to obtain style features of the reference style image includes:

[0012] Parsing the reference style image using an image title generation method to obtain a reference text of the reference style image;

[0013] Extracting text features of the reference text using a text encoder in a pre-trained model, and using the text features as content features of the reference style image;

[0014] Extracting image features of the reference style image using an image encoder in a pre-trained model;

[0015] The image features and the content features are calculated using the pre-trained model to obtain style features of the reference style image.

[0016] The present invention provides a method for stylizing three-dimensional scenes. First, a first image stylization network is used to learn the content features of the image to be edited. Then, a second image stylization network is used to learn the style features of a reference style image. The two image stylization networks with added low-rank adaptation functions are optimized using image diffusion loss to construct a third image stylization network. Since the third image stylization network adds a low-rank adaptation function, such as a LoRA layer, to its cross-attention layer and is decoupled from the content and style features of the reference style image, the third image stylization network is used to process the image to be edited and the reference style image, outputting a single-view image that conforms to the style of the reference style image. Finally, based on the joint optimization of the 3D content and style, the single-view image after stylization is iteratively updated to obtain a stylized 3D model. This improves the 3D stylization effect based on the reference style image and ensures the fidelity and multi-view consistency of the 3D content while ensuring style consistency.

[0017] In conjunction with the first aspect, in another possible implementation, if there are multiple reference style images that belong to the same style, using a text encoder in a pre-trained model to extract text features of the reference text and using the text features as content features of the reference style image includes:

[0018] Use a large language model to synthesize multiple reference texts into a text description;

[0019] Extracting total text features of the text description using a text encoder in the pre-trained model, and using the total text features as content features of multiple reference style images;

[0020] Use the image encoder in the pre-trained model to extract image features of the reference style image, including:

[0021] Using the image encoder to extract image features of multiple reference style images respectively;

[0022] An average of the image features of the multiple reference style images is taken to obtain the image feature.

[0023] In combination with the first aspect, in yet another possible implementation, before using the first image stylization editing network to learn the content features of the image to be edited, the method further includes:

[0024] Get the basic diffusion model;

[0025] Adding a low-rank adaptation function to train and optimize the basic diffusion model to generate the first image stylization editing network;

[0026] Among them, the first image stylization editing network includes 11 converter modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0027] In combination with the first aspect, in another possible implementation, the content features and the style features are used as denoising information, and the image diffusion loss is used to optimize the first image stylization editing network and the second image stylization editing network that add a low-rank adaptation function, respectively, to construct a third image stylization editing network, including:

[0028] Inputting the content features of the image to be edited into the first image stylization editing network as first denoising information, adding a low-rank adaptation function to the first image stylization editing network, and optimizing the first image stylization editing network with the low-rank adaptation function using the image diffusion loss;

[0029] Inputting the style features of the reference style image into the second image stylization editing network as second denoising information, adding a low-rank adaptation function to the second image stylization editing network, and optimizing the second image stylization editing network with the added low-rank adaptation function using the image diffusion loss;

[0030] Based on the two optimized image stylization editing networks, a third image stylization editing network is constructed.

[0031] In combination with the first aspect, in another possible implementation, constructing a third image stylization editing network based on the two optimized image stylization editing networks includes:

[0032] According to the optimized first image stylization editing network and the optimized second image stylization editing network, among the 11 converter modules, the 5th and 6th converter modules are replaced with the 5th and 6th converter modules in the second image stylization editing network; the 1st to 4th and 7th to 11th converter modules are replaced with the corresponding modules in the first image stylization editing network to generate the third image stylization editing network.

[0033] Among them, a low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

[0034] In combination with the first aspect, in yet another possible implementation, the method further includes:

[0035] In the third image stylization editing network, inputting the content features of the image to be edited into all converter modules in the third image stylization editing network as content denoising guidance information;

[0036] The style features of the reference style image are input into the fifth and sixth converter modules of the third image stylization editing network as style denoising guidance information.

[0037] In conjunction with the first aspect, in yet another possible implementation, the iterative updating of the stylized edited monoscopic image to obtain the stylized edited 3D model using an optimization strategy based on a combination of 3D content and style includes:

[0038] Constructing a multi-loss function, wherein the multi-loss function includes: stylization loss, content preservation loss, geometry preservation loss and consistency loss;

[0039] Obtaining a loss weight coefficient corresponding to each loss function, and calculating multiple loss parameters based on the loss weight coefficient and the multiple loss functions;

[0040] The stylized edited single-view image is iteratively updated according to the multi-loss parameter constraints by utilizing an optimization strategy based on a combination of 3D content and style to obtain the stylized edited 3D model.

[0041] In combination with the first aspect, in yet another possible implementation, constructing a stylized loss in a multi-loss function includes:

[0042] Using the features extracted by the pre-trained convolutional neural network, the loss function between the rendered image and the style image under a set of training perspectives is calculated. The stylized loss is expressed as:

[0043]

[0044] in, is the stylized loss, Features extracted from pre-trained convolutional neural networks for stylized 3D Gaussian model rendering images, The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

[0045] In combination with the first aspect, in yet another possible implementation, constructing the content preservation loss in the multi-loss function includes:

[0046] The content retention loss is calculated using a content retention loss relationship, which is:

[0047]

[0048] in, For content retention loss, The features extracted by the pre-trained convolutional neural network are used to render the stylized 3D Gaussian model image. The features of the source 3D Gaussian model rendered image extracted by the pre-trained convolutional neural network, H is the length of the rendered image, and W is the width of the rendered image.

[0049] In combination with the first aspect, in another possible implementation, constructing the geometry-preserving loss in the multi-loss function includes:

[0050] Obtaining a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model;

[0051] The geometry preservation loss is calculated based on the depth image obtained by rendering the source three-dimensional model, the depth image obtained by rendering the edited three-dimensional model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is:

[0052]

[0053] in, is the geometry preservation loss, The depth image rendered from the source 3D model. The depth image obtained by rendering the edited 3D model, the depth image of the source 3D model after rendering The depth of each pixel Calculated by the following relationship:

[0054]

[0055]

[0056] in, is the 2D Gaussian after the ith 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the 2D Gaussian after the i-th projection of the source 3D model.

[0057] In combination with the first aspect, in another possible implementation, constructing a consistency loss in a multi-loss function includes:

[0058] Use the pre-trained image caption generation model to describe the content and style of the edited 3D model rendering image, and obtain content description text and style description text respectively;

[0059] Encoding the content description text and the style description text using a text encoder to obtain content description text features and style description text features;

[0060] The consistency loss is calculated according to the content description text features and the style description text features using a semantic consistency loss relationship, where the semantic consistency loss relationship is:

[0061]

[0062] in, is the consistency loss, The features of the image after the source 3D model is rendered are extracted by the image editor. Characterize the text for style description, is the style feature of the reference style image.

[0063] In a second aspect, the present invention provides a device for stylizing an image of a three-dimensional scene, the device comprising:

[0064] An acquisition module, configured to acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network;

[0065] a processing module, configured to render the source 3D model to obtain a single-view image as the image to be edited, and decouple the content features of the image to be edited and the style features of the reference style image;

[0066] A construction module is configured to use the content features and the style features as denoising information, and to optimize the first image stylization editing network and the second image stylization editing network to which a low-rank adaptation function is added using image diffusion loss, thereby constructing a third image stylization editing network, wherein the low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network;

[0067] The processing module is further configured to input the image to be edited and the reference style image into the third image stylization editing network for processing, and output a stylized single-view image;

[0068] The editing module is used to iteratively update the stylized edited single-view image using an optimization strategy based on a combination of 3D content and style to obtain a stylized edited 3D model.

[0069] In a third aspect, the present invention provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image stylization editing method for a three-dimensional scene according to the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0070] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image stylization editing method for a three-dimensional scene according to the first aspect or any corresponding embodiment thereof.

[0071] In addition, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the image stylization editing method for a three-dimensional scene according to the first aspect or any corresponding embodiment thereof.

[0072] The present invention provides a method, apparatus, device, and medium for stylizing images of three-dimensional scenes. The method first utilizes a first image stylization network to learn the content features of the image to be edited, and then utilizes a second image stylization network to learn the style features of a reference style image. The method then utilizes image diffusion loss to optimize the two image stylization networks that have been added with low-rank adaptation functions, thereby constructing a third image stylization network. Because the third image stylization network adds a low-rank adaptation function, such as a LoRA layer, to its cross-attention layer and is decoupled from the content and style features of the reference style image, the third image stylization network is used to process the image to be edited and the reference style image, outputting a single-view image that conforms to the style of the reference style image. Finally, the method jointly optimizes two perspectives based on 3D content and style, it iteratively updates the stylized single-view image, and obtains a stylized 3D model. This improves the 3D stylization effect based on the reference style image, while ensuring the fidelity and multi-view consistency of the 3D content while maintaining consistent style.

[0073] This paper also proposes a method for stylized image editing that decouples content and style. This method decouples the content and style features of a reference style image, extracting both the content and style features of the reference style image as denoising conditions to guide image stylized editing. Furthermore, after decoupling, the content and style of the reference style image are decoupled in the aligned space, effectively preventing content leakage during the stylization process.

[0074] This method also reduces training costs, accelerates the training process, and obtains a trained image stylization network. Using an optimization strategy based on the combination of 3D content and style, the stylized single-view image is iteratively updated. Multiple loss functions, such as stylization loss, content preservation loss, geometry preservation loss, and consistency loss, are added to this process to further ensure that the content structure of the source 3D model remains unchanged, prevent excessive 3D geometry deformation during the stylization of the 3D Gaussian model, and improve the semantic consistency of the 3D stylization. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0076] Figure 1 3D stylized editing process based on content and style decoupling according to an embodiment of the present invention;

[0077] Figure 2is a schematic flow chart of a method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention;

[0078] Figure 3 is a flowchart of another method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention;

[0079] Figure 4 is a flowchart of image stylization editing based on content and style decoupling according to an embodiment of the present invention;

[0080] Figure 5 is a flowchart of another method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention;

[0081] Figure 6 is a schematic diagram of a Web-based 3D stylized editing platform interface according to an embodiment of the present invention;

[0082] Figure 7 is a structural block diagram of an image stylization editing device according to an embodiment of the present invention;

[0083] Figure 8 FIG. 4 is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0084] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0085] In addition, the terms “first” and “second” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0086] The technical features involved in different embodiments of the present invention described in the present invention can be combined with each other as long as they do not conflict with each other.

[0087] To address the issues of reference style image content leakage and over-editing of 3D scene geometry, an embodiment of the present invention proposes a three-dimensional stylized editing method. This method improves the 3D stylization effect based on the reference style image from two perspectives: decoupling the reference style image content and style features, and jointly optimizing 3D content and style. While ensuring style consistency, it also ensures the fidelity and multi-view consistency of the 3D content.

[0088] Specifically, this embodiment provides a method for stylizing an image of a three-dimensional scene based on decoupling of content and style. Figure 1 As shown, the main steps are as follows: First, a source 3D model to be edited and a reference style image are given. Then, the source 3D model is rendered to obtain a monoscopic image. For this rendered monoscopic image, the proposed image stylization editing model based on content and style decoupling is used to complete image editing that matches the style of the reference style image. The rendered monoscopic image is then processed using the image stylization editing model or network to obtain a monoscopic image in the target style. Finally, a strategy based on the combined optimization of 3D content and style is used to iteratively update the set of stylized edited images, promoting the stylized image editing results to a 3D scene, thereby achieving 3D stylized editing.

[0089] The following describes in detail the image stylization editing method for a three-dimensional scene provided by this embodiment.

[0090] An embodiment of the present invention provides an embodiment of a method for stylizing an image of a three-dimensional scene. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0091] In this embodiment, a method for stylizing an image of a three-dimensional scene is provided, which can be used on the above-mentioned server, client or terminal device, such as a mobile phone, tablet computer, etc. Figure 2 is a flowchart of a method according to an embodiment of the present invention, which includes:

[0092] Step S101: Obtain a source 3D model and a reference style image, and obtain a first image stylization editing network and a second image stylization editing network.

[0093] The source 3D model refers to a model containing 3D data constructed in a virtual 3D space using 3D production software. These models are commonly used in computer and video games, special effects production for films and television programs, and even in fields such as engineering design, architectural design, and product manufacturing. In this embodiment, the source 3D model refers to a 3D environment image captured in a real-world setting. The reference style image is a given image.

[0094] Optionally, the reference style image, i.e. the image to be edited, is represented as The number of reference style images can be one or more, and the number of images to be edited can also be one or more.

[0095] The first and second image stylization editing networks have the same structure and can include multiple transformer modules for editing the style of an image. The first image stylization editing network is used to learn the content features of the image to be edited, while the second image stylization editing network is used to learn the style features of the reference style image.

[0096] Step S102: Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image.

[0097] Specifically, the decoupling method is used to decouple the image to be edited to obtain the content features of the image to be edited. The decoupling method is also used to decouple the reference style image to obtain the content features and style features corresponding to the reference style image.

[0098] In this embodiment, the content and style features of the reference style image are decoupled. The reference style image content and style features are obtained as denoising conditions to guide image stylization editing. Furthermore, the reference style image content and style are decoupled in the alignment space to prevent leakage of the reference style image content during the stylization process.

[0099] Step S103: Content features and style features are used as denoising information, and image diffusion loss is used to optimize the first image stylization editing network and the second image stylization editing network with added low-rank adaptation function, and then the third image stylization editing network is constructed.

[0100] First, a low-rank adaptation function, such as a LoRA layer, is added to the Transformer modules in the first and second image stylization networks. Then, the first and second image stylization networks with the LoRA layer are optimized and trained using image diffusion loss to generate a third image stylization network. A low-rank adaptation function (LoRA layer) is also added to the cross-attention layer in this third image stylization network.

[0101] LoRA layer: Low-Rank Adaptation reduces the number of parameters by decomposing the weight matrix into the product of low-rank matrices, thereby reducing hardware resources and accelerating the fine-tuning process.

[0102] Step S104: input the image to be edited and the reference style image into a third image stylization editing network for processing, and output a stylized single-view image.

[0103] Step S105: using an optimization strategy based on the combination of 3D content and style, iteratively updating the stylized edited single-view image to obtain a stylized edited 3D model.

[0104] This step uses a strategy based on the joint optimization of 3D content and style to enhance the stylized editing results of the single-view image to the 3D scene, thereby achieving 3D stylized editing.

[0105] The present invention provides a method for stylizing three-dimensional scenes. First, a first image stylization network is used to learn the content features of the image to be edited. Then, a second image stylization network is used to learn the style features of a reference style image. The two image stylization networks with added low-rank adaptation functions are optimized using image diffusion loss to construct a third image stylization network. Since the third image stylization network adds a low-rank adaptation function, such as a LoRA layer, to its cross-attention layer and is decoupled from the content and style features of the reference style image, the third image stylization network is used to process the image to be edited and the reference style image, outputting a single-view image that conforms to the style of the reference style image. Finally, based on the joint optimization of the 3D content and style, the single-view image after stylization is iteratively updated to obtain a stylized 3D model. This improves the 3D stylization effect based on the reference style image and ensures the fidelity and multi-view consistency of the 3D content while ensuring style consistency.

[0106] Specifically, in a possible implementation of this embodiment, as Figure 3 As shown, in step S102, decoupling to obtain the style features of the reference style image includes:

[0107] Step S301: Analyze the reference style image using the image title generation method to obtain the reference text of the reference style image. The reference style image can be represented as “ ”.

[0108] Step S302: Extract using the text encoder in the pre-trained model , and use text features as content features of the reference style image.

[0109] Step S303: Utilize the image encoder in the pre-trained model to extract image features of the reference style image.

[0110] Step S304: Calculate the image features and content features using the pre-trained model to obtain style features of the reference style image.

[0111] Specifically, input a single reference style image , use image caption generation methods, such as BLIP (Bootstrapping Language-Image Pre-training, a pre-training model that unifies visual language understanding and generation) to describe the content of the reference style image and obtain the reference style text , extract reference style text using the text encoder in the pre-trained CLIP model The text features of the reference style image are used as the content features of the reference style image , which is the reference text in the above step S301.

[0112] The BLIP model aims to unify vision and language understanding and generation tasks through pre-training. It can extract the main content of an image and generate descriptive captions or answer questions related to the image based on this content. Furthermore, the BLIP model demonstrates strong generalization capabilities and can be applied to a variety of vision-language tasks, such as image-text retrieval, image description generation, and visual question answering.

[0113] CLIP (Contrastive Language-Image Pre-Training, a text-image pre-training model using contrastive learning) can map images and text into a shared vector space, which can be used for various vision and language tasks. In this embodiment, the image encoder of CLIP is used to extract the reference style image. Image features , through the pre-trained CLIP model, and Transformed to the same embedding space, addition and subtraction can be performed to obtain the reference style image Style characteristics . Among them, the reference style image Style characteristics .

[0114] In this embodiment, the reference style image is displayed Content Features and style characteristics Decoupling is performed to prevent the leakage of reference style image content during the stylization process.

[0115] Furthermore, in another embodiment, if there are multiple reference style images and they belong to the same style, then the above-mentioned step S302 includes: using a large language model to synthesize multiple reference texts into a text description; using the text encoder in the pre-trained model to extract the total text features of the text description, and using the total text features as the content features of multiple reference style images; step S303 includes: using an image encoder to respectively extract image features of multiple reference style images; taking the average of the image features of multiple reference style images to obtain image features.

[0116] Specifically, for multiple reference style images of the same style (for example, 10 reference style images ), we first use the image title generation method BLIP to process multiple reference style images to obtain the corresponding image content description text (i.e., reference text), and then use the language model to synthesize multiple description texts into a text description to facilitate the extraction of text features. The text editor in the pre-trained CLIP model is used to extract the above merged text description to obtain the content features of multiple reference style images. .

[0117] At the same time, the image encoder in CLIP is used to obtain the image features of multiple reference style images respectively , and then take the mean to obtain the image features of the reference style image , or called , and thus calculate the style feature as .

[0118] The following embodiments are described based on the input of a single reference style image.

[0119] In another embodiment of the present application, the above step S102: obtaining a first image stylization editing network includes:

[0120] Get the basic diffusion model, add the low-rank adaptation function (LoRA layer) to train and optimize the basic diffusion model, and generate the first image stylization editing network. For example, the symbolic representation of the first image stylization editing network is .

[0121] Among them, the first image stylization editing network It includes 11 transformer modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature cues as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0122] Optionally, the base diffusion model can be an SDXL model. Furthermore, SDXL (Stable DiffusionXL) is a generative pre-trained model based on the diffusion model, specifically for text-to-image generation tasks. Diffusion process: In SDXL, the diffusion process involves gradually adding noise to the original data, gradually approximating it to a random distribution. This process is one of the key steps in the model's learning of the data distribution. Reverse process: Starting from a random distribution, the reverse process gradually removes noise, ultimately generating high-quality data samples. This process can be viewed as a denoising process. Through continuous iteration and optimization, the model can generate data samples that are increasingly close to real data.

[0123] This embodiment first constructs a first image stylization editing network and trains the network. The goal is to input a single reference style image The content and style of the image are further decoupled and processed by separate modules in the first image stylization editing network to mitigate content leakage of the reference style image and improve the generalization of the stylization network to different stylizations.

[0124] like Figure 4 As shown, this embodiment provides a method for constructing an image stylization editing network based on content and style decoupling. The method utilizes a pre-trained SDXL text-to-image generation model and uses LoRA to train and optimize the content and style transformation modules in the SDXL model.

[0125] like Figure 4 As shown, the first image stylization editing network Based on the pre-trained SDXL diffusion model, the first image stylization editing network The SDXL model consists of 11 Transformer modules, including four downsampling modules, one intermediate module, and six upsampling modules. Each Transformer module includes both self-attention and cross-attention layers. The cross-attention layers are used to guide image generation by incorporating text or image feature cues. Optimizing all parameters of SDXL from scratch is resource-intensive and difficult to train. Therefore, we propose adding the LoRA layer and optimizing only the cross-attention layers in the Transformer.

[0126] In addition, in the method provided in this embodiment, the first image stylization editing network is used Learning the image to be edited ( ) content features, and using the second image stylization editing network Learning from reference style images Style features include:

[0127] Get the image to be edited Content Features ; The image to be edited Content Features Input first image stylization editing network , after being processed by 11 converter modules, the image to be edited is obtained ; The reference style image Style characteristics Input the second image stylization editing network and generate a reference style image after processing .

[0128] Specifically, in order to reduce the training cost and speed up the training process, a trained image stylization network is obtained. Given a reference style image and the image to be edited , using the above step S102, the reference style image Decouple the content and style features of the image to obtain the decoupled image content and style features and Then, the image caption generation model BLIP is used to generate the given image to be edited. Generate a text description of the image and use the text encoder in the pre-trained CLIP to encode the text description to obtain the content features of the image to be edited .

[0129] Specifically, a possible implementation of the above step S103 is: the content features of the image to be edited Input to the first stylized editing network All Transformer modules in the image are used as the first denoising information, and LoRA layers are added to the cross-attention layers of all Transformer modules to optimize the first image stylization editing network using image diffusion loss. The LoRA layer is used to optimize the first image stylization editing network after training. , the network can be used to generate images to be edited .

[0130] Similarly, the above reference style image Image style features Input to the second stylized editing network All Transformer modules in the image are used as the second denoising information, and LoRA layers are added to the cross-attention layers of all Transformer modules to optimize the second image stylization editing network using image diffusion loss. , get the second image stylization editing network after optimization training , the network can be used to generate reference style images .

[0131] Finally, based on the two optimized image stylization editing networks, a third image stylization editing network is constructed ( ).

[0132] In the above step S103, a third image stylization editing network is constructed ( One implementation of the proposed method is to replace the 5th and 6th transformer modules with a second image stylization network in the 11 transformer modules. The 5th and 6th converter modules in

[15] are replaced by the first image stylization editing network. The corresponding module in generates the third image stylization editing network .

[0133] Also includes: Input to the third image stylization editing network All Transformer modules in , as content denoising guidance information, will Input to the third image stylization editing network The 5th and 6th Transformer modules are used as style denoising guidance information. Add LoRA layers to the cross-attention layers of the 5th and 6th Transformer modules, and optimize the stylized editing network using image diffusion loss Generate stylized images .

[0134] In the above step S104, during inference, the image to be edited is input and reference style images , stylized editing network , you can output the stylized edited image .

[0135] Furthermore, in another embodiment, Figure 5 As shown, the above step S105: using the optimization strategy based on the combination of 3D content and style to iteratively update the stylized single-view image to obtain the stylized 3D model, specifically includes:

[0136] Step S1051: Constructing a multi-loss function, wherein the multi-loss function includes at least one of a stylization loss, a content preservation loss, a geometry preservation loss, and a consistency loss.

[0137] Step S1052: Obtain the loss weight coefficient corresponding to each loss function, and calculate multiple loss parameters based on the loss weight coefficient and multiple loss functions.

[0138] One implementation method is to calculate the multi-loss function using the following formula:

[0139]

[0140] in, is the corresponding loss weight coefficient, Stylized loss The corresponding loss weight coefficient is, is the stylized loss, Keep loss for content The corresponding loss weight coefficient is, Preserve loss for geometry The corresponding loss weight coefficient is, is the geometry preservation loss, Consistency loss The corresponding loss weight coefficient is, is the consistency loss.

[0141] Among them, the loss weight coefficient corresponding to each loss can be customized by the system.

[0142] Step S1053: Using the optimization strategy based on 3D content and style, iteratively update the stylized single-view image according to the multi-loss parameter constraint , and obtain the stylized edited 3D model.

[0143] This embodiment uses the above-mentioned multiple loss functions to constrain the content fidelity and style consistency of the stylized 3D editing model, so that the 3D model is stylized according to the reference style image while reducing the substantial damage to the geometric content.

[0144] The following is a detailed description of the process of determining each loss function in each multi-loss function.

[0145] First, given the source 3D model, it is represented by a 3D Gaussian. Given the multi-view images of the source 3D scene, the source 3D model is obtained by reconstruction and represented by a 3D Gaussian. ,in , , Indicates the position of the center point, Represents the covariance matrix, and the color of each point is determined by Indicates transparency , then the three-dimensional Gaussian expression is:

[0146]

[0147] in, Represents the query point and the center point distance, represents the covariance matrix, -1 represents the inverse transformation, represents the i-th three-dimensional Gaussian point, T represents the transposed matrix,

[0148] Then, based on the rendered single-view image of the source 3D model The color of a pixel is:

[0149]

[0150]

[0151] The goal of this embodiment is to perform 3D stylization on the source 3D model so that the Gaussian representation of the stylized 3D model Capturing the style of a reference style image while maintaining its original content structure .

[0152] Utilizing the strategy of joint optimization of 3D content and style, the rendered image is iteratively stylized and edited, and 3D stylized editing is achieved using multi-loss optimization.

[0153] Specifically, one implementation method is to use a three-dimensional Gaussian to represent the edited three-dimensional model. ,in , , Indicates the position of the center point, Represents the covariance matrix, and the color of each point is determined by Indicates transparency , then the three-dimensional Gaussian representation of the edited three-dimensional model is:

[0154]

[0155] in, Represents the query point and the center point distance.

[0156] A single-view image of the edited 3D model rendering The color of a pixel is:

[0157]

[0158]

[0159] This three-dimensional Gaussian model Source 3D Gaussian model Perform optimization and then perform optimization training.

[0160] Edited 3D Gaussian model Render along a certain perspective to get the current rendered image, according to the input reference style image , using the third image stylization editing model Edit and obtain a single-view stylized edited image that matches the style of the reference image .

[0161] The rendered image from the new perspective is iteratively rendered, and the above editing process is repeated to update the edited image data. The edited image is directly back-propagated into the 3D Gaussian scene so that it is continuously updated according to the input reference style image, thereby generating an edited 3D Gaussian scene that conforms to the given style.

[0162] Specifically, in step S1051, the stylized loss in the multi-loss function is constructed. ,include:

[0163] Using features extracted by a pre-trained convolutional neural network (e.g., VGG), a loss function is calculated between the rendered image and the style image at a set of training viewpoints. To transfer detailed high-frequency style features from the 2D style image to the 3D scene, this embodiment minimizes the cosine distance between the rendered image features and their nearest neighbors in the style image features. The stylization loss is expressed as:

[0164]

[0165] in, is the stylized loss, Pre-training convolutional neural network features for stylized 3D Gaussian model rendering images, The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

[0166] Optionally, in another embodiment, in order to further ensure that the content structure of the source 3D model remains unchanged, a content preservation loss is proposed. , the above step S1051: constructing the content preservation loss in the multi-loss function , including: calculating the content preservation loss using a content preservation loss relation, the content preservation loss relation is:

[0167]

[0168] in, For content retention loss, The features extracted by the pre-trained convolutional neural network are used to render the 3D Gaussian model after stylized editing. The features of the source 3D Gaussian model rendered image are extracted by the pre-trained convolutional neural network. H is the length of the rendered image, and W is the width of the rendered image.

[0169] Optionally, in another embodiment, since the geometric parameters of the 3D Gaussian model still cause unexpected geometric transformations of the scene during the optimization process, in order to further prevent the 3D geometry from being excessively deformed during the stylization process of the 3D Gaussian model, the above step S1051: constructing the geometry preservation loss in the multi-loss function ,include:

[0170] Obtain a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model; calculate the geometry preservation loss based on the depth image obtained by rendering the source 3D model, the depth image obtained by rendering the edited 3D model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is:

[0171]

[0172] in, is the geometry preservation loss, The depth image rendered from the source 3D model. The depth image resulting from the rendered 3D model after editing, where d is the depth of each pixel. The 3D Gaussian fusion algorithm can be an alpha-blending method. Alpha-blending is a computer graphics technique used to blend two or more images or renderings based on their alpha values ​​(usually indicating transparency). This method is often used to achieve transparency effects, such as rendering semi-transparent objects like glass, smoke, or flames.

[0173] Depth image of the source 3D model after rendering The depth of each pixel Calculated by the following relationship:

[0174]

[0175]

[0176] in, is the 2D Gaussian after the ith 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the 2D Gaussian after the i-th projection of the source 3D model.

[0177] Optionally, in another embodiment, in order to further improve the semantic consistency of 3D stylization, this embodiment also proposes a semantic consistency loss The above step S1051: constructing the consistency loss in the multi-loss function includes:

[0178] The pre-trained image caption generation model BLIP is used to describe the content and style of the edited 3D model rendering image, and the content description text and style description text are obtained respectively. The content description text and style description text are encoded using a text encoder to obtain content description text features and style description text features. Based on the content description text features and style description text features, the semantic consistency loss relationship is used to calculate the consistency loss. The semantic consistency loss relationship is:

[0179]

[0180] in, is the consistency loss, The features of the image after the source 3D model is rendered are extracted by the image editor. Characterize the text for style description, is the style feature of the reference style image.

[0181] The method provided in this embodiment also includes: utilizing an interactive interface to implement user-friendly 3D stylized editing.

[0182] Specifically, a Web-based 3D model editing platform is constructed to provide an operable UI interface to assist users in completing editing. To assist users in completing editing, this embodiment also constructs a Web-based 3D model editing platform. The interface diagram is shown in FIG. Figure 6 Click "Enter source 3D model" to open the 3D model input box, which supports inputting a 3D Gaussian model. Click "Enter reference style image" to open the image input box, where you can enter the reference style image. Click "Start stylized editing" to automatically train the 3D model stylized editing in the background, with prompts popping up at each stage of training completion. The final output is the stylized 3D model.

[0183] The method proposed in this embodiment utilizes a strategy for joint optimization of 3D content and style to iteratively stylize rendered images, using multi-loss optimization to achieve 3D stylized editing. Furthermore, an interactive interface is proposed, providing an operational UI for user-friendly 3D stylized editing.

[0184] The present invention provides a method for stylizing images of three-dimensional scenes. First, a first image stylization network is used to learn the content features of the image to be edited. Then, a second image stylization network is used to learn the style features of a reference style image. The two trained image stylization networks are then fused to form a third image stylization network. Since a low-rank adaptation function, such as a LoRA layer, is added to the cross-attention layer of the third image stylization network and the network is decoupled from the content and style features of the reference style image, the third image stylization network is used to process the image to be edited and the reference style image, outputting a single-view image that conforms to the style of the reference style image. Finally, based on the joint optimization of the 3D content and style, the single-view image after stylization is iteratively updated to obtain a stylized 3D model. This improves the 3D stylization effect based on the reference style image and ensures the fidelity and multi-view consistency of the 3D content while ensuring style consistency.

[0185] This paper also proposes a method for stylized image editing that decouples content and style. This method decouples the content and style features of a reference style image, extracting both the content and style features of the reference style image as denoising conditions to guide image stylized editing. Furthermore, after decoupling, the content and style of the reference style image are decoupled in the aligned space, effectively preventing content leakage during the stylization process.

[0186] This method also reduces training costs, accelerates the training process, and obtains a trained image stylization network. Using an optimization strategy based on the combination of 3D content and style, the stylized single-view image is iteratively updated. Multiple loss functions, such as stylization loss, content preservation loss, geometry preservation loss, and consistency loss, are added to this process to further ensure that the content structure of the source 3D model remains unchanged, prevent excessive 3D geometry deformation during the stylization of the 3D Gaussian model, and improve the semantic consistency of the 3D stylization.

[0187] It should be noted that the method proposed in this invention can be used to efficiently edit existing 3D assets, obtain a wider variety of 3D assets, and save 3D asset modeling time and costs. It can also support the development of Metaverse servers such as MetaEngine and AI servers, as well as their applications in 3D content generation and editing.

[0188] This embodiment also provides a device for stylizing an image of a three-dimensional scene. The device is used to implement the above-mentioned embodiments and preferred embodiments. Details already described will not be repeated here. As used below, the term "module" may refer to a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation using hardware, or a combination of software and hardware, is also possible and contemplated.

[0189] This embodiment provides a three-dimensional scene image stylization editing device, such as Figure 7 As shown, the device includes: an acquisition module 710, a processing module 720, a construction module 730 and an editing module 740. In addition, the device may also include other more or fewer units or modules, which is not limited in this embodiment.

[0190] The acquisition module 710 is configured to acquire a source 3D model and a reference style image, and to acquire a first image stylization editing network and a second image stylization editing network.

[0191] The processing module 720 is configured to render the source 3D model to obtain a single-view image as the image to be edited, and decouple the content features of the image to be edited and the style features of the reference style image.

[0192] Construction module 730 is used to use content features and style features as denoising information, use image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network that add low-rank adaptation functions, construct a third image stylization editing network, and add a low-rank adaptation function to the cross-attention layer in the third image stylization editing network.

[0193] The processing module 720 is further configured to input the image to be edited and the reference style image into a third image stylization editing network for processing, and output a stylized, edited single-view image.

[0194] The editing module 740 is configured to iteratively update the stylized edited single-view image using an optimization strategy based on a combination of 3D content and style to obtain a stylized edited 3D model.

[0195] In some optional implementations, the processing module 720 is specifically configured to generate a title for the reference style image ( ) to parse and obtain the reference text of the reference style image ( ); Extract reference text using the text encoder in the pre-trained model ( ) and uses the text features as the content features of the reference style image ( ); Use the image encoder in the pre-trained model to extract the reference style image ( ) image features ( ); Use the pre-trained model to transform the image features ( ) and content features ( ) is transformed into the same embedding space to obtain the style features of the reference style image ( ).

[0196] Optionally, in some optional implementations, the processing module 720 is further configured to, if there are multiple reference style images and they are of the same style, synthesize the multiple reference texts into a text description using the language model; extract the total text features of the text description using the text encoder in the pre-trained model, and use the total text features as the content features of the multiple reference style images ( ); Use the image encoder to extract the image features of multiple reference style images respectively; take the average of the image features of multiple reference style images to obtain the image features ( ).

[0197] Optionally, in some other optional implementations, the construction module 730 is further used to obtain a basic diffusion model, add a low-rank adaptation function (LoRA layer) to train and optimize the basic diffusion model, and generate a first image stylization editing network.

[0198] Among them, the first image stylization editing network includes 11 transformer modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0199] Optionally, in another optional embodiment, a construction module 730 is specifically used to input the content features of the image to be edited into the first image stylization editing network as the first denoising information, and add a low-rank adaptation function in the first image stylization editing network, and optimize the first image stylization editing network with the added low-rank adaptation function using image diffusion loss; input the style features of the reference style image into the second image stylization editing network as the second denoising information, and add a low-rank adaptation function in the second image stylization editing network, and optimize the second image stylization editing network with the added low-rank adaptation function using image diffusion loss; and construct a third image stylization editing network based on the two optimized image stylization editing networks.

[0200] Optionally, in some further optional embodiments, the construction module 730 is further used to replace the 5th and 6th converter modules among the 11 converter modules with the 5th and 6th converter modules in the second image stylization editing network based on the first image stylization editing network and the second image stylization editing network after feature learning; and replace the 1st to 4th and 7th to 11th converter modules with corresponding modules in the first image stylization editing network to generate a third image stylization editing network.

[0201] Among them, low-rank adaptation functions are added to the cross-attention layers of the 5th and 6th converter modules in the third image stylization editing network.

[0202] In addition, construction module 730 is also used to input the content features of the image to be edited into all converter modules in the third image stylization editing network as second denoising information in the third image stylization editing network; and input the style features of the reference style image into the 5th and 6th converter modules of the third image stylization editing network as second denoising information.

[0203] Optionally, in some other optional embodiments, the editing module 740 is specifically configured to construct multiple loss functions, obtain the loss weight coefficient corresponding to each loss function, and calculate multiple loss parameters based on the loss weight coefficient and the multiple loss functions; utilize an optimization strategy based on the combination of 3D content and style, and iteratively update the stylized edited single-view image according to the constraints of the multiple loss parameters ( ), and obtain the stylized 3D model.

[0204] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0205] The image stylization editing of the three-dimensional scene in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0206] An embodiment of the present invention further provides an electronic device having the above Figure 7 Stylized image editing of the 3D scene shown.

[0207] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, the electronic device comprising: one or more processors 10, a memory 20, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other by means of different buses for communication, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface).

[0208] In some optional embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple storages if desired. Similarly, multiple electronic devices can be connected, with each device providing part of the necessary operations (e.g., as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0209] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0210] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable the at least one processor 10 to execute the image stylization editing method for the three-dimensional scene shown in the above embodiment.

[0211] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system and applications required for at least one function; the data storage area may store data created based on the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may include a memory remotely located relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0212] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0213] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0214] The input device 30 can receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. Examples include a touch screen, keypad, mouse, trackpad, touchpad, pointer, one or more mouse buttons, trackball, joystick, etc. The output device 40 may include a display device, auxiliary lighting devices (e.g., LEDs), and tactile feedback devices (e.g., vibration motors). Such display devices include, but are not limited to, liquid crystal displays, light emitting diodes, monitors, and plasma displays. In some optional embodiments, the display device may be a touch screen.

[0215] The electronic device further includes at least one communication interface for the electronic device to communicate with other devices or a communication network.

[0216] An embodiment of the present invention also provides a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded on a storage medium, or downloaded via a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware.

[0217] The storage medium may be a magnetic disk, an optical disk, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, among others; further, the storage medium may include a combination of the aforementioned types of memory. It is understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the image stylization editing method for a three-dimensional scene as described in the above embodiment is implemented.

[0218] The embodiments of the present application may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the above method. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, wherein the programming languages ​​include object-oriented programming languages ​​such as Java, C++, etc., and also include conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0219] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stylizing an image of a three-dimensional scene, characterized in that: The method comprises: Obtaining a source 3D model and a reference style image, and obtaining a first image stylization editing network and a second image stylization editing network; Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image, wherein the style features of the reference style image are obtained by and After transforming to the same embedding space and performing addition and subtraction calculations, is the content feature of the reference style image, is the image feature of the reference style image; Using the content features and the style features as denoising information, and using image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network with added low-rank adaptation function, to construct a third image stylization editing network; Inputting the image to be edited and the reference style image into the third image stylization editing network for processing, and outputting a stylized single-view image; Using an optimization strategy based on a combination of 3D content and style, and in accordance with multiple loss parameter constraints, iteratively updating the stylized single-view image to obtain a stylized 3D model; the multiple loss parameter constraints are calculated by a multiple loss function; The multi-loss function is: in, is the corresponding loss weight coefficient, Stylized loss The corresponding loss weight coefficient is, is the stylized loss, Keep loss for content The corresponding loss weight coefficient is, Preserve loss for geometry The corresponding loss weight coefficient is, is the geometry preservation loss, Consistency loss The corresponding loss weight coefficient is, is the consistency loss; The step of obtaining a first image stylization editing network includes: Using the pre-trained SDXL as the basic diffusion model, the basic diffusion model is obtained; adding a low-rank adaptation function to train and optimize the basic diffusion model to generate the first image stylization editing network, wherein the low-rank adaptation function is used to optimize the cross-attention layer in the converter module; The content features and the style features are used as denoising information, and the image diffusion loss is used to optimize the first image stylization editing network and the second image stylization editing network with the low-rank adaptation function, respectively, to construct a third image stylization editing network, including: Inputting the content features of the image to be edited into the first image stylization editing network as first denoising information, adding a low-rank adaptation function to the first image stylization editing network, and optimizing the first image stylization editing network with the low-rank adaptation function using the image diffusion loss; Inputting the style features of the reference style image into the second image stylization editing network as second denoising information, adding a low-rank adaptation function to the second image stylization editing network, and optimizing the second image stylization editing network with the added low-rank adaptation function using the image diffusion loss; According to the optimized first image stylization editing network and the optimized second image stylization editing network, among the 11 converter modules, the 5th and 6th converter modules are replaced by the 5th and 6th converter modules in the second image stylization editing network; the 1st to 4th and the 7th to 11th converter modules are replaced by the corresponding modules in the first image stylization editing network to generate the third image stylization editing network, wherein a low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

2. The method according to claim 1, characterized in that Decoupling to obtain the style features of the reference style image includes: Parsing the reference style image using an image title generation method to obtain a reference text of the reference style image; Extracting text features of the reference style image using a text encoder in a pre-trained model, and using the text features as content features of the reference style image; Extracting image features of the reference style image using an image encoder in a pre-trained model; The image features and the content features are calculated using the pre-trained model to obtain style features of the reference style image.

3. The method according to claim 2, characterized in that If there are multiple reference style images of the same style, the text features of the reference style images are extracted using the text encoder in the pre-trained model, and the text features are used as the content features of the reference style images, including: Use a large language model to synthesize multiple reference texts into a text description; Extracting total text features of the text description using a text encoder in the pre-trained model, and using the total text features as content features of multiple reference style images; Extracting image features of the reference style image by using an image encoder in the pre-trained model includes: extracting image features of a plurality of reference style images respectively using the image encoder; An average of the image features of the multiple reference style images is taken to obtain the image feature.

4. The method according to claim 1, wherein The first image stylization editing network includes 11 converter modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation.

5. The method according to claim 1, wherein The method further comprises: In the third image stylization editing network, inputting the content features of the image to be edited into all converter modules in the third image stylization editing network as content denoising guidance information; The style features of the reference style image are input into the fifth and sixth converter modules of the third image stylization editing network as style denoising guidance information.

6. The method according to any one of claims 1 to 5, characterized in that The method of iteratively updating the stylized single-view image to obtain the stylized 3D model by utilizing an optimization strategy based on a combination of 3D content and style includes: Constructing a multi-loss function, wherein the multi-loss function includes: stylization loss, content preservation loss, geometry preservation loss and consistency loss; Obtaining a loss weight coefficient corresponding to each loss function, and calculating multiple loss parameters based on the loss weight coefficient and the multiple loss functions; The stylized edited single-view image is iteratively updated according to the multi-loss parameter constraint by utilizing an optimization strategy based on a combination of 3D content and style to obtain the stylized edited 3D model.

7. The method according to claim 1, characterized in that Constructing the stylized loss in the multi-loss function includes: Using the features extracted by the pre-trained convolutional neural network, the loss function between the rendered image and the style image under a set of training perspectives is calculated. The stylized loss is expressed as: in, is the stylized loss, The features extracted by the pre-trained convolutional neural network are used to render the stylized 3D Gaussian model image. The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

8. The method according to claim 7, characterized in that Constructing the content preservation loss in the multi-loss function includes: The content retention loss is calculated using a content retention loss relationship, which is: in, For content retention loss, The features extracted by the pre-trained convolutional neural network are used to render the stylized 3D Gaussian model image. The features of the source 3D Gaussian model rendered image extracted by the pre-trained convolutional neural network, H is the length of the rendered image, and W is the width of the rendered image.

9. The method according to claim 8, characterized in that Constructing the geometry preservation loss in the multi-loss function includes: Obtaining a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model; The geometry preservation loss is calculated based on the depth image obtained by rendering the source three-dimensional model, the depth image obtained by rendering the edited three-dimensional model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is: in, is the geometry preservation loss, The depth image obtained by rendering the source 3D model, The depth image obtained by rendering the edited 3D model, the depth image of the source 3D model after rendering The depth of each pixel Calculated by the following relationship: in, is the 2D Gaussian after the ith 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the 2D Gaussian after the i-th projection of the source 3D model.

10. The method according to claim 7, characterized in that Constructing the consistency loss in the multi-loss function includes: Use the pre-trained image caption generation model to describe the content and style of the edited 3D model rendering image, and obtain content description text and style description text respectively; Encoding the content description text and the style description text using a text encoder to obtain content description text features and style description text features; The consistency loss is calculated according to the content description text features and the style description text features using a semantic consistency loss relationship, where the semantic consistency loss relationship is: in, is the consistency loss, The features of the image after the source 3D model is rendered are extracted by the image editor. Characterize the text for style description, is the style feature of the reference style image.

11. A device for stylizing an image of a three-dimensional scene, characterized in that: The device comprises: An acquisition module, configured to acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network; A processing module is configured to render the source 3D model to obtain a single-view image as the image to be edited, and decouple the content features of the image to be edited and the style features of the reference style image, wherein the style features of the reference style image are obtained by and After transforming to the same embedding space and performing addition and subtraction calculations, is the content feature of the reference style image, is the image feature of the reference style image; A construction module is configured to use the content features and the style features as denoising information, and to optimize the first image stylization editing network and the second image stylization editing network with a low-rank adaptation function using image diffusion loss to construct a third image stylization editing network; The processing module is further configured to input the image to be edited and the reference style image into the third image stylization editing network for processing, and output a stylized single-view image; An editing module, configured to iteratively update the stylized edited single-view image to obtain a stylized edited 3D model using an optimization strategy based on a combination of 3D content and style and in accordance with multiple loss parameter constraints; The acquisition module is specifically used to obtain the basic diffusion model based on the pre-trained SDXL; add a low-rank adaptation function to train and optimize the basic diffusion model to generate the first image stylization editing network, and the low-rank adaptation function is used to optimize the cross-attention layer in the converter module; the multi-loss parameter constraint is calculated by a multi-loss function; the multi-loss function is: in, is the corresponding loss weight coefficient, Stylized loss The corresponding loss weight coefficient is, is the stylized loss, Keep loss for content The corresponding loss weight coefficient is, Preserve loss for geometry The corresponding loss weight coefficient is, is the geometry preservation loss, Consistency loss The corresponding loss weight coefficient is, is the consistency loss; A construction module is specifically used to input the content features of the image to be edited into the first image stylization editing network as the first denoising information, and add a low-rank adaptation function in the first image stylization editing network, and use the image diffusion loss to optimize the first image stylization editing network with the added low-rank adaptation function; input the style features of the reference style image into the second image stylization editing network as the second denoising information, and add a low-rank adaptation function in the second image stylization editing network, and use the image diffusion loss to optimize the second image stylization editing network with the added low-rank adaptation function; according to the optimized first image stylization editing network and the optimized second image stylization editing network, among the 11 converter modules, the 5th and 6th converter modules are replaced by the 5th and 6th converter modules in the second image stylization editing network; the 1st to 4th and the 7th to 11th converter modules are replaced by the corresponding modules in the first image stylization editing network to generate the third image stylization editing network, wherein the low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

12. An electronic device, characterized in that: comprising a memory and a processor, wherein the memory and the processor are connected; The memory stores computer instructions, and the processor executes the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 10 by executing the computer instructions.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 10.

14. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 10.

Images