Image stylized editing method and device for three-dimensional scene, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to an image stylized editing method and device for a three-dimensional scene, equipment and a medium. The method comprises the following steps: acquiring a source three-dimensional model, a reference style image, a first image stylization editing network and a second image stylization editing network; obtaining content features of the to-be-edited image and style features of the reference style image; respectively optimizing two image stylization editing networks added with a low-rank adaptation function by using image diffusion loss to construct a third image stylization editing network, inputting the to-be-edited image and the reference style image into the third image stylization editing network for processing, and outputting a single-view image after stylization editing; and iteratively updating the stylized and edited single-view image by using an optimization strategy based on 3D content and style combination to obtain a stylized and edited 3D model. According to the method, the stylization effect of the reference style image is improved, and the fidelity and multi-view consistency of the 3D content are ensured on the premise of consistent styles.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, equipment and medium for stylized editing of images of three-dimensional scenes. Background Art

[0002] 3D model stylization is an important way to edit the appearance of 3D models. It can achieve consistent 3D stylization given a single reference style image, that is, a consistent style transfer from a 2D style image of any style to a 3D scene. The purpose of 3D scene style transfer is to transfer the style of the reference style image to the 3D scene, while taking into account style consistency, content fidelity, and multi-view consistency.

[0003] Currently, image editing is promoted to three-dimensional space using iterative dataset update strategies, thereby enabling editing of 3D models such as color and texture. These edits are applied to the dataset images of the original generated scene. Recently, an efficient feature rendering strategy was proposed that can quickly render high-dimensional VGG (Visual Geometry Group) features. However, this method requires learning style images in advance and integrating them into three-dimensional Gaussian features. Although this method achieves certain 3D stylized editing effects, it still has problems such as leakage of reference style image content and excessive editing of 3D scene geometry (such as deformation). Summary of the invention

[0004] In view of this, the present invention provides a method, device, equipment and medium for stylized editing of images of three-dimensional scenes to improve the 3D stylization effect based on a reference style image, and ensure the fidelity and multi-perspective consistency of 3D content while ensuring consistency of style.

[0005] In a first aspect, the present invention provides a method for stylizing an image of a three-dimensional scene, the method comprising: Acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network; Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image; The content features and the style features are used as denoising information, and the first image stylization editing network and the second image stylization editing network to which a low-rank adaptation function is added are optimized respectively by using image diffusion loss, and a third image stylization editing network is constructed, and a low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network; Inputting the image to be edited and the reference style image into the third image stylization editing network for processing, and outputting a single-view image after stylization editing; The stylized edited single-view image is iteratively updated using an optimization strategy based on the combination of 3D content and style to obtain a stylized edited 3D model.

[0006] In combination with the first aspect, in a possible implementation manner, decoupling to obtain style features of the reference style image includes: Parsing the reference style image using an image title generation method to obtain a reference text of the reference style image; Extracting text features of the reference text using a text encoder in a pre-trained model, and using the text features as content features of the reference style image; Extracting image features of the reference style image using an image encoder in a pre-trained model; The image features and the content features are calculated using the pre-trained model to obtain style features of the reference style image.

[0007] The image stylization editing method of a three-dimensional scene provided by the present invention first uses the first image stylization editing network to learn the content features of the image to be edited, and then uses the second image stylization editing network to learn the style features of the reference style image; uses the image diffusion loss to respectively optimize the two image stylization editing networks that add low-rank adaptation functions, and constructs a third image stylization editing network. Since the cross-attention layer of the third image stylization editing network adds a low-rank adaptation function, such as a LoRA layer, and is decoupled from the content and style features of the reference style image, the third image stylization editing network is used to process the image to be edited and the reference style image, and a single-view image that conforms to the style of the reference style image is output. Finally, based on the joint optimization of the two angles of 3D content and style, the above-mentioned single-view image after stylization editing is iteratively updated to obtain a 3D model after stylization editing, thereby improving the 3D stylization effect based on the reference style image, and ensuring the fidelity and multi-view consistency of the 3D content under the premise of ensuring consistent style.

[0008] In combination with the first aspect, in another possible implementation, if there are multiple reference style images and they belong to the same style, a text encoder in a pre-trained model is used to extract text features of the reference text, and the text features are used as content features of the reference style image, including: Use a large language model to synthesize multiple reference texts into a text description; Extracting total text features of the text description using a text encoder in the pre-trained model, and using the total text features as content features of multiple reference style images; Use the image encoder in the pre-trained model to extract image features of the reference style image, including: The image encoder is used to respectively extract image features of a plurality of reference style images; An average of the image features of the multiple reference style images is taken to obtain the image feature.

[0009] In combination with the first aspect, in yet another possible implementation, before using the first image stylization editing network to learn the content features of the to-be-edited image, the method further includes: Get the basic diffusion model; Adding low-rank adaptation function training to optimize the basic diffusion model to generate the first image stylization editing network; Among them, the first image stylization editing network includes 11 converter modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0010] In combination with the first aspect, in another possible implementation, the content feature and the style feature are used as denoising information, and the first image stylization editing network and the second image stylization editing network with a low-rank adaptation function are optimized by using image diffusion loss to construct a third image stylization editing network, including: Inputting the content features of the image to be edited into the first image stylization editing network as first denoising information, adding a low-rank adaptation function in the first image stylization editing network, and optimizing the first image stylization editing network with the low-rank adaptation function by using the image diffusion loss; Inputting the style features of the reference style image into the second image stylization editing network as second denoising information, adding a low-rank adaptation function in the second image stylization editing network, and optimizing the second image stylization editing network with the added low-rank adaptation function by using the image diffusion loss; Based on the two optimized image stylization editing networks, a third image stylization editing network is constructed.

[0011] In combination with the first aspect, in another possible implementation, constructing a third image stylization editing network according to the two optimized image stylization editing networks includes: According to the optimized first image stylization editing network and the optimized second image stylization editing network, among the 11 converter modules, the 5th and 6th converter modules are replaced by the 5th and 6th converter modules in the second image stylization editing network; the 1st to 4th and the 7th to 11th converter modules are replaced by the corresponding modules in the first image stylization editing network to generate the third image stylization editing network.

[0012] Among them, a low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

[0013] In combination with the first aspect, in yet another possible implementation, the method further includes: In the third image stylization editing network, inputting the content features of the image to be edited into all converter modules in the third image stylization editing network as content denoising guidance information; The style features of the reference style image are input into the fifth and sixth converter modules of the third image stylization editing network as style denoising guidance information.

[0014] In combination with the first aspect, in another possible implementation, the iterative updating of the stylized edited single-view image to obtain the stylized edited 3D model by using an optimization strategy based on a combination of 3D content and style includes: Constructing a multi-loss function, wherein the multi-loss function includes: stylization loss, content preservation loss, geometry preservation loss and consistency loss; Obtaining a loss weight coefficient corresponding to each loss function, and calculating multiple loss parameters according to the loss weight coefficient and the multiple loss functions; The stylized edited single-view image is iteratively updated according to the multi-loss parameter constraints by utilizing an optimization strategy based on the combination of 3D content and style to obtain the stylized edited 3D model.

[0015] In combination with the first aspect, in yet another possible implementation, constructing a stylized loss in a multi-loss function includes: Using the features extracted by the pre-trained convolutional neural network, the loss function between the rendered image and the style image under a set of training perspectives is calculated. The stylized loss is expressed as:

[0016] in, is the stylized loss, Pre-trained convolutional neural network features extracted for stylized 3D Gaussian model rendered images, The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

[0017] In combination with the first aspect, in yet another possible implementation, constructing the content preservation loss in the multi-loss function includes: The content retention loss is calculated using a content retention loss relational expression, where the content retention loss relational expression is:

[0018] in, To maintain loss for content, The features extracted by the pre-trained convolutional neural network are used to render the 3D Gaussian model for stylized editing. The source 3D Gaussian model rendering image is extracted from the features of the pre-trained convolutional neural network. H is the length of the rendered image and W is the width of the rendered image.

[0019] In combination with the first aspect, in yet another possible implementation, constructing a geometry preservation loss in the multi-loss function includes: Obtaining a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model; The geometry preservation loss is calculated according to the depth image obtained by rendering the source three-dimensional model, the depth image obtained by rendering the edited three-dimensional model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is:

[0020] in, is the geometry preservation loss, The depth image rendered from the source 3D model. The depth image obtained by rendering the edited 3D model, the depth image of the source 3D model after rendering The depth of each pixel is Calculated by the following relationship:

[0021]

[0022] in, is the 2D Gaussian after the i-th 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the two-dimensional Gaussian after the i-th projection of the source three-dimensional model.

[0023] In combination with the first aspect, in another possible implementation, constructing a consistency loss in a multi-loss function includes: Use the pre-trained image caption generation model to describe the content and style of the edited 3D model rendering image, and obtain content description text and style description text respectively; Encoding the content description text and the style description text by using a text encoder to obtain content description text features and style description text features; According to the content description text features and the style description text features, the consistency loss is calculated using a semantic consistency loss relationship, where the semantic consistency loss relationship is:

[0024] in, is the consistency loss, The features extracted from the image after the source 3D model is rendered by the image editor. Characterize the text for style description, is the style feature of the reference style image.

[0025] In a second aspect, the present invention provides a device for stylizing an image of a three-dimensional scene, the device comprising: An acquisition module, used to acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network; A processing module, configured to render the source 3D model to obtain a single-view image as the image to be edited, and to decouple content features of the image to be edited and style features of the reference style image; A construction module is used to use the content features and the style features as denoising information, and use image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network to which the low-rank adaptation function is added, and to construct a third image stylization editing network, wherein the low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network; The processing module is further used to input the image to be edited and the reference style image into the third image stylization editing network for processing, and output a single-view image after stylization editing; The editing module is used to iteratively update the stylized edited single-view image by using an optimization strategy based on the combination of 3D content and style to obtain a stylized edited 3D model.

[0026] In a third aspect, the present invention provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the image stylization editing method for three-dimensional scenes of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the image stylization editing method for a three-dimensional scene of the first aspect or any corresponding embodiment thereof.

[0028] In addition, the present invention provides a computer program product, including computer instructions, which are used to enable a computer to execute the image stylization editing method of a three-dimensional scene of the above-mentioned first aspect or any corresponding embodiment.

[0029] The present invention provides a method, device, equipment and medium for stylized editing of images of three-dimensional scenes. The first image stylized editing network is used to learn the content features of the image to be edited, and then the second image stylized editing network is used to learn the style features of the reference style image; the two image stylized editing networks with low-rank adaptation functions are optimized by using image diffusion loss to construct a third image stylized editing network. Since the cross-attention layer of the third image stylized editing network adds a low-rank adaptation function, such as a LoRA layer, and is decoupled from the content and style features of the reference style image, the third image stylized editing network is used to process the image to be edited and the reference style image, and a single-view image that conforms to the style of the reference style image is output. Finally, based on the joint optimization of the two angles of 3D content and style, the above-mentioned single-view image after stylized editing is iteratively updated to obtain a 3D model after stylized editing, thereby improving the 3D stylization effect based on the reference style image, and ensuring the fidelity and multi-view consistency of the 3D content under the premise of ensuring the consistency of style.

[0030] In addition, the present invention also proposes a method for stylized image editing with decoupled content and style, decouples the content and style features of a reference style image, obtains the content and style features of the reference style image, and uses them as denoising conditions to guide stylized image editing. In addition, after the decoupling process, the content and style of the reference style image are decoupled in the alignment space, effectively preventing the leakage of the reference style image content during the stylization process.

[0031] This method can also reduce the training cost, accelerate the training process, and obtain a trained image stylization network. Using an optimization strategy based on the combination of 3D content and style, the stylized single-view image is iteratively updated. In this process, multiple loss functions are added, such as stylization loss, content preservation loss, geometry preservation loss, and consistency loss, to further ensure that the content structure of the source 3D model remains unchanged, to keep the 3D geometry from being excessively deformed during the stylization of the 3D Gaussian model, and to improve the semantic consistency of 3D stylization. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a schematic diagram of a three-dimensional stylized editing process based on content and style decoupling according to an embodiment of the present invention; Figure 2 is a schematic flow chart of a method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention; Figure 3 is a schematic flow chart of another method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention; Figure 4 is a flowchart of image stylization editing based on content and style decoupling according to an embodiment of the present invention; Figure 5 is a schematic flow chart of another method for stylizing an image of a three-dimensional scene according to an embodiment of the present invention; Figure 6 is a schematic diagram of a Web-based three-dimensional stylized editing platform interface according to an embodiment of the present invention; Figure 7 is a structural block diagram of an image stylization editing device according to an embodiment of the present invention; Figure 8 It is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0036] The technical features involved in different embodiments of the present invention described in the present invention can be combined with each other as long as they do not conflict with each other.

[0037] In order to solve the problems of reference style image content leakage and over-editing of 3D scene geometry, an embodiment of the present invention proposes a three-dimensional stylized editing method. The method improves the 3D stylization effect based on the reference style image from two perspectives: decoupling the reference style image content and style features, and jointly optimizing the 3D content and style. While ensuring the consistency of style, the fidelity and multi-view consistency of the 3D content are guaranteed.

[0038] Specifically, this embodiment provides a method for stylizing an image of a three-dimensional scene based on decoupling content and style. Figure 1 As shown, it mainly includes: first, given the source 3D model to be edited and the reference style image; then, the source 3D model is rendered to obtain a single-view image, and for this rendered single-view image, the proposed image stylization editing model based on content and style decoupling is used to complete the image editing that conforms to the style of the reference style image. The rendered single-view image is processed using the image stylization editing model or network to obtain a single-view image of the target style. Finally, using a strategy based on the optimization of the joint 3D content and style, the set of stylized edited images is iteratively updated, and the image stylization editing results are promoted to 3D scenes, thereby realizing 3D stylized editing.

[0039] The following is a detailed description of the image stylization editing method for a three-dimensional scene provided by this embodiment.

[0040] An embodiment of the present invention provides an embodiment of a method for stylizing an image of a three-dimensional scene. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0041] In this embodiment, a method for stylizing an image of a three-dimensional scene is provided, which can be used in the above-mentioned server, client or terminal device, such as a mobile phone, tablet computer, etc. Figure 2 is a flow chart of a method according to an embodiment of the present invention, which comprises: Step S101: obtaining a source 3D model and a reference style image, and obtaining a first image stylization editing network and a second image stylization editing network.

[0042] The source 3D model refers to a model with 3D data constructed by using 3D production software in a virtual 3D space. These models are usually used in computer and video games, as well as special effects production for movies and TV programs, and even in the fields of engineering design, architectural design, and product manufacturing. In this embodiment, the source 3D model refers to a 3D environment image taken in a real scene. The reference style image is a given image.

[0043] Optionally, the reference style image, i.e. the image to be edited, is represented as The number of reference style images may be one or more, and the number of images to be edited may also be one or more.

[0044] The first image stylization editing network and the second image stylization editing network have the same structure and may include multiple transformer modules for editing the style of an image. The first image stylization editing network is used to learn the content features of the image to be edited, and the second image stylization editing network is used to learn the style features of the reference style image.

[0045] Step S102: Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image.

[0046] Specifically, the image to be edited is decoupled using a decoupling method to obtain content features of the image to be edited. The reference style image is decoupled using a decoupling method to obtain content features and style features corresponding to the reference style image.

[0047] In this embodiment, the content and style features of the reference style image are decoupled, and the content and style features of the reference style image are obtained as denoising conditions to guide image stylized editing. In addition, the content and style of the reference style image are decoupled in the alignment space to prevent the leakage of the reference style image content during the stylization process.

[0048] Step S103: Content features and style features are used as denoising information, and image diffusion loss is used to optimize the first image stylization editing network and the second image stylization editing network with low-rank adaptation function, respectively, to construct the third image stylization editing network.

[0049] First, low-rank adaptation functions, such as LoRA layers, are added to the Transformer modules in the first image stylization editing network and the second image stylization editing network, respectively; then, the first image stylization editing network and the second image stylization editing network with LoRA layers added are optimized and trained using image diffusion loss to generate a third image stylization editing network. A low-rank adaptation function (LoRA layer) is also added to the cross-attention layer in the third image stylization editing network.

[0050] LoRA layer: Low-Rank Adaptation, by decomposing the weight matrix into the product of low-rank matrices, reduces the number of parameters, thereby achieving the purpose of reducing hardware resources and accelerating the fine-tuning process.

[0051] Step S104: input the image to be edited and the reference style image into a third image stylization editing network for processing, and output a single-view image after stylization editing.

[0052] Step S105: using an optimization strategy based on the combination of 3D content and style, iteratively updating the stylized edited single-view image to obtain a stylized edited 3D model.

[0053] This step utilizes a strategy based on the joint optimization of 3D content and style to enhance the stylized editing results of the single-view image to a 3D scene, thereby achieving 3D stylized editing.

[0054] The image stylization editing method of a three-dimensional scene provided by the present invention first uses the first image stylization editing network to learn the content features of the image to be edited, and then uses the second image stylization editing network to learn the style features of the reference style image; uses the image diffusion loss to respectively optimize the two image stylization editing networks that add low-rank adaptation functions, and constructs a third image stylization editing network. Since the cross-attention layer of the third image stylization editing network adds a low-rank adaptation function, such as a LoRA layer, and is decoupled from the content and style features of the reference style image, the third image stylization editing network is used to process the image to be edited and the reference style image, and a single-view image that conforms to the style of the reference style image is output. Finally, based on the joint optimization of the two angles of 3D content and style, the above-mentioned single-view image after stylization editing is iteratively updated to obtain a 3D model after stylization editing, thereby improving the 3D stylization effect based on the reference style image, and ensuring the fidelity and multi-view consistency of the 3D content under the premise of ensuring consistent style.

[0055] Specifically, in a possible implementation of this embodiment, as Figure 3 As shown, in step S102, decoupling to obtain style features of the reference style image includes: Step S301: parse the reference style image using the image title generation method to obtain the reference text of the reference style image. The reference style image can be represented as “ ”.

[0056] Step S302: Extract using the text encoder in the pre-trained model , and use text features as content features of the reference style image.

[0057] Step S303: extracting image features of the reference style image using the image encoder in the pre-trained model.

[0058] Step S304: Calculate the image features and content features using the pre-trained model to obtain style features of the reference style image.

[0059] Specifically, input a single reference style image , use image caption generation methods, such as BLIP (Bootstrapping Language-Image Pre-training, a pre-training model for unified visual language understanding and generation) to describe the content of the reference style image and obtain the reference style text , extract reference style text using the text encoder in the pre-trained CLIP model The text features of the reference style image are used as the content features of the reference style image. , which is the reference text in the above step S301.

[0060] Among them, the BLIP model aims to unify the understanding and generation tasks of vision and language through pre-training. It can extract the main content of the image and generate descriptive titles or answer questions related to the image based on these contents. In addition, the BLIP model also shows strong generalization ability and can be applied to a variety of visual language tasks, such as image text retrieval, image description generation, and visual question answering.

[0061] CLIP (Contrastive Language-Image Pre-Training) can map images and text into a shared vector space, which can be used for various vision and language tasks. In this embodiment, the image encoder of CLIP is used to extract the reference style image. Image features , through the pre-trained CLIP model, and Transformed to the same embedding space, addition and subtraction can be performed to obtain the reference style image Style characteristics Among them, the reference style image Style characteristics .

[0062] In this embodiment, the reference style image is displayed Content Features and style features Decoupling is performed to prevent leakage of reference style image content during the stylization process.

[0063] Furthermore, in another embodiment, if there are multiple reference style images that belong to the same style, the above step S302 includes: using a large language model to synthesize multiple reference texts into a text description; using a text encoder in a pre-trained model to extract the total text features of the text description, and using the total text features as content features of multiple reference style images; step S303 includes: using an image encoder to respectively extract image features of multiple reference style images; and taking the average of the image features of multiple reference style images to obtain image features.

[0064] Specifically, for multiple reference style images of the same style (for example, 10 reference style images), ), firstly, the image title generation method BLIP is used to process multiple reference style images to obtain the corresponding image content description text (i.e., reference text), and then the language large model is used to synthesize multiple description texts into a text description to facilitate the extraction of text features, and the text editor in the pre-trained CLIP model is used to extract the above merged text description to obtain the content features of multiple reference style images. .

[0065] At the same time, the image encoder in CLIP is used to obtain the image features of multiple reference style images respectively. , and then take the mean to obtain the image features of the reference style image , or , so the style feature is calculated as .

[0066] The following embodiments are described based on the input of a single reference style image.

[0067] In another embodiment of the present application, the above step S102: obtaining a first image stylization editing network includes: Get the basic diffusion model, add the low-rank adaptation function (LoRA layer) to train and optimize the basic diffusion model, and generate the first image stylization editing network. For example, the symbolic representation of the first image stylization editing network is .

[0068] Among them, the first image stylization editing network The paper includes 11 transformer modules, each of which includes a self-attention layer and a cross-attention layer. The cross-attention layer is used to use text or image feature cues as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0069] Optionally, the base diffusion model can be an SDXL model. Furthermore, SDXL (Stable DiffusionXL) is a generative pre-trained model based on the diffusion model, especially for text-to-image generation tasks. Diffusion process: In SDXL, the diffusion process involves gradually adding noise to the original data so that it gradually approaches a random distribution. This process is one of the key steps for the model to learn the data distribution. Reverse process: The reverse process starts from a random distribution, gradually removes noise, and finally generates high-quality data samples. This process can be regarded as a denoising process. Through continuous iteration and optimization, the model can generate data samples that are closer and closer to the real data.

[0070] This embodiment first constructs a first image stylization editing network and trains the network. The goal is to input a single reference style image The content and style of the image are further decoupled by utilizing separate modules in the first image stylization editing network to process the content and style, so as to mitigate the content leakage of the reference style image and improve the generalization of the stylization network to different stylizations.

[0071] like Figure 4 As shown, a method for constructing an image stylization editing network based on content and style decoupling provided in this embodiment utilizes a pre-trained SDXL text-to-image generation model and uses LoRA to train and optimize the content and style transformation modules in the SDXL model.

[0072] like Figure 4 As shown, the first image stylization editing network Based on the pre-trained SDXL diffusion model, the first image stylization editing network It contains 11 Transformer modules, of which 4 are downsampling modules, 1 is an intermediate module, and 6 are upsampling modules. Each Transformer module contains a self-attention layer and a cross-attention layer, where the cross-attention layer is used to use text or image feature cues as guidance information to guide image generation. Optimizing all parameters of SDXL from scratch is resource-intensive and difficult to train, so it is proposed to add a LoRA layer and only optimize the cross-attention layer in the Transformer.

[0073] In addition, in the method provided in this embodiment, the first image stylization editing network is used Learning the image to be edited ( ) and using a second image stylization editing network Learning from reference style images The style features include: Get the image to be edited Content Features ; The image to be edited Content Features Input first image stylization editing network , after being processed by 11 converter modules, the image to be edited is obtained ; The reference style image Style characteristics Input the second image stylization editing network and generate a reference style image after processing .

[0074] Specifically, in order to reduce the training cost and speed up the training process, a trained image stylization network is obtained. Given a reference style image and the image to be edited , using the above step S102, the reference style image Decouple the content and style features of the image to obtain the decoupled image content and style features and Then, the image caption generation model BLIP is used to generate a given image to be edited. Generate a text description of the image and encode the text description using the text encoder in the pre-trained CLIP to obtain the content features of the image to be edited .

[0075] Specifically, a possible implementation of the above step S103 is: Input to the first stylized editing network All Transformer modules in the image are used as the first denoising information, and LoRA layers are added to the cross-attention layers of all Transformer modules to optimize the first image stylization editing network using image diffusion loss. The LoRA layer is used to optimize the first image stylization editing network after training. , the network can be used to generate images to be edited .

[0076] Similarly, the above reference style image Image style features Input to the second stylized editing network All Transformer modules in the image are used as the second denoising information, and LoRA layers are added to the cross-attention layers of all Transformer modules to optimize the second image stylization editing network using image diffusion loss. , and obtain the second image stylized editing network after optimization training , the network can be used to generate a reference style image .

[0077] Finally, based on the two optimized image stylization editing networks, a third image stylization editing network is constructed ( ).

[0078] In the above step S103, a third image stylization editing network is constructed ( ) is to replace the 5th and 6th transformer modules with the second image stylization editing network in the 11 transformer modules. The 5th and 6th converter modules in the image stylization network are replaced by the first image stylization network. The corresponding module in generates the third image stylization editing network .

[0079] Also includes: Input to the third image stylization editing network All Transformer modules in , as content denoising guidance information, will Input to the third image stylization editing network The 5th and 6th Transformer modules are used as style denoising guidance information. Add LoRA layers to the cross-attention layers of the 5th and 6th Transformer modules, and use image diffusion loss to optimize the stylized editing network Generate stylized images .

[0080] In the above step S104, during inference, the image to be edited is input and reference style images , stylized editing network , you can output the stylized image .

[0081] Furthermore, in another embodiment, Figure 5 As shown, the above step S105: using the optimization strategy based on the combination of 3D content and style, iteratively updating the stylized single-view image to obtain the stylized 3D model, specifically includes: Step S1051: constructing a multi-loss function, wherein the multi-loss function includes at least one of a stylization loss, a content preservation loss, a geometry preservation loss, and a consistency loss.

[0082] Step S1052: Obtain the loss weight coefficient corresponding to each loss function, and calculate multiple loss parameters based on the loss weight coefficient and multiple loss functions.

[0083] One implementation method is to calculate the multi-loss function using the following formula:

[0084] in, is the corresponding loss weight coefficient, The stylized loss The corresponding loss weight coefficient is, is the stylized loss, Keep loss for content The corresponding loss weight coefficient is, Keep loss for geometry The corresponding loss weight coefficient is, is the geometry preservation loss, Consistency loss The corresponding loss weight coefficient is, is the consistency loss.

[0085] Among them, the loss weight coefficient corresponding to each loss can be customized by the system.

[0086] Step S1053: Using the optimization strategy based on the combination of 3D content and style, iteratively update the stylized single-view image according to the multi-loss parameter constraint , and obtain the stylized edited 3D model.

[0087] This embodiment constrains the content fidelity and style consistency of the stylized 3D editing model through the above-mentioned multi-loss function, so that the 3D model is stylized according to the reference style image while reducing the substantial damage to the geometric content.

[0088] The following is a detailed description of the process of determining each loss function in each multi-loss function.

[0089] First, given the source 3D model, it is represented by a 3D Gaussian. Given the multi-view images of the source 3D scene, the source 3D model is obtained by reconstruction and represented by a 3D Gaussian. ,in , , Indicates the location of the center point, Represents the covariance matrix, and the color of each point is determined by Indicates that transparency , then the three-dimensional Gaussian expression is:

[0090] in, Represents the query point and the center point The distance represents the covariance matrix, -1 represents the inverse transformation, represents the i-th three-dimensional Gaussian point, T represents the transposed matrix, Then, based on the rendered single-view image of the source 3D model The color of a pixel is:

[0091]

[0092] The goal of this embodiment is to perform 3D stylization on the source 3D model so that the Gaussian representation of the stylized 3D model Capturing the style of a reference style image while maintaining its original content structure .

[0093] Using the strategy of joint optimization of 3D content and style, the rendered image is iteratively stylized and 3D stylized editing is achieved using multi-loss optimization.

[0094] Specifically, one implementation method is to use a three-dimensional Gaussian to represent the edited three-dimensional model. ,in , , Indicates the location of the center point, Represents the covariance matrix, and the color of each point is determined by Indicates that transparency , then the three-dimensional Gaussian representation of the edited three-dimensional model is:

[0095] in, Represents the query point and the center point distance.

[0096] A single-view image of the 3D model rendered after editing The color of a pixel is:

[0097]

[0098] This three-dimensional Gaussian model Source 3D Gaussian Model Perform optimization and then perform optimization training.

[0099] After editing, the three-dimensional Gaussian model Render along a certain perspective to get the current rendered image, according to the input reference style image , using the third image stylization editing model Edit and obtain a single-view stylized edited image that matches the style of the reference style image .

[0100] Iteratively render the rendered image under the new perspective, repeat the above editing process, update the edited image data, and directly back-propagate the edited image to the three-dimensional Gaussian scene so that it is continuously updated according to the input reference style image, thereby generating an edited three-dimensional Gaussian scene that conforms to the given style.

[0101] Specifically, in step S1051, the stylized loss in the multi-loss function is constructed. ,include: Using the features extracted by a pre-trained convolutional neural network (such as VGG), the loss function between the rendered image and the style image under a set of training perspectives is calculated. In order to transfer detailed high-frequency style features from the 2D style image to the 3D scene, this embodiment minimizes the cosine distance between the features of the rendered image and the nearest neighbors in its style image features. The stylized loss is expressed as:

[0102] in, is the stylized loss, Pre-training convolutional neural network to extract features for stylized 3D Gaussian model rendering images, The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

[0103] Optionally, in another embodiment, in order to further ensure that the content structure of the source 3D model remains unchanged, a content preservation loss is proposed. , the above step S1051: constructing the content preservation loss in the multi-loss function , including: calculating the content preservation loss using a content preservation loss relation, the content preservation loss relation is:

[0104] in, To maintain loss for content, The features extracted by the pre-trained convolutional neural network are used to render the 3D Gaussian model after stylization. The source 3D Gaussian model rendering image is extracted from the features of the pre-trained convolutional neural network. H is the length of the rendered image and W is the width of the rendered image.

[0105] Optionally, in another embodiment, since the geometric parameters of the 3D Gaussian model still cause unexpected geometric transformation of the scene during the optimization process, in order to further prevent the 3D geometry from being excessively deformed during the stylization process of the 3D Gaussian model, the above step S1051: constructing the geometry preservation loss in the multi-loss function ,include: Obtain a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model; calculate the geometry preservation loss according to the depth image obtained by rendering the source 3D model, the depth image obtained by rendering the edited 3D model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is:

[0106] in, is the geometry preservation loss, The depth image rendered from the source 3D model. The depth image obtained by rendering the edited 3D model, d is the depth of each pixel. The 3D Gaussian fusion algorithm can be an alpha-blending method, which is a computer graphics technique used to blend two or more images or rendering results together according to their alpha values ​​(usually indicating transparency). This method is often used to achieve transparency effects, such as rendering translucent objects such as glass, smoke or flames.

[0107] Depth image of the source 3D model after rendering The depth of each pixel is Calculated by the following relationship:

[0108]

[0109] in, is the 2D Gaussian after the i-th 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the two-dimensional Gaussian after the i-th projection of the source three-dimensional model.

[0110] Optionally, in another implementation, in order to further improve the semantic consistency of 3D stylization, this embodiment also proposes a semantic consistency loss The above step S1051: constructing the consistency loss in the multi-loss function includes: The pre-trained image caption generation model BLIP is used to describe the content and style of the edited 3D model rendering image, and the content description text and style description text are obtained respectively; the content description text and style description text are encoded using a text encoder to obtain content description text features and style description text features; according to the content description text features and style description text features, the semantic consistency loss relationship is used to calculate the consistency loss, and the semantic consistency loss relationship is:

[0111] in, is the consistency loss, The features extracted from the image after the source 3D model is rendered by the image editor. Characterize the text for style description, is the style feature of the reference style image.

[0112] The method provided in this embodiment also includes: using an interactive interface to achieve user-friendly 3D stylized editing.

[0113] Specifically, a Web-based 3D model editing platform is constructed to provide an operable UI interface to assist users in completing editing. To assist users in completing editing, this embodiment also constructs a Web-based 3D model editing platform. The interface diagram is shown in FIG. Figure 6 Click "Please enter the source 3D model" and a 3D model input box will pop up, which supports the input of a 3D Gaussian model; click "Please enter the reference style image" and an image input box will pop up to input a reference style image; click "Start stylized editing" and the background will automatically perform 3D model stylized editing training, and a prompt will pop up at each stage of training completion; finally, the stylized 3D model will be output.

[0114] The method proposed in this embodiment utilizes the strategy of joint optimization of 3D content and style to iteratively perform stylized editing on the rendered image, and realizes 3D stylized editing by utilizing multi-loss optimization. In addition, an interactive interface is proposed to provide an operable UI interface to realize user-friendly 3D stylized editing.

[0115] The present invention provides a method for stylizing an image of a three-dimensional scene. The method first uses a first image stylizing editing network to learn the content features of an image to be edited, and then uses a second image stylizing editing network to learn the style features of a reference style image. The two trained image stylizing editing networks are then fused to form a third image stylizing editing network. Since a low-rank adaptation function, such as a LoRA layer, is added to the cross-attention layer of the third image stylizing editing network, and the third image stylizing editing network is decoupled from the content and style features of the reference style image, the third image stylizing editing network is used to process the image to be edited and the reference style image, and a single-view image that conforms to the style of the reference style image is output. Finally, based on the joint optimization of the two angles of 3D content and style, the above-mentioned single-view image after stylizing editing is iteratively updated to obtain a 3D model after stylizing editing, thereby improving the 3D stylization effect based on the reference style image, and ensuring the fidelity and multi-view consistency of the 3D content while ensuring the consistency of style.

[0116] In addition, the present invention also proposes a method for stylized image editing with decoupled content and style, decouples the content and style features of a reference style image, obtains the content and style features of the reference style image, and uses them as denoising conditions to guide stylized image editing. In addition, after the decoupling process, the content and style of the reference style image are decoupled in the alignment space, effectively preventing the leakage of the reference style image content during the stylization process.

[0117] This method can also reduce the training cost, accelerate the training process, and obtain a trained image stylization network. Using an optimization strategy based on the combination of 3D content and style, the stylized single-view image is iteratively updated. In this process, multiple loss functions are added, such as stylization loss, content preservation loss, geometry preservation loss, and consistency loss, to further ensure that the content structure of the source 3D model remains unchanged, to keep the 3D geometry from being excessively deformed during the stylization of the 3D Gaussian model, and to improve the semantic consistency of 3D stylization.

[0118] It should be noted that the method proposed in the present invention can be used to efficiently edit existing 3D assets, obtain more diverse 3D assets, and save the modeling time and economic cost of 3D assets. It can be used to support the product development of Metaverse servers such as MetaEngine and AI servers and their applications in the field of 3D content generation and editing.

[0119] In this embodiment, a device for stylizing an image of a three-dimensional scene is also provided, and the device is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0120] This embodiment provides a three-dimensional scene image stylization editing device, such as Figure 7 As shown, the device includes: an acquisition module 710, a processing module 720, a construction module 730 and an editing module 740. In addition, the device may also include other more or fewer units or modules, which is not limited in this embodiment.

[0121] The acquisition module 710 is used to acquire a source 3D model and a reference style image, and to acquire a first image stylization editing network and a second image stylization editing network.

[0122] The processing module 720 is used to render the source 3D model to obtain a single-view image as the image to be edited, and to decouple the content features of the image to be edited and the style features of the reference style image.

[0123] A construction module 730 is used to use content features and style features as denoising information, and use image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network that add low-rank adaptation functions, construct a third image stylization editing network, and add a low-rank adaptation function to the cross-attention layer in the third image stylization editing network.

[0124] The processing module 720 is further used to input the image to be edited and the reference style image into a third image stylization editing network for processing, and output a single-view image after stylization editing.

[0125] The editing module 740 is used to iteratively update the stylized edited single-view image to obtain a stylized edited 3D model by using an optimization strategy based on the combination of 3D content and style.

[0126] In some optional implementations, the processing module 720 is specifically configured to generate a title for the reference style image ( ) to parse and obtain the reference text of the reference style image ( ); Extract reference text using the text encoder in the pre-trained model ( ) and use the text features as the content features of the reference style image ( ); Use the image encoder in the pre-trained model to extract the reference style image ( ) image features ( ); Use the pre-trained model to transform image features ( ) and content features ( ) is transformed into the same embedding space to obtain the style features of the reference style image ( ).

[0127] Optionally, in some optional implementations, the processing module 720 is further configured to, if there are multiple reference style images and they are of the same style, use the language model to synthesize the multiple reference texts into a text description; use the text encoder in the pre-trained model to extract the total text features of the text description, and use the total text features as the content features of the multiple reference style images ( ); Use the image encoder to extract the image features of multiple reference style images respectively; take the average of the image features of multiple reference style images to obtain the image features ( ).

[0128] Optionally, in some other optional implementations, the construction module 730 is further used to obtain a basic diffusion model, add a low-rank adaptation function (LoRA layer) to train and optimize the basic diffusion model, and generate a first image stylization editing network.

[0129] Among them, the first image stylization editing network includes 11 transformer modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

[0130] Optionally, in another optional embodiment, module 730 is constructed, which is specifically used to input content features of the image to be edited into the first image stylization editing network as first denoising information, and add a low-rank adaptation function in the first image stylization editing network, and optimize the first image stylization editing network with the low-rank adaptation function using image diffusion loss; input style features of the reference style image into the second image stylization editing network as second denoising information, and add a low-rank adaptation function in the second image stylization editing network, and optimize the second image stylization editing network with the low-rank adaptation function using image diffusion loss; and construct a third image stylization editing network based on the two optimized image stylization editing networks.

[0131] Optionally, in some other optional embodiments, the construction module 730 is also used to replace the 5th and 6th converter modules among the 11 converter modules with the 5th and 6th converter modules in the second image stylization editing network according to the first image stylization editing network and the second image stylization editing network after feature learning; and replace the 1st to 4th and the 7th to 11th converter modules with the corresponding modules in the first image stylization editing network to generate a third image stylization editing network.

[0132] Among them, a low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

[0133] In addition, construction module 730 is also used to input the content features of the image to be edited into all converter modules in the third image stylization editing network as the second denoising information; and input the style features of the reference style image into the 5th and 6th converter modules of the third image stylization editing network as the second denoising information.

[0134] Optionally, in some other optional implementations, the editing module 740 is specifically used to construct multiple loss functions, obtain the loss weight coefficient corresponding to each loss function, and calculate multiple loss parameters according to the loss weight coefficient and the multiple loss functions; using an optimization strategy based on the combination of 3D content and style, according to the constraints of multiple loss parameters, iteratively update the single-view image after stylized editing ( ), and obtain the stylized edited 3D model.

[0135] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0136] The image stylization editing of the three-dimensional scene in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0137] The embodiment of the present invention also provides an electronic device having the above Figure 7 A stylized edit of the image of the 3D scene shown.

[0138] See also Figure 8 , is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface).

[0139] In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0140] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0141] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the image stylization editing method for the three-dimensional scene shown in the above embodiment.

[0142] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0143] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0144] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0145] The input device 30 can receive input digital or character information, and generate key signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device can be a touch screen.

[0146] The electronic device also includes at least one communication interface for the electronic device to communicate with other devices or a communication network.

[0147] An embodiment of the present invention also provides a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented by downloading through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware.

[0148] The storage medium may be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid state drive, etc.; further, the storage medium may also include a combination of the above-mentioned types of memories. It is understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor or hardware, the image stylization editing method of the three-dimensional scene shown in the above embodiment is implemented.

[0149] Embodiments of the present application may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the above method. Wherein, the computer program product may be written in any combination of one or more programming languages ​​to perform program codes for performing the operations of the disclosed embodiments, wherein the programming languages ​​include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server.

[0150] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention are described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for stylizing an image of a three-dimensional scene, characterized in that: The method comprises: Acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network; Rendering the source 3D model to obtain a single-view image as the image to be edited, and decoupling to obtain content features of the image to be edited and style features of the reference style image; The content features and the style features are used as denoising information, and the first image stylization editing network and the second image stylization editing network to which a low-rank adaptation function is added are optimized respectively by using image diffusion loss, and a third image stylization editing network is constructed, and a low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network; Inputting the image to be edited and the reference style image into the third image stylization editing network for processing, and outputting a single-view image after stylization editing; The stylized edited single-view image is iteratively updated using an optimization strategy based on the combination of 3D content and style to obtain a stylized edited 3D model.

2. The method according to claim 1, characterized in that: Decoupling to obtain the style features of the reference style image includes: Parsing the reference style image using an image title generation method to obtain a reference text of the reference style image; Extracting text features of the reference style image using a text encoder in a pre-trained model, and using the text features as content features of the reference style image; Extracting image features of the reference style image using an image encoder in a pre-trained model; The image features and the content features are calculated using the pre-trained model to obtain style features of the reference style image.

3. The method according to claim 2, characterized in that If there are multiple reference style images that are of the same style, a text encoder in a pre-trained model is used to extract text features of the reference style images, and the text features are used as content features of the reference style images, including: Use a large language model to synthesize multiple reference texts into a text description; Extracting total text features of the text description using a text encoder in the pre-trained model, and using the total text features as content features of multiple reference style images; The step of extracting image features of the reference style image by using an image encoder in the pre-trained model includes: Using the image encoder to extract image features of multiple reference style images respectively; An average of the image features of the multiple reference style images is taken to obtain the image feature.

4. The method according to claim 1, characterized in that The step of obtaining a first image stylization editing network includes: Get the basic diffusion model; Adding low-rank adaptation function training to optimize the basic diffusion model to generate the first image stylization editing network; Among them, the first image stylization editing network includes 11 converter modules, each of which includes a self-attention layer and a cross-attention layer; the cross-attention layer is used to use text or image feature prompts as guidance information to guide image generation, and the low-rank adaptation function is used to optimize the cross-attention layer.

5. The method according to claim 4, characterized in that The content feature and the style feature are used as denoising information, and the first image stylization editing network and the second image stylization editing network with low-rank adaptation function are optimized by using image diffusion loss to construct a third image stylization editing network, including: Inputting the content features of the image to be edited into the first image stylization editing network as first denoising information, adding a low-rank adaptation function in the first image stylization editing network, and optimizing the first image stylization editing network with the low-rank adaptation function by using the image diffusion loss; Inputting the style features of the reference style image into the second image stylization editing network as second denoising information, adding a low-rank adaptation function in the second image stylization editing network, and optimizing the second image stylization editing network with the added low-rank adaptation function by using the image diffusion loss; Based on the two optimized image stylization editing networks, a third image stylization editing network is constructed.

6. The method according to claim 5, characterized in that The step of constructing a third image stylization editing network according to the two optimized image stylization editing networks includes: According to the optimized first image stylization editing network and the optimized second image stylization editing network, among the 11 converter modules, the fifth and sixth converter modules are replaced by the fifth and sixth converter modules in the second image stylization editing network; the first to fourth and the seventh to eleventh converter modules are replaced by the corresponding modules in the first image stylization editing network, so as to generate the third image stylization editing network; Among them, a low-rank adaptation function is added to the cross-attention layer of the 5th and 6th converter modules in the third image stylization editing network.

7. The method according to claim 6, characterized in that The method further comprises: In the third image stylization editing network, inputting the content features of the image to be edited into all converter modules in the third image stylization editing network as content denoising guidance information; The style features of the reference style image are input into the fifth and sixth converter modules of the third image stylization editing network as style denoising guidance information.

8. The method according to any one of claims 1 to 7, characterized in that The method of iteratively updating the stylized single-view image to obtain the stylized 3D model by using an optimization strategy based on a combination of 3D content and style includes: Constructing a multi-loss function, wherein the multi-loss function includes: stylization loss, content preservation loss, geometry preservation loss and consistency loss; Obtaining a loss weight coefficient corresponding to each loss function, and calculating multiple loss parameters according to the loss weight coefficient and the multiple loss functions; By using an optimization strategy based on the combination of 3D content and style, the stylized edited single-view image is iteratively updated according to the multi-loss parameter constraint to obtain the stylized edited 3D model.

9. The method according to claim 8, characterized in that Constructing the stylized loss in the multi-loss function includes: Using the features extracted by the pre-trained convolutional neural network, the loss function between the rendered image and the style image under a set of training perspectives is calculated. The stylized loss is expressed as: in, is the stylized loss, The features extracted by the pre-trained convolutional neural network are used to render the 3D Gaussian model for stylized editing. The features extracted by the pre-trained convolutional neural network for the reference style image, Represents each pixel position of the image, Represents each pixel position of the 3D Gaussian model rendered image after stylized editing, represents the cosine distance, and N represents the number of pixels.

10. The method according to claim 8, characterized in that Constructing the content preservation loss in the multi-loss function includes: The content retention loss is calculated using a content retention loss relational expression, where the content retention loss relational expression is: in, To maintain loss for content, The features extracted by the pre-trained convolutional neural network are used to render the 3D Gaussian model for stylized editing. The source 3D Gaussian model rendering image is extracted from the features of the pre-trained convolutional neural network. H is the length of the rendered image and W is the width of the rendered image.

11. The method according to claim 10, characterized in that Constructing the geometry preservation loss in the multi-loss function includes: Obtaining a depth image obtained by rendering the source 3D model and a depth image obtained by rendering the edited 3D model; The geometry preservation loss is calculated according to the depth image obtained by rendering the source three-dimensional model, the depth image obtained by rendering the edited three-dimensional model, and a depth geometry preservation loss relationship, where the geometry preservation loss relationship is: in, is the geometry preservation loss, The depth image rendered from the source 3D model. The depth image obtained by rendering the edited 3D model, the depth image of the source 3D model after rendering The depth of each pixel is Calculated by the following relationship: in, is the 2D Gaussian after the i-th 3D Gaussian projection of the source 3D model, is the depth value corresponding to the two-dimensional Gaussian after the i-th projection of the source three-dimensional model, is the transparency of the two-dimensional Gaussian after the i-th projection of the source three-dimensional model.

12. The method according to claim 9, characterized in that Constructing the consistency loss in the multi-loss function includes: Use the pre-trained image caption generation model to describe the content and style of the edited 3D model rendering image, and obtain content description text and style description text respectively; Encoding the content description text and the style description text by using a text encoder to obtain content description text features and style description text features; According to the content description text features and the style description text features, the consistency loss is calculated using a semantic consistency loss relationship, where the semantic consistency loss relationship is: in, is the consistency loss, The features extracted from the image after the source 3D model is rendered by the image editor. Describe text features for style, is the style feature of the reference style image.

13. A three-dimensional scene image stylization editing device, characterized in that: The device comprises: An acquisition module, used to acquire a source 3D model and a reference style image, and acquire a first image stylization editing network and a second image stylization editing network; A processing module, configured to render the source 3D model to obtain a single-view image as the image to be edited, and to decouple content features of the image to be edited and style features of the reference style image; A construction module is used to use the content features and the style features as denoising information, and use image diffusion loss to optimize the first image stylization editing network and the second image stylization editing network to which the low-rank adaptation function is added, and to construct a third image stylization editing network, wherein the low-rank adaptation function is added to the cross-attention layer in the third image stylization editing network; The processing module is further used to input the image to be edited and the reference style image into the third image stylization editing network for processing, and output a single-view image after stylization editing; The editing module is used to iteratively update the stylized edited single-view image by using an optimization strategy based on the combination of 3D content and style to obtain a stylized edited 3D model.

14. An electronic device, characterized in that: comprising a memory and a processor, wherein the memory and the processor are connected; The memory stores computer instructions, and the processor executes the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 12 by executing the computer instructions.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 12.

16. A computer program product, characterized in that The method comprises computer instructions for causing a computer to execute the image stylization editing method for a three-dimensional scene according to any one of claims 1 to 12.

Images