A 3D model editing method, device, equipment and medium based on multi-modal

Abstract

The present invention relates to the field of computer technology, and particularly relates to a multi-modal based three-dimensional model editing method, device, equipment and medium. The method includes: obtaining a multi-modal fusion image personalized generation method and a source three-dimensional model; obtaining a personalized image generation network corresponding to the source three-dimensional model, and training the personalized image generation network; fusing based on a reference image list and corresponding text descriptions to obtain multi-modal embedding information; taking the trained personalized image generation network as a framework, adding multiple personalized learning modules to obtain multiple personalized image editing networks; using the multiple personalized image editing networks to edit the source three-dimensional scene image according to the multi-modal embedding information to obtain a single-view editing image. The multi-modal embedding information generated by this method allows users to input prompt information in multiple modalities such as text, images, and sketches, mining the complementarity of prompts in each modality, thereby improving the controllability of editing and achieving high-quality three-dimensional model editing.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a multi-modal based three-dimensional model editing method, device, equipment and medium. Background Art

[0002] Creating 3D assets plays a key role in many applications and industries, such as augmented virtual reality, movie or game production, art creation, etc. However, when creating high-quality 3D assets, modelers usually require rich experience and techniques, and spend a lot of labor and time to complete each step, from sketches to constructing geometric models, creating textures, materials, etc. A relatively efficient and inexpensive way to obtain 3D assets is to change the geometry or appearance of existing 3D assets through 3D editing means, so as to generate diverse 3D assets.

[0003] 3D editing based on text description has received wide attention due to its convenience and user-friendliness. The general process is to first use text description-based image editing, and then promote the image editing results to three-dimensional models. One way is to complete the editing by adding additional three-dimensional or two-dimensional constraints to avoid using image editing. Although this method realizes the editing that conforms to the text description, due to the inherent limitations of text description, the existing methods still lack precise control over the specified appearance and position of the editing results. Summary of the Invention

[0004] To solve the above problems, an embodiment of the present invention proposes a multi-modal based three-dimensional model editing method, which takes multiple modal cues such as text, images, sketches, etc. as inputs, mines the complementarity of multiple modal cues, realizes precise control over the editing results, and thus achieves controllable generation and editing of 3D models.

[0005] In a first aspect, the present invention provides a multi-modal based three-dimensional model editing method, the method comprising:

[0006] Obtain a multi-modal fusion image personalized generation method and a source three-dimensional model;

[0007] Based on the multi-modal fusion image personalized generation method, obtain a personalized image generation network corresponding to the source three-dimensional model, and train the personalized image generation network;

[0008] Based on a given list of reference images and text descriptions corresponding to the list of reference images, fuse to obtain multi-modal embedding information;

[0009] Using the trained personalized image generation network as a backbone, add multiple personalized learning modules to obtain multiple personalized image editing networks;

[0010] Using the multiple personalized image editing networks, the source 3D scene image in the source 3D model is edited according to the multimodal embedding information to obtain a single-view edited image.

[0011] For the method provided in this embodiment, first, based on the method for personalized image generation through multimodal fusion, a personalized image generation network corresponding to the source 3D model is obtained and trained. Then, based on the given reference image list and the text description corresponding to the reference image list, multimodal embedding information is obtained through fusion. This multimodal embedding information allows users to input prompt information in multiple modalities such as text, images, and sketches, and exploits the complementarity of prompts in each modality, thereby improving the controllability of editing and achieving high-quality 3D model editing.

[0012] In addition, during the image editing process, this method allows users to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows users to perform multiple object edits at once to achieve controllable multi-object editing.

[0013] Combined with the first aspect, in a possible implementation manner, the training of the personalized image generation network includes:

[0014] Extracting the text embedding and image embedding of the source 3D model;

[0015] Fusing the text embedding and the image embedding according to the multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and calculating the diffusion loss with the multimodal fusion embedding as a condition;

[0016] Training the image personalized generation network using the diffusion loss and the class loss to obtain a trained image personalized generation network.

[0017] Combined with the first aspect, in another possible implementation manner, the extracting of the text embedding and image embedding of the source 3D model includes:

[0018] According to the source 3D model, obtaining the text description corresponding to the source 3D model;

[0019] Constructing a text prompt according to the text description;

[0020] Encoding the text prompt using a pre-trained text encoder to generate the text embedding;

[0021] Rendering the source 3D model using an image rendering method to obtain an image set, where the image set includes at least one image;

[0022] Encoding the at least one image using a pre-trained image encoder to generate the image embedding.

[0023] In combination with the first aspect, in another possible implementation manner, if the generated image embedding is a target image embedding, then fusing the text embedding and the image embedding according to the multi-modal prompt fusion algorithm to obtain a multi-modal fusion embedding includes:

[0024] Connecting the target image embedding and the text embedding and inputting them into the self-attention layer of N multi-layer perceptrons and transformers to output the multi-modal fusion embedding; the expression of the multi-modal fusion embedding is:

[0025]

[0026] Wherein, is the multi-modal fusion embedding, is the text prompt, is the target image, is the CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0027] In combination with the first aspect, in another possible implementation manner, for the image personalized generation method based on the multi-modal fusion, obtaining the personalized image generation network corresponding to the source three-dimensional model includes:

[0028] Obtaining a diffusion model;

[0029] Using the diffusion model as a skeleton and adding a CLIP image encoder to construct the image personalized generation network;

[0030] Wherein, the image personalized generation network includes: one or more of a variational autoencoder, a U-Net network, a text encoder and an image encoder among them.

[0031] In combination with the first aspect, in another possible implementation manner, fusing the given reference image list and the text description corresponding to the reference image list to obtain multi-modal embedding information includes:

[0032] Obtaining a reference sketch and converting the reference sketch into a reference image;

[0033] According to the given reference image and the corresponding text description, fusing to obtain multi-modal reference prompt information and using the multi-modal reference prompt information as the denoising condition for multiple personalized image editing networks;

[0034] After obtaining the multiple personalized image editing networks, it further includes:

[0035] Calculate the corresponding diffusion loss according to the denoising condition; use the diffusion loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks; construct a multi-modal fusion method for reference images based on sketches;

[0036] Use the multiple personalized image editing networks to edit the source 3D scene image in the source 3D model according to the multi-modal embedding information to obtain a single-view edited image, including:

[0037] According to the rendered image of the source 3D scene and the corresponding multi-modal editing prompt information, use the trained multiple personalized image editing networks to perform editing to generate a single-view edited image that conforms to the editing prompt.

[0038] Combined with the first aspect, in another possible implementation manner, using the multi-modal reference prompt information as the denoising condition for the multiple personalized image editing networks includes:

[0039] Use a pre-trained multi-view image generation model to generate a corresponding list of multi-view reference images;

[0040] Obtain any reference image in the reference image list and select any one perspective image in the reference image; input the selected perspective image and the text prompt into the multi-modal prompt fusion network to obtain the fused multi-modal embedding information as the denoising condition for the multiple personalized image editing networks;

[0041] Before using the diffusion loss to train the multiple personalized image editing networks, it further includes: obtaining a localization loss and a class prior loss;

[0042] Using the localization loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks includes:

[0043] Use the localization loss, the diffusion loss, and the class prior loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks.

[0044] Combined with the first aspect, in another possible implementation manner, the expression of the localization loss is:

[0045]

[0046] where, represents the localization loss, m represents the number of editing masks, represents the cross-attention map of the i-th reference image; Mj represents the j-th editing mask on the specified perspective, and the editing mask Mj , constraining the j-th cross-attention map Infinitely close to the j-th editing mask , , the mean operation represents mean calculation;

[0047] The expression of the diffusion loss is as follows:

[0048]

[0049] where, is the diffusion function, is the implicit encoding at time t, is the multi-modal embedding information, ε is Gaussian noise, θ is the parameter, and E(x) is the variational autoencoder;

[0050] The expression of the class prior loss is as follows:

[0051]

[0052] where, represents the class prior loss, represents the class prior text prompt.

[0053] Combined with the first aspect, in another possible implementation, after obtaining the single-view editing image, it further includes:

[0054] Construct a multi-modal based 3D model editing network;

[0055] Perform 3D editing using the multi-modal based 3D editing method to obtain the edited 3D model;

[0056] Change the input multi-modal editing prompt information, perform stylized editing on the edited 3D model to change the original style, and obtain the 3D model with the target style through iterative training using multiple trained personalized image editing networks;

[0057] Construct a 3D model editing platform, the 3D model editing platform includes an editable user interface for assisting the user to complete the editing operation on the user interface and display one or more of the following functions: source 3D model input, multi-modal editing prompt input, 3D editing box specification, 3D model editing, and result display function.

[0058] Combined with the first aspect, in another possible implementation, when performing 3D editing using the multi-modal based 3D editing method to obtain the edited 3D model, it further includes:

[0059] Obtain the 3D Gaussian representation of the edited 3D model;

[0060] Initialize the 3D Gaussian representation of the edited 3D model to obtain the initialized edited 3D model;

[0061] Render the initialized edited 3D model to obtain the rendered single-view edited image;

[0062] Input the rendered single-view edited image into the multiple personalized image editing networks for denoising to obtain a clean single-view image.

[0063] Combined with the first aspect, in another possible implementation, during the denoising process of the rendered single-view edited image, it further includes:

[0064] Obtain the constraint conditions of the multi-condition loss function and the background reconstruction loss;

[0065] During the denoising process, use the constraint conditions of the multi-condition loss function and the background reconstruction loss for constraint to obtain the clean single-view image;

[0066] Among them, the constraint conditions of the multi-condition loss function are:

[0067]

[0068] The background reconstruction loss is;

[0069]

[0070] Among them, is the multi-condition loss function, 1, 2, 3 are the input text prompts, is the image rendering pose parameter, is the implicit encoding at time, is the 3D model editing network based on multi-modal, is the noise ground truth, represents expectation, represents weight, is the background reconstruction loss, is the image rendered from the source 3D model, is the image rendered from the edited 3D model, M is the editing mask projected from the 3D editing box specified by the user, is the logical operator, is the parameter of the 3D model, image I e Generated by rendering from the 3D model.

[0071] Optionally, the above 3D model is the edited 3D model.

[0072] In a second aspect, the present invention further provides a multi-modal based three-dimensional model editing device, the device comprising:

[0073] An acquisition module, configured to acquire a multi-modal fusion image personalized generation method and a source three-dimensional model;

[0074] A generation module, configured to obtain a personalized image generation network corresponding to the source three-dimensional model based on the multi-modal fusion image personalized generation method, and train the personalized image generation network;

[0075] A processing module, configured to fuse a given list of reference images and text descriptions corresponding to the list of reference images to obtain multi-modal embedding information;

[0076] The generation module is further configured to use the trained personalized image generation network as a backbone and add a plurality of personalized learning modules to obtain a plurality of personalized image editing networks;

[0077] An editing module, configured to use the plurality of personalized image editing networks to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view editing image.

[0078] In a third aspect, the present invention provides an electronic device, comprising: a memory and a processor, which are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to execute the multi-modal based three-dimensional model editing method according to the first aspect or any corresponding embodiment thereof.

[0079] In a fourth aspect, the present invention provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to cause a computer to execute the multi-modal based three-dimensional model editing method according to the first aspect or any corresponding embodiment thereof.

[0080] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, and the computer instructions are used to cause a computer to execute the multi-modal based three-dimensional model editing method according to the first aspect or any corresponding embodiment thereof.

[0081] In addition, the present invention further provides a computer program product, comprising computer instructions, and the computer instructions are used to cause a computer to execute the multi-modal based three-dimensional model editing method according to the first aspect or any corresponding embodiment thereof.

[0082] A multi-modal based 3D model editing method, device, equipment and medium provided by the present invention allow users to input prompt information in multiple modalities such as text, images, and sketches, and exploit the complementarity of each modal prompt, thereby improving the controllability of editing and achieving high-quality 3D model editing. In addition, it allows users to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows users to perform multiple object edits at once to achieve multi-object controllable editing.

[0083] The method provided by the present invention combines an attention-based local positioning loss to achieve multiple personalized image edits. During inference, a single forward pass can complete personalized image edits of multiple objects. Additionally, applying the proposed multiple personalized image editing method to achieve single-view image editing and using SDS and local reconstruction losses for iterative optimization to achieve 3D model editing. Furthermore, a multi-functional 3D editing method is proposed that can achieve various edits such as object insertion and object replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the description of the specific embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention, and those of ordinary skill in the art can obtain other drawings based on these drawings without creative efforts.

[0085] Figure 1 is a schematic diagram of a multi-modal based controllable 3D model editing inference provided according to an embodiment of the present invention;

[0086] Figure 2 is a flowchart of a multi-modal based 3D model editing method provided according to an embodiment of the present invention;

[0087] Figure 3 is a flowchart of a multi-modal fusion image personalized generation method provided according to an embodiment of the present invention;

[0088] Figure 4 is a flowchart of training an image personalized generation network provided according to an embodiment of the present invention;

[0089] Figure 5 is a flowchart of another multi-modal based 3D model editing method provided according to an embodiment of the present invention;

[0090] Figure 6 is a flowchart of a multiple personalized image editing method provided according to an embodiment of the present invention;

[0091] Figure 7It is a flowchart of another multi-modal based 3D model editing method provided according to an embodiment of the present invention;

[0092] Figure 8 It is a flowchart of yet another multi-modal based 3D model editing method provided according to an embodiment of the present invention;

[0093] Figure 9 It is a schematic diagram of a 3D model editing platform based on the Web provided according to an embodiment of the present invention;

[0094] Figure 10 It is a structural block diagram of a 3D model editing device provided according to an embodiment of the present invention;

[0095] Figure 11 It is a schematic diagram of the hardware structure of an electronic device provided according to an embodiment of the present invention. Detailed Embodiments

[0096] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0097] In the description of the present invention, it should be noted that unless otherwise clearly defined and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two components. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0098] In addition, the terms "first" and "second" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0099] The technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0100] To address the inherent limitations of text descriptions in image editing and the lack of precise control over the specified appearance and position of the editing results, embodiments of the present invention propose a multi-modal based controllable 3D model editing method that allows users to input prompt information in multiple modalities such as text, images, and sketches, explores the complementarity of prompts in each modality, thereby improving the controllability of editing and achieving high-quality 3D model editing.

[0101] In addition, it allows users to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows users to perform multiple object edits at once to achieve multi-object controllable editing.

[0102] Specifically, this 3D model editing method follows the process of first completing image editing and then elevating to 3D to complete 3D editing. In the training stage: First, given a source 3D model (i.e., the 3D model to be edited) and its corresponding text prompt, based on a multi-modal fusion based image personalization generation method, to learn the information of the source 3D model and obtain the image personalization generation network corresponding to the source 3D model. On this basis, according to the input multiple reference images or sketches and their corresponding text prompts, using multiple personalized image editing methods, edit the image rendered from the source 3D model. Obtain multiple personalized edited images of a single view. Finally, based on the multi-modal based 3D model editing method, in an iterative manner, use the trained multiple personalized image editing networks to edit the single-view image and use it to update the 3D model. After multiple iterations, obtain the edited 3D model.

[0103] In the inference stage, as Figure 1 shown, given the source 3D scene model, according to the input multi-modal prompts, use the trained multi-modal 3D scene editing network to edit the source 3D scene model to obtain a 3D scene model that meets the requirements of the given prompts.

[0104] The following provides a detailed description of the multi-modal based 3D model editing method provided by the embodiments of the present invention.

[0105] Embodiments of the present invention provide an embodiment of a multi-modal based 3D model editing method. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order than here.

[0106] In this embodiment, a multi-modal based 3D model editing method is provided, which can be used in devices such as terminal devices, clients, and servers. Figure 2 is a flowchart of the multi-modal based 3D model editing method according to the embodiments of the present invention. As Figure 2 shown, the method includes:

[0107] Step S101: Obtain a multi-modal fusion-based personalized image generation method and a source 3D model.

[0108] Among them, multi-modal refers to two or more expression forms of 3D images, such as by inputting text, images, sketches, etc. It should be understood that in this embodiment, in addition to the above input text, images, and sketches Figure 3 modalities, there may also be other more modalities, which are not limited in this embodiment.

[0109] The source 3D model refers to: a model with 3D data constructed through a virtual 3D space using 3D production software. These models are usually used in computers and video games, as well as in the special effects production of movies and TV shows, and even include fields such as engineering design, architectural design, and product manufacturing. In this embodiment, the source 3D model refers to a 3D environmental image captured in a real scene.

[0110] In addition, the multi-modal fusion-based personalized image generation method is used to generate a personalized image generation network, and the personalized image generation network is used to learn the source 3D scene and construct a multi-modal and personalized image editable network.

[0111] Step S102: Based on the multi-modal fusion-based personalized image generation method, obtain the personalized image generation network corresponding to the source 3D model and train the personalized image generation network.

[0112] Among them, training the personalized image generation network is used to learn the features of the source 3D model.

[0113] One implementation is as Figure 3 shown. To ensure that the non-editable area in the source 3D model (i.e., the area that will not be edited in the 3D scene) remains unchanged during the editing process, a multi-modal fusion-based personalized image learning method is proposed. Using this method, the existing pre-trained image generation diffusion model is fine-tuned to generate the image corresponding to the source 3D model. The specific steps are as follows:

[0114] First, using the given source 3D model, render the image corresponding to the source 3D model. Given the source 3D model represented by a 3D Gaussian , where , , represents the position of the center point, is the i-th 3D Gaussian point, represents the covariance matrix, -1 represents the inverse matrix, and the color of each point is represented by , and the transparency , then the 3D Gaussian representation is:

[0115]

[0116] Among them, represents the distance between the query point and the center point .

[0117] Therefore, the color of a certain pixel point in the single-view image of the source 3D model obtained by rendering is:

[0118]

[0119]

[0120] The above two formulas give the calculation process of the color of any pixel point. If an image is rendered, all pixel points on the current image can be rendered one by one according to the above formulas.

[0121] Then, a multi-modal prompt fusion algorithm is proposed to fuse multi-modal prompt information, so as to fully exploit the complementarity of multi-modal information. Among them, the multi-modal prompt information can be generated by fusing text embedding and image embedding.

[0122] Step S103: Based on the given reference image list and the corresponding text descriptions of the reference image list, multi-modal embedding information is obtained after fusion.

[0123] Among them, the fused multi-modal embedding information can be a unified prompt information. For example, by combining the text description information and the reference image information and inputting them into N MLP layers and the self-attention layer of the Transformer, unified prompt information is obtained, and finally multi-modal embedding information is obtained.

[0124] Optionally, the multi-modal embedding information can be expressed as "C f ".

[0125] MLP (Multilayer Perceptron, multi-layer perceptron). MLP is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. In a feedforward neural network, data flows from the input layer to the output layer, passing through one or more hidden layers in the middle. Transformer: A deep learning network based on self-attention and cross-attention mechanisms. Transformer is a deep learning network based on self-attention and cross (cross)-attention mechanisms. Transformer consists of two parts: an encoder (Encoder) and a decoder (Decoder). Both the encoder and the decoder are stacked by multiple identical layers, and each layer contains a self-attention mechanism and a feedforward neural network.

[0126] Step S104: Using the trained personalized image generation network as the backbone, add multiple personalized learning modules to obtain multiple personalized image editing networks.

[0127] The purpose of this step is to construct multiple personalized image editing networks. In one implementation, these multiple personalized image editing networks use the trained image personalization generation model in "Step S102" as the backbone network, add multiple personalized learning modules, and then generate. Among them, the multiple personalized learning modules are LoRA layers added to the cross-attention layer in the U-Net network.

[0128] The LoRA layer (Low-Rank Adaptation of Large Language Models) is a low-rank adaptation technique for fine-tuning large language models. In the application of image generation diffusion models, LoRA is used as a plugin, allowing users to train a model with specific painting styles or character features using a small amount of data without modifying the source model.

[0129] Step S105: Use the multiple personalized image editing networks to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image.

[0130] According to the input multiple reference images or reference sketches and the corresponding text prompts, use the multiple personalized image editing networks to edit the image rendered from the source three-dimensional model, so as to obtain multiple single-view personalized edited images.

[0131] If a reference sketch of a three-dimensional scene is received, use a conditional image generation model, such as ControlNet, to first convert the reference sketch into a reference image, and then execute Step S105.

[0132] The method provided in this embodiment first obtains a personalized image generation network corresponding to the source three-dimensional model based on the multi-modal fusion-based image personalization generation method, trains this personalized image generation network, and then based on the given reference image and the text description corresponding to the reference image, after fusion, obtains multi-modal embedding information. This multi-modal embedding information allows users to input prompt information in multiple modalities such as text, images, and sketches, and exploits the complementarity of prompts in each modality, thereby improving the controllability of editing and achieving high-quality three-dimensional model editing.

[0133] In a possible implementation of this embodiment, as Figure 4 shown, the above Step S102 for training the personalized image generation network specifically includes:

[0134] Step S201: Extract the text embedding and image embedding of the source three-dimensional model.

[0135] Step S202: The text embedding and the image embedding are fused according to the multi-modal prompt fusion algorithm to obtain a multi-modal fusion embedding, and the diffusion loss is calculated based on the multi-modal fusion embedding.

[0136] Step S203: The image personalized generation network is trained using the diffusion loss and the class loss to obtain a trained image personalized generation network.

[0137] Specifically, the method for obtaining the text embedding is as Figure 5 shown and includes the following:

[0138] First, according to the source 3D model, the text description corresponding to the source 3D model is obtained. For example, for a table "atable". Then, the text prompt Ts is constructed based on the text description. Given the text description "atable" corresponding to the source 3D model, the text prompt , or {one, V1, table}, is constructed and the pre-trained CLIP text encoder is used to encode the text prompt Ts into a text embedding.

[0139] Among them, CLIP (Contrastive Language-Image Pre-Training, a text-image pre-training model using contrastive learning) can map images and text to a shared vector space, and thus can be used for various visual and language tasks.

[0140] As Figure 5 shown, a method for obtaining the image embedding includes: First, the source 3D model is rendered using an image rendering method to obtain an image set, and the image set includes at least one image. Then, the pre-trained CLIP text encoder is used to encode the text prompt into a text embedding.

[0141] Furthermore, the method for obtaining the image embedding is as follows: Given the source 3D model and the above-mentioned multi-modal fusion image personalized generation method, the image set rendered in different poses , where is the number of source images, and the pre-trained CLIP image encoder is used to encode the image into an image embedding.

[0142] During the training process, a random image in the image set is selected, and the feature vector corresponding to the image embedding of it is concatenated with the text embedding of " " in the text prompt and incorporated into MLP layers and the self-attention layer of the Transformer to generate unified prompt information, and finally the multi-modal fusion embedding is obtained. The specific calculation is as follows:

[0143]

[0144] Among them, , is a multimodal fusion embedding, is a text prompt, is the target image, is the CLIP text encoder, is the CLIP image encoder, MLP is the Multilayer Perceptron, and ATT is the self-attention layer.

[0145] The above formula can make the multimodal embeddings fully fused through the MLP layer and the self-attention ATT layer. Especially, the self-attention layer can adaptively establish a stronger connection between the input image embeddings and and "table" in the text embeddings, making the fusion of the two modalities of image and text better.

[0146] Then, construct and train an image personalization generation network based on multimodal fusion to generate an image corresponding to the source 3D model. A specific implementation method is:

[0147] Obtain a diffusion model, such as DreamBooth. DreamBooth is a personalized text-to-image diffusion model. It is a method for fine-tuning the text-to-image diffusion model, and its biggest feature is Subject-Driven (theme-driven generation), that is, for a specific subject, generate context images of this subject in different scenes, poses, and perspectives.

[0148] Using the diffusion model (DreamBooth) as the backbone, additionally add the CLIP image encoder to construct an image personalization generation network. The structure of this image personalization generation network can be composed of at least one module or layer such as the variational autoencoder , U-Net network, text encoder and image encoder and so on.

[0149] Using the variational autoencoder E, compress the image into a smaller latent representation z, which is then perturbed by Gaussian noise ε during the forward diffusion process, and the U-Net parameterized by θ denoises by predicting the noise. This denoising process can be conditioned on the multimodal fusion embedding . Among them, the diffusion loss is specifically calculated as:

[0150]

[0151] Among them, is Implicit encoding of time is the diffusion loss, z is the latent representation denotes expectation

[0152] During the training process, in order to prevent category forgetting, a category loss function is added during the training of the image personalization generation network. The following formula is an expression of a category loss function:

[0153]

[0154] where is the category loss is the category prior text prompt, such as "table" to distinguish different tables

[0155] The image personalization generation network is trained using the category prior loss and the diffusion loss together to obtain the trained image personalization generation network

[0156] During the inference process, input the text description "a table", and an image describing the source 3D model can be generated, that is, the trained image personalization generation model network can recognize a specific source 3D model and ensure that the non-edited area remains unchanged during subsequent editing

[0157] After constructing and training the above image personalization generation model network, it further includes: using multiple personalized image editing methods, according to multi-modal editing prompts, editing the source 3D scene image to obtain a single-view editing image (i.e., a 2D image)

[0158] Such as Figure 6 shown, specifically including: fusing multi-modal reference prompt information as the denoising condition for multiple personalized image editing networks. In the above step S103, given a reference image and its corresponding text description, taking two input reference images as an example, an image list is constructed , an example of a text prompt is as follows

[0159] "a", " ", "cake", "and", " ", "apple", "on", " "table"}

[0160] Given a reference image, obtain multi-view reference images. In order for the multiple personalized image editing networks to learn the multi-view information of the reference images and thus improve the multi-view consistency of 3D model editing, for any given reference image, a pre-trained multi-view image generation model, such as MVDream, is used to generate a corresponding set of multi-view reference images or a list of reference images

[0161] Specifically, taking the reference image as an example, it is input into the pre-trained multi-view image generation model MVDream to obtain the corresponding reference image set , where M is the number of multi-view images in the reference image set.

[0162] Fuse multi-modal reference prompt information to construct the denoising conditions for multiple personalized image editing networks. During the training process, for any reference image in the reference image list , any one of the perspective images in the reference image is randomly selected, and the selected image is used as the target image. Correspondingly, the embedding information of the target image is called the target image embedding.

[0163] The selected target image and its corresponding text prompt are sent into the multi-modal prompt fusion network in step S102 to obtain the fused multi-modal embedding, which is used as the denoising condition for multiple personalized image editing networks.

[0164] Specifically, the text descriptor is sent into the text encoder to obtain the text embedding, and then the reference image and a certain perspective reference image corresponding to, such as the target image, passes through the image encoder to obtain the target image embedding, and is respectively connected with and the corresponding text embeddings, and are sent into multiple MLP layers and the self-attention layer of the Transformer to finally obtain the multi-modal embedding information , and the specific calculation is as follows:

[0165]

[0166] Among them, is the multi-modal fusion embedding, or the target image embedding, is the text prompt, and are the reference images, is the CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0167] Construct multiple personalized image editing networks and train them using the proposed localization loss. This process is similar to the method of constructing the image personalized generation network in the foregoing embodiment. The multiple personalized image editing networks use the image personalized generation network trained in step S103 as the backbone network, and by adding personalized learning modules, the personalized learning modules are LoRA layers added to the cross-attention layer in the U-Net network. Corresponding to the foregoing step S104.

[0168] During the training process, keep the weights of the image personalized generation network trained in step S102 unchanged, and use the proposed localization loss to train multiple personalized learning modules.

[0169] Specifically, construct a localization loss to train multiple personalized image editing networks. To enable multiple personalized image editing networks to distinguish different reference images and text prompts, a "localization loss" is proposed. Use the user-specified 3D editing box to locally locate the LoRA layer and train multiple personalized learning modules. Ideally, different cross-attention maps should only focus on the corresponding reference images instead of covering the entire image to prevent confusion between multiple reference images. To achieve this goal, this embodiment proposes to use the user-specified 3D bounding box to locally locate the cross-attention map.

[0170] Specifically, the cross-attention layer outputs cross-attention maps and connect the latent pixels to the denoising conditional embeddings of each layer, where represents the correlation relationship from the k-th conditional embedding to the (i, j)-th latent pixel, represents the size of the cross-attention map, and n represents the dimension of the cross-attention map. The user specifies the 3D region to be edited on the source 3D model, which is represented by a 3D bounding box , and respectively represent the editing regions that the reference image and should correspond to. According to the given 3D bounding box, during the training process, project the editing box onto the specified perspective to obtain the editing mask region on the specified perspective , I represents the index list of which reference image each word in the text prompt corresponds to, represents the cross-attention map of the i-th reference image, and constrain the j-th cross-attention map to be infinitely close to the j-th editing mask , that is , and use the L1 loss to minimize the distance between the cross-attention map and the editing mask:

[0171]

[0172] where, represents the localization loss, m represents the number of editing masks, and the mean operation represents the mean calculation.

[0173] In fact, the localization loss is applied to the cross-attention maps of the upsampling and downsampling in the U-Net network structure, and these cross-attention maps are known to contain more semantic information. The localization loss enables the model to accurately allocate attention to the reference image during testing, thereby preventing confusion between multiple personalized reference images.

[0174] In addition, in this embodiment, the localization loss , diffusion loss and class prior loss are also used to train multiple personalized image editing networks together. An expression of a diffusion loss is:

[0175]

[0176] where is the diffusion function, is the implicit encoding at time is the multi-modal embedding information, ε is Gaussian noise, θ is the parameter, and E(x) is the variational autoencoder.

[0177] During the training process, in order to prevent class forgetting, a class loss can also be added as follows:

[0178]

[0179] where is the class prior text prompt, such as "a cake and an apple on a table", which is translated as "a cake V2 and an apple V3 placed on a table V1".

[0180] The above method further includes: constructing a multi-modal fusion method for sketch-based reference images. A possible implementation is: input the reference image in the form of a sketch, and use the conditional image generation model ControlNet to convert the reference sketch into a reference image, and then the above operations can be performed.

[0181] The above step S105 specifically includes: according to the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt information, using the trained multiple personalized image editing networks for editing to generate a single-view editing image that conforms to the editing prompt.

[0182] Specifically, given the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt (prompt information C f ), using the trained multiple personalized image editing networks, a single-view editing image that conforms to the editing prompt can be obtained.

[0183] Such as Figure 7As shown, based on the source 3D editing model, through text prompt 1: A cake V2 and an apple V3 are placed on a table V1. Image prompt 1, including an image / photo of a cake and an apple. After multiple personalized image editing processes, a single-view editing image is output.

[0184] The method provided in this embodiment further includes: using a multi-modal based 3D model editing network to complete 3D scene editing that meets the requirements of multi-modal editing prompts. Specifically, as Figure 8 shown, after obtaining the single-view editing image in step S105 above, it further includes:

[0185] Step S301: Construct a multi-modal based 3D model editing network.

[0186] Specifically, based on the multi-modal 3D editing model, using a personalized image editing network as the backbone network, and keeping the pre-trained network parameters in the above steps unchanged. Represent the edited 3D model with a 3D Gaussian, where , where , , represents the position of the center point, represents the covariance matrix, and the color of each point is represented by , and the transparency , then the 3D Gaussian representation of the edited 3D model is:

[0187]

[0188] Among them, represents the distance between the query point and the center point .

[0189] The edited 3D model is initialized with the 3D Gaussian representation of the source 3D model.

[0190] Step S302: Use a multi-modal based 3D editing method for 3D editing to obtain an edited 3D model.

[0191] Step S303: Change the input multi-modal editing prompt information, perform stylized editing on the edited 3D model to change the original style, and use the trained personalized image editing network to obtain a 3D model with the target style through iterative training.

[0192] Step S304: Construct a 3D model editing platform, which includes an editable user interface for assisting the user to complete editing operations on the user interface.

[0193] In addition, the 3D model editing platform can also display one or more of the following functions: input of the source 3D model, input of multi-modal editing prompts, specification of a 3D editing box, 3D model editing, and result display functions.

[0194] Furthermore, step S302 specifically further includes: obtaining the 3D Gaussian representation of the edited 3D model; initializing the 3D Gaussian representation of the edited 3D model to obtain the initialized edited 3D model; using the initialized edited 3D model for rendering to obtain a rendered single-view edited image; inputting the rendered single-view edited image into multiple personalized image editing networks for denoising to obtain a clean single-view image.

[0195] Specifically, during the training process, according to the 3D model, a rendered image is obtained using a rendering method based on Gaussian representation, the rendered image is input into multiple personalized image editing networks to obtain an edited single-view edited image; the multi-condition loss and the background reconstruction loss are calculated, and the gradient is backpropagated to the 3D Gaussian representation of the edited 3D model, and the training process is iteratively repeated until a preset number of training times is reached.

[0196] During the inference process, given the source 3D model and the corresponding multi-modal editing prompts, a trained multi-modal-based 3D model editing method is used to obtain the edited 3D model.

[0197] Among them, a constraint expression for a multi-condition loss is:

[0198]

[0199] Among them, the multi-condition loss function, is the input text prompt, is the image rendering pose parameter, is the rendering function of the 3D Gaussian representation, is the multi-modal-based 3D model editing network, represents optimizing the 3D Gaussian part included in the 3D editing box specified by the user.

[0200] According to step S302, using the edited 3D model, the color of a certain pixel point of the rendered single-view image is:

[0201]

[0202]

[0203] Obtain the rendered single-view image '.

[0204] In addition, during the training phase, it further includes: constructing a loss function based on background reconstruction loss and multi-condition SDS (Score Distillation Sampling) loss, and training a multi-modal 3D editing model. During the training process, the weights of multiple personalized image editing networks are kept unchanged, and the 3D Gaussian representation of the edited 3D model is optimized using the reconstruction loss and SDS loss. Specifically, the single-view image rendered from the edited 3D model is input into multiple personalized image editing models (networks) for denoising, and the rendered single-view image 'is input into multiple personalized image editing models (networks) for denoising.

[0205] In this embodiment, the input text prompt , the text prompt , the text prompt will generate three SDS losses, and the final multi-condition SDS loss is:

[0206]

[0207] Meanwhile, the background reconstruction loss is added to ensure that the non-edited area remains unchanged. This reconstruction loss is:

[0208]

[0209] Among them, is the multi-condition loss function, 1, 2, 3 are the input text prompts, is the image rendering pose parameter, is the implicit encoding at time is the multi-modal 3D model editing network, is the noise ground truth, represents the expectation, represents the weight, is the background reconstruction loss, is the image rendered from the source 3D model, is the image rendered from the edited 3D model, M is the editing mask projected from the 3D editing box specified by the user, is a logical operator. In logic, ⊙ usually represents the exclusive NOR operation (XNOR), are the parameters of the 3D model, and the image I e is generated by rendering from the 3D model.

[0210] Optionally, the above 3D model is the edited 3D model.

[0211] During the training process, for the current edited 3D model, a rendered image is obtained using a rendering method based on Gaussian representation, and then input into multiple personalized image editing networks to obtain the edited single-view image, and the SDS loss is calculated. and the background reconstruction loss , and the gradient is backpropagated to optimize the Gaussian representation of the edited 3D model, and the above process is iteratively repeated until a certain number of training times is reached.

[0212] During the inference process, given the source 3D model and the corresponding multi-modal editing prompt, using the trained 3D model editing method based on multi-modal, the edited 3D model can be obtained.

[0213] Among them, the SDS loss, that is, the score distillation sampling loss, is a loss function used to optimize deep learning models. It is usually used in image generation tasks, especially in Neural Radiance Fields (NeRF) and related technologies. The SDS loss measures the difference between the generated image and the real image in a specific way, so as to guide the training process of the model.

[0214] In this embodiment, the above method further includes: changing the input multi-modal editing prompt to complete multi-functional editing.

[0215] The above method can complete addition editing for the existing 3D model. By changing the added multi-modal editing prompt, the existing 3D model can be stylized and its style can be changed.

[0216] For example, if you want to change the current 3D model into Van Gogh style, keep steps S101 - S102 unchanged. In step S103, given a reference image in Van Gogh style, the corresponding text description "a photo of table of style", and the editing position specified by the user-defined 3D bounding box, retrain multiple personalized image editing networks, then the editing of the rendered image of the source 3D model can be completed to make it in Van Gogh style. Replace the model in step S103 with the already trained multiple personalized image editing networks, and after iterative training, a 3D model in Van Gogh style can be obtained.

[0217] As Figure 9 shown, the method provided in this embodiment further includes: constructing a Web-based 3D model editing platform, providing an operable UI interface to assist users in completing editing.

[0218] To assist users in completing editing, a Web-based 3D model editing platform is constructed, providing an operable UI interface (such as Figure 9As shown in the figure, the main functions include: input of the source 3D model, input of multimodal editing prompts, specification of the 3D editing box, 3D model editing, and result display function. These functions can respectively correspond to the following prompt words or prompt messages: Please input the source 3D model and the corresponding text prompt, Please input the 3D editing box, Please input multimodal editing prompts, 3D model editing, etc.

[0219] Specifically, a user interaction process includes:

[0220] When it is detected that the user clicks the control of "Please input the source 3D model and the corresponding text prompt", a 3D model and text prompt input box will pop up; after entering the text description, the background will automatically perform image personalized generation network training, and after the training is completed, a prompt "Training completed, please continue to input the 3D bounding box and multimodal editing prompts" will pop up.

[0221] When it is detected that the user clicks the control of "Please input the 3D bounding box", manually mark the editing position to be added. Click on "Please input reference image" and "Please input text prompt" of "Please input multimodal editing prompts", and respectively input the reference image and text prompt. The background will automatically train multiple personalized image generation networks, and after the training is completed, a prompt "Training completed, please click 3D model editing" will pop up. When it is detected that the user clicks "3D model editing", the background will automatically train the 3D model editing based on multimodality and output the edited 3D model.

[0222] A 3D model editing method based on multimodality provided by the present invention allows users to input prompt information in multiple modalities such as text, images, and sketches, and explores the complementarity of prompt information in each modality, thereby improving the controllability of editing and achieving high-quality 3D model editing. In addition, it allows users to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows users to perform multiple object edits at one time to achieve multi-object controllable editing.

[0223] In addition, this method also fine-tunes the image generation diffusion model to output images that conform to multiple modality descriptions, improving the controllability of image generation. At the same time, the consistency of non-editing areas during the editing process is improved.

[0224] A multiple personalized image editing method provided by an embodiment of the present invention applies the proposed multimodal fusion method and combines it with an attention-based local positioning loss to achieve multiple personalized image editing. During inference, a single forward process can complete the personalized image editing of multiple objects.

[0225] In addition, the multi-modal based 3D model editing method of this embodiment realizes single-view image editing, and uses SDS and local reconstruction loss for iterative optimization to realize 3D model editing. In addition, a multi-functional 3D editing method is also proposed, which can realize various edits such as object insertion and object replacement.

[0226] It should be noted that the method proposed in this embodiment can be used to efficiently edit existing 3D assets, so as to obtain more diverse 3D assets, saving the modeling time and economic cost of 3D assets. And it can be used to support the product development of metaverse servers, such as MetaEngine and AI servers, and their applications in the field of 3D content generation and editing.

[0227] The multi-modal based 3D model editing method proposed in the embodiment of the present invention can be added to existing 3D modeling software as a plug-in, thereby improving the efficiency and quality of modeling.

[0228] In this embodiment, a multi-modal based 3D model editing device is also provided. This device is used to implement the above-mentioned embodiments and preferred implementation manners, and those that have been described will not be repeated. As used hereinafter, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0229] This embodiment provides a 3D model editing device, as Figure 10 shown, the device includes: an acquisition module 110, a generation module 120, a processing module 130, and an editing module 140. In addition, the device may further include other more or fewer modules, such as a transceiver module, a storage module, etc., and this embodiment does not limit this.

[0230] Among them, the acquisition module 110 is used to acquire the multi-modal fusion image personalized generation method and the source 3D model.

[0231] The generation module 120 is used to obtain the personalized image generation network corresponding to the source 3D model based on the multi-modal fusion image personalized generation method, and train the personalized image generation network.

[0232] The processing module 130 is used to fuse the given reference image list and the text description corresponding to the reference image list to obtain multi-modal embedding information.

[0233] The generation module 120 is also used to use the trained personalized image generation network as a framework and add multiple personalized learning modules to obtain multiple personalized image editing networks.

[0234] An editing module 140, configured to use multiple personalized image editing networks to edit the source 3D scene image in the source 3D model according to the multi-modal embedding information to obtain a single-view edited image.

[0235] In some optional embodiments, the generating module 120 is specifically configured to extract the text embedding and the image embedding of the source 3D model; fuse the text embedding and the image embedding according to the multi-modal prompt fusion algorithm to obtain a multi-modal fusion embedding, and calculate the diffusion loss based on the multi-modal fusion embedding; train the image personalization generation network by using the diffusion loss and the category loss to obtain a trained image personalization generation network.

[0236] In some other optional embodiments, the generating module 120 is further specifically configured to obtain the text description corresponding to the source 3D model according to the source 3D model; construct a text prompt according to the text description; encode the text prompt by using a pre-trained text encoder to generate a text embedding; render the source 3D model by using an image rendering method to obtain an image set, where the image set includes at least one image; encode at least one image by using a pre-trained image encoder to generate an image embedding.

[0237] In some other alternative embodiments, the generating module 120 is further configured to connect the target image embedding and the text embedding and input them into the self-attention layer of N multi-layer perceptrons and transformers to output a multi-modal fusion embedding.

[0238] The expression of the multi-modal fusion embedding is:

[0239]

[0240] Wherein, is the multi-modal fusion embedding, is the text prompt, is the target image, is the CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0241] In some other alternative embodiments, the obtaining module 110 is further configured to obtain a diffusion model; the generating module 120 is further configured to use the diffusion model as a backbone and add a CLIP image encoder to construct a generated image personalization generation network.

[0242] Wherein, the image personalization generation network includes: a variational autoencoder, a U-Net network, a text encoder and an image encoder one or more of them.

[0243] In some other alternative embodiments, the processing module 130 is specifically configured to obtain a reference sketch, convert the reference sketch into a reference image, fuse the reference image list and the corresponding text description to obtain multi-modal reference prompt information, and use the multi-modal reference prompt information as the denoising condition for multiple personalized image editing networks.

[0244] The generation module 120 is further configured to calculate the corresponding diffusion loss according to the denoising condition after obtaining the multiple personalized image editing networks; use the diffusion loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks; and construct a multi-modal fusion method for reference images based on sketches.

[0245] The editing module 140 is specifically configured to edit according to the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt information, and use the trained multiple personalized image editing networks to generate a single-view editing image that conforms to the editing prompt.

[0246] In some other alternative embodiments, the acquisition module 110 is further configured to obtain any reference image in the reference image list and select an image of any perspective in the multi-view image set; the generation module 120 is further configured to input the selected perspective image and the text prompt into the multi-modal prompt fusion network to obtain the fused multi-modal embedding information as the denoising condition for the multiple personalized image editing networks;

[0247] The acquisition module 110 is further configured to obtain the localization loss and the class prior loss before the generation module 120 uses the diffusion loss to train the multiple personalized image editing networks.

[0248] The generation module 120 is further configured to use the localization loss, the diffusion loss, and the class prior loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks.

[0249] In some other alternative embodiments, after obtaining the single-view editing image by using the editing module 140, the processing module 130 is further configured to construct a three-dimensional model editing network based on multi-modal; perform three-dimensional editing by using the multi-modal three-dimensional editing method to obtain the edited three-dimensional model; change the input multi-modal editing prompt information to perform stylized editing on the edited three-dimensional model to change the original style, and use the trained multiple personalized image editing networks to iteratively train to obtain a three-dimensional model with the target style; and construct a three-dimensional model editing platform.

[0250] The above device further includes a display module for displaying a 3D model editing platform. The 3D model editing platform includes an editable user interface for assisting the user to complete editing operations on the user interface and displaying one or more of the following functions: source 3D model input, multi-modal editing prompt input, 3D editing box specification, 3D model editing, and result display function.

[0251] In some further alternative embodiments, the editing module 140 is specifically configured to, during the training process, obtain a rendered image according to the 3D model by using a rendering method based on Gaussian representation, and input the rendered image into a plurality of personalized image editing networks to obtain an edited single-view image.

[0252] The processing module 130 is further configured to obtain the 3D Gaussian representation of the edited 3D model; initialize the 3D Gaussian representation of the edited 3D model to obtain an initialized edited 3D model; perform rendering by using the initialized edited 3D model to obtain a rendered single-view edited image; and input the rendered single-view edited image into the plurality of personalized image editing networks for denoising to obtain a clean single-view image.

[0253] The further function descriptions of the above-mentioned various modules and units are the same as those in the corresponding foregoing embodiments, and will not be elaborated herein.

[0254] The 3D model editing device in this embodiment is presented in the form of functional units. Here, the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and a memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0255] The 3D model editing device provided in this embodiment allows the user to input prompt information in multiple modalities such as text, images, and sketches, explores the complementarity of prompts in each modality, thereby improving the controllability of editing and achieving high-quality 3D model editing. In addition, it also allows the user to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows the user to perform multiple object edits at one time to achieve multi-object controllable editing.

[0256] An embodiment of the present invention further provides an electronic device having the above Figure 11 shown 3D model editing device.

[0257] Please refer to Figure 11, which is a schematic structural diagram of an electronic device provided by an alternative embodiment of the present invention. The electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including a high-speed interface and a low-speed interface. Each component communicates with each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface).

[0258] In some alternative embodiments, if needed, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (for example, as a server array, a set of blade servers, or a multi-processor system). Figure 11 Taking one processor 10 as an example.

[0259] The processor 10 can be a central processing unit, a network processor, or a combination thereof. Among them, the processor 10 can further include a hardware chip. The above hardware chip can be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The above programmable logic device can be a complex programmable logic device, a field programmable gate array, a generic array logic, or any combination thereof.

[0260] Among them, the memory 20 stores instructions executable by at least one processor 10, so that at least one processor 10 executes the multi-modal-based three-dimensional model editing method shown in the above embodiments.

[0261] The memory 20 can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 20 can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some alternative embodiments, the memory 20 can include a memory remotely set relative to the processor 10, and these remote memories can be connected to the electronic device through a network. Examples of the above network include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0262] The memory 20 can include a volatile memory, for example, a random access memory; the memory can also include a non-volatile memory, for example, a flash memory, a hard disk, or a solid-state drive; the memory 20 can also include a combination of the above types of memories.

[0263] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30, and the output device 40 can be connected through a bus or other means. Figure 11 Here, taking the connection through the bus as an example.

[0264] The input device 30 can receive input digital or character information, and generate signal inputs related to user settings and function controls of the electronic device, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, a pointing stick, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 can include a display device, an auxiliary lighting device (e.g., an LED), and a haptic feedback device (e.g., a vibration motor), etc. The above display device includes but is not limited to a liquid crystal display, a light-emitting diode, a display, and a plasma display. In some alternative embodiments, the display device can be a touch screen.

[0265] In addition, the electronic device further includes at least one communication interface for the device to communicate with other devices or communication networks.

[0266] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or be implemented as computer code that can be recorded on a storage medium, or be implemented as computer code that is originally stored in a remote storage medium or a non-transitory machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processed on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware.

[0267] Among them, the storage medium can be a magnetic disk, an optical disc, a read-only memory, a random access memory, a flash memory, a hard disk, or a solid-state drive, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, the processor, or the hardware, the three-dimensional model editing method based on multi-modal shown in the above embodiment is implemented.

[0268] Embodiments of the present application may further provide a computer program product, including computer program instructions, which, when run by a processor, cause the processor to execute the steps in the above method. Among them, the computer program product can be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present disclosure. The programming languages include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computing device, partially on the user's device, executed as an independent software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0269] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them; although the embodiments of the present invention have been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional model editing method based on multi-modalities, characterized in that, The method includes: Obtaining a method for personalized generation of multimodal fusion images and a source 3D model, where the multimodality refers to two or more expression forms of 3D images, including at least three modalities: input text, image, and sketch, and the source 3D model refers to a 3D environment model captured in a real scene; Obtaining the text description corresponding to the source 3D model, constructing a text prompt according to the text description, and encoding the text prompt to generate a text embedding; Rendering the source 3D model using an image rendering method to obtain an image set, and encoding an image in the image set using a pre-trained image encoder to generate an image embedding; According to the text embedding and the image embedding, fusing the text embedding and the image embedding according to the multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and calculating a diffusion loss with the multimodal fusion embedding as a condition, and training the image personalized generation network corresponding to the source 3D model using the diffusion loss and the class loss to obtain a trained image personalized generation network; Based on a given list of reference images and the text description corresponding to the list of reference images, fusing to obtain multimodal embedding information; Using the trained personalized image generation network as a backbone, adding multiple personalized learning modules to obtain multiple personalized image editing networks, where the multiple personalized learning modules are LoRA layers added to the cross-attention layer in the network; Using the multiple personalized image editing networks to edit the source 3D scene image in the source 3D model according to the multimodal embedding information to obtain a single-view edited image; Among them, the fusing based on a given list of reference images and the text description corresponding to the list of reference images to obtain multimodal embedding information includes: Obtaining multi-view reference images according to the given reference images, and generating the list of reference images according to the multi-view reference images; where when the given reference image is a reference sketch, converting the reference sketch into a reference image and adding it to the list of reference images; Fusing according to the list of reference images and the corresponding text description to obtain multimodal reference prompt information, and using the multimodal reference prompt information as the denoising condition for the multiple personalized image editing networks, where the text description includes a text prompt.

2. The method according to claim 1, wherein If the generated image embedding is a target image embedding, then the fusing the text embedding and the image embedding according to the multimodal prompt fusion algorithm to obtain a multimodal fusion embedding includes: Connecting the target image embedding with the text embedding and inputting it into the self-attention layer of N multi-layer perceptrons and transformers to output the multimodal fusion embedding; the expression of the multimodal fusion embedding is: Among them, is a multimodal fusion embedding, is a text prompt, is the target image, is the CLIP text encoder, is the CLIP image encoder, MLP is a multi-layer perceptron, and ATT is a self-attention layer.

3. The method according to claim 1, wherein Before training the image personalized generation network corresponding to the source 3D model using the diffusion loss and the class loss, it further includes: Obtaining a diffusion model; Using the diffusion model as a backbone and adding a CLIP image encoder to construct the image personalized generation network; Among them, the image personalized generation network includes: a variational autoencoder, a U-Net network, and a text encoder and an image encoder .

4. The method according to claim 1, characterized in that, After obtaining the multiple personalized image editing networks, it further includes: Calculate the corresponding diffusion loss according to the denoising condition; Use the diffusion loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks; Use the multiple personalized image editing networks to edit the source 3D scene image in the source 3D model according to the multi-modal embedding information to obtain a single-view edited image, including: According to the rendered image of the source 3D scene and the corresponding multi-modal editing prompt information, use the trained multiple personalized image editing networks to perform editing to generate a single-view edited image that conforms to the editing prompt.

5. The method according to claim 1, characterized in that, Use the multi-modal reference prompt information as the denoising condition for the multiple personalized image editing networks, including: Use a pre-trained multi-view image generation model to generate a corresponding list of multi-view reference images; Obtain any reference image in the reference image list and select a perspective image in the reference image; Input the selected perspective image and the text prompt into the multi-modal prompt fusion network to obtain the fused multi-modal embedding information as the denoising condition for the multiple personalized image editing networks; Before using the diffusion loss to train the multiple personalized image editing networks, it further includes: Obtain the localization loss and the class prior loss; Using the localization loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks, including: Use the localization loss, the diffusion loss, and the class prior loss to train the multiple personalized image editing networks to obtain the trained multiple personalized image editing networks.

6. The method according to claim 5, characterized in that, The expression of the localization loss is: Among them, represents the positioning loss, m represents the number of editing masks, represents the cross-attention map of the j-th editing mask corresponding to the i-th reference image; M j represents the j-th editing mask on the specified perspective, and the editing mask , and the mean operation represents the mean calculation; The expression of the diffusion loss is: Among them, is the diffusion function, is the implicit encoding at time is the multimodal embedding information, ε is Gaussian noise, θ is the parameter, E(x) is the variational autoencoder, is the expectation; The expression of the class prior loss is: Among them, represents the class prior loss, represents the class prior text prompt.

7. The method according to any one of claims 1-6, characterized in that, After obtaining the single-view edited image, it further includes: Construct a multi-modal based 3D model editing network; Use the multi-modal based 3D editing method to perform 3D editing to obtain the edited 3D model; Change the input multi-modal editing prompt information to perform stylized editing on the edited 3D model to change the original style, and use the trained multiple personalized image editing networks to obtain the 3D model with the target style through iterative training; Construct a 3D model editing platform, and the 3D model editing platform includes an editable user interface for assisting the user to complete editing operations on the user interface and display one or more of the following functions: source 3D model input, multi-modal editing prompt input, 3D editing box specification, 3D model editing, and result display functions.

8. The method according to claim 7, characterized in that, Using the multi-modal based 3D editing method to perform 3D editing to obtain the edited 3D model, it further includes: Obtain the 3D Gaussian representation of the edited 3D model; Initialize the 3D Gaussian representation of the edited 3D model to obtain the initialized edited 3D model; Use the initialized edited 3D model for rendering to obtain the rendered single-view edited image; Input the rendered single-view edited image into the multiple personalized image editing networks for denoising to obtain a clean single-view image.

9. The method according to claim 8, characterized in that During the denoising process of the rendered single-view edited image, it further includes: Obtain the constraint conditions of the multi-condition loss function and the background reconstruction loss; During the denoising process, use the constraint conditions of the multi-condition loss function and the background reconstruction loss for constraint to obtain the clean single-view image; Among them, the constraint conditions of the multi-condition loss function are: The background reconstruction loss is; Among them, is a multi - condition loss function, 1, 2, 3 are the input text prompts, is the image rendering pose parameter, is the implicit encoding at time is a multi - modal based 3D model editing network, is the noise ground truth, represents expectation, represents weight, is the background reconstruction loss, is the image rendered from the source 3D model, is the image rendered from the edited 3D model, M is the editing mask projected from the 3D editing box specified by the user, is a logical operator.

10. A three-dimensional model editing device based on multi-modalities, characterized in that, The device includes: An acquisition module, configured to acquire an image personalized generation method and a source three-dimensional model for multi-modal fusion, where the multi-modal refers to two or more expression forms of three-dimensional images, at least including three modalities of input text, image, and sketch, and the source three-dimensional model refers to a three-dimensional environment model captured in a real scene; The acquisition module is further configured to acquire the text description corresponding to the source three-dimensional model, construct a text prompt according to the text description, and encode the text prompt to generate a text embedding; A generation module, configured to render the source three-dimensional model by using an image rendering method to obtain an image set, encode an image in the image set by using a pre-trained image encoder to generate an image embedding, and fuse the text embedding and the image embedding according to the multi-modal prompt fusion algorithm based on the text embedding and the image embedding to obtain a multi-modal fusion embedding, and calculate a diffusion loss with the multi-modal fusion embedding as a condition, and train the image personalized generation network corresponding to the source three-dimensional model by using the diffusion loss and the category loss to obtain a trained image personalized generation network; A processing module, configured to fuse based on a given list of reference images and the text description corresponding to the list of reference images to obtain multi-modal embedding information; The generation module is further configured to use the trained personalized image generation network as a framework and add multiple personalized learning modules to obtain multiple personalized image editing networks, and the multiple personalized learning modules are LoRA layers added on the cross-attention layer in the network; An editing module, configured to use the multiple personalized image editing networks to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image; Among them, the processing module is specifically configured to obtain multi-view reference images according to the given reference images, and generate the list of reference images according to the multi-view reference images, where when the given reference image is a reference sketch, convert the reference sketch into a reference image and add it to the list of reference images; Fuse according to the list of reference images and the corresponding text description to obtain multi-modal reference prompt information, and use the multi-modal reference prompt information as the denoising condition for the multiple personalized image editing networks, where the text description includes a text prompt.

11. An electronic device, characterized in that, Includes a memory and a processor, and the memory is connected to the processor; The memory stores computer instructions, and the processor executes the computer instructions to execute the multi-modal based three-dimensional model editing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing a computer to execute the multimodal-based three-dimensional model editing method according to any one of claims 1 to 9.

13. A computer program product, characterized in that, It includes computer instructions for causing a computer to execute the multimodal-based three-dimensional model editing method according to any one of claims 1 to 9.

Images