Three-dimensional model editing method and device based on multiple modes, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to a multi-mode-based three-dimensional model editing method and device, equipment and a medium. The method comprises the following steps: acquiring a multi-modal fusion image personalized generation method and a source three-dimensional model; obtaining a personalized image generation network corresponding to the source three-dimensional model, and training the personalized image generation network; based on the reference image list and the corresponding text description, multi-modal embedded information is obtained after fusion; taking the trained personalized image generation network as a skeleton, and adding a multi-personalized learning module to obtain a multi-personalized image editing network; and editing the source three-dimensional scene image according to the multi-modal embedded information by using a multi-personalized image editing network to obtain a single-view-angle edited image. The multi-mode embedded information generated by the method allows a user to input prompt information of multiple modes such as texts, images, sketches and the like, complementarity of prompt of each mode is mined, so that controllability of editing is improved, and high-quality three-dimensional model editing is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a multimodal three-dimensional model editing method, device, equipment and medium. Background Art

[0002] Creating 3D assets plays a key role in many applications and industries, such as augmented virtual reality, film or game production, artistic creation, etc. However, when creating high-quality 3D assets, modelers usually need rich experience and skills, and spend a lot of labor and time to complete each step, from sketching to building geometric models, creating textures, materials, etc. A more efficient and inexpensive way to obtain 3D assets is to generate diverse 3D assets based on existing 3D assets by changing their geometry or appearance through 3D editing methods.

[0003] 3D editing based on text description has attracted wide attention due to its convenience and user-friendliness. The usual process is to first use text-based image editing and then promote the image editing results to a 3D model. One way is to complete the editing by adding additional 3D or 2D constraints, thereby avoiding the use of image editing. Although this method achieves editing that conforms to the text description, due to the inherent limitations of text description, existing methods still lack precise control over the specified appearance and position of the editing results. Summary of the invention

[0004] In order to solve the above problems, an embodiment of the present invention proposes a three-dimensional model editing method based on multimodality, which takes multiple modal prompts such as text, images, sketches, etc. as input, explores the complementarity of multiple modal prompts, and realizes precise control of the editing results, thereby achieving controllable generation and editing of 3D models.

[0005] In a first aspect, the present invention provides a multimodal 3D model editing method, the method comprising: Obtaining personalized image generation methods and source 3D models of multimodal fusion; Based on the multimodal fusion image personalized generation method, obtaining a personalized image generation network corresponding to the source three-dimensional model, and training the personalized image generation network; Based on a given reference image list and a text description corresponding to the reference image list, multimodal embedding information is obtained after fusion; Taking the trained personalized image generation network as the skeleton, add multiple personalized learning modules to obtain a multiple personalized image editing network; The multi-personalized image editing network is used to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image.

[0006] The method provided in this embodiment first obtains a personalized image generation network corresponding to a source three-dimensional model based on a personalized image generation method of multimodal fusion, and trains the personalized image generation network. Then, based on a given reference image list and a text description corresponding to the reference image list, multimodal embedding information is obtained after fusion. The multimodal embedding information allows a user to input prompt information of multiple modalities such as text, image, sketch, etc., and explores the complementarity of prompts of each modality, thereby improving the controllability of editing and achieving high-quality three-dimensional model editing.

[0007] In addition, during the image editing process, this method allows the user to perform various editing operations, such as object insertion, object replacement, etc., to achieve functional editing; at the same time, it allows the user to edit multiple objects at a time to achieve controllable editing of multiple objects.

[0008] In combination with the first aspect, in a possible implementation manner, the training the personalized image generation network includes: extracting text embeddings and image embeddings of the source three-dimensional model; The text embedding and the image embedding are fused according to a multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and the diffusion loss is calculated based on the multimodal fusion embedding; The image personalized generation network is trained using the diffusion loss and the category loss to obtain a trained image personalized generation network.

[0009] In combination with the first aspect, in another possible implementation, extracting text embedding and image embedding of the source three-dimensional model includes: According to the source three-dimensional model, obtaining a text description corresponding to the source three-dimensional model; constructing a text prompt according to the text description; Encoding the text prompt using a pre-trained text encoder to generate the text embedding; Rendering the source three-dimensional model using an image rendering method to obtain an image set, wherein the image set includes at least one image; The at least one image is encoded using a pre-trained image encoder to generate the image embedding.

[0010] In combination with the first aspect, in another possible implementation, if the generated image embedding is a target image embedding, fusing the text embedding and the image embedding according to a multimodal prompt fusion algorithm to obtain a multimodal fusion embedding includes: The target image embedding is connected with the text embedding and input into the self-attention layer of N multi-layer perceptrons and transformers, and the multimodal fusion embedding is output; the expression of the multimodal fusion embedding is:

[0011] in, is multimodal fusion embedding, is a text prompt, is the target image, For CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0012] In combination with the first aspect, in another possible implementation, the method for generating personalized images based on the multimodal fusion, obtaining a personalized image generation network corresponding to the source three-dimensional model, includes: Get the diffusion model; The diffusion model is used as a skeleton, a CLIP image encoder is added, and the image personalized generation network is constructed and generated; The image personalized generation network includes: variational autoencoder, U-Net network, text encoder and image encoder One or more of .

[0013] In combination with the first aspect, in another possible implementation, the multimodal embedding information is obtained by fusing a given reference image list and a text description corresponding to the reference image list, including: Obtaining a reference sketch, and converting the reference sketch into a reference image; According to the given reference image and the corresponding text description, multimodal reference prompt information is obtained by fusing, and the multimodal reference prompt information is used as a denoising condition of a multi-personalized image editing network; After obtaining the multi-personalized image editing network, the method further includes: Calculating the corresponding diffusion loss according to the denoising condition; using the diffusion loss to train the multi-personalized image editing network to obtain the trained multi-personalized image editing network; constructing a sketch-based reference image multimodal fusion method; Using the multi-personalized image editing network, editing the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image, including: According to the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt information, the trained multi-personalized image editing network is used for editing to generate a single-view editing image that meets the editing prompt.

[0014] In combination with the first aspect, in another possible implementation, using the multimodal reference prompt information as a denoising condition of a multi-personalized image editing network includes: Using the pre-trained multi-view image generation model, generate a corresponding multi-view reference image list; Obtain any reference image in the reference image list, and select any view image in the reference image; input the selected view image and the text prompt into the multimodal prompt fusion network to obtain fused multimodal embedding information as a denoising condition for the multi-personalized image editing network; Before the use of the diffusion loss to train the multi-personalized image editing network, the method further includes: obtaining a positioning loss and a category prior loss; The method of training the multi-personalized image editing network by using the positioning loss to obtain the trained multi-personalized image editing network includes: The multi-personalized image editing network is trained using the positioning loss, the diffusion loss, and the category prior loss to obtain a trained multi-personalized image editing network.

[0015] In combination with the first aspect, in another possible implementation manner, the positioning loss is expressed as:

[0016] in, represents the positioning loss, m represents the number of edit masks, represents the cross attention map of the i-th reference image; Mj represents the j-th edit mask at the specified perspective, and the edit mask Mj , constraining the j-th cross attention map Infinitely close to the jth edit mask , , mean operation represents mean calculation; The expression of the diffusion loss is:

[0017] in, is the diffusion function, for Implicit encoding of moments, is the multimodal embedding information, ε is Gaussian noise, θ is a parameter, and E(x) is a variational autoencoder; The expression of the category prior loss is:

[0018] in, represents the category prior loss, Represents category prior textual cues.

[0019] In combination with the first aspect, in another possible implementation manner, after obtaining the single-view edited image, the method further includes: Construct a multimodal 3D model editing network; Performing three-dimensional editing using a multi-modality-based three-dimensional editing method to obtain an edited three-dimensional model; Changing the input multi-modal editing prompt information, stylizing the edited three-dimensional model, changing the original style, and obtaining a three-dimensional model of a target style through iterative training using a trained multi-personalized image editing network; Construct a three-dimensional model editing platform, which includes an editable user interface for assisting users in completing editing operations on the user interface and displaying one or more of the following functions: source three-dimensional model input, multimodal editing prompt input, three-dimensional editing box specification, three-dimensional model editing and result display functions.

[0020] In combination with the first aspect, in another possible implementation, the performing three-dimensional editing using a multi-modality-based three-dimensional editing method to obtain an edited three-dimensional model further includes: Obtaining a three-dimensional Gaussian representation of the edited three-dimensional model; Initializing the three-dimensional Gaussian representation of the edited three-dimensional model to obtain an initialized edited three-dimensional model; Rendering is performed using the initialized edited three-dimensional model to obtain the rendered single-view edited image; The rendered single-view edited image is input into the multi-personalized image editing network for denoising to obtain a clean single-view image.

[0021] In combination with the first aspect, in yet another possible implementation, during the process of denoising the rendered single-view edited image, the process further includes: Obtain the constraints of the multi-conditional loss function and the background reconstruction loss; In the denoising process, the constraints of the multi-conditional loss function and the background reconstruction loss are used to constrain and obtain the clean single-view image; Among them, the constraints of the multi-conditional loss function are:

[0022] The background reconstruction loss is:

[0023] in, is a multi-conditional loss function, 1, 2, 3 is the input text prompt, is the image rendering pose parameter, for Implicit encoding of moments, Editing networks for multimodal 3D models, is the true value of the noise, Express expectations, represents the weight, is the background reconstruction loss, The image is rendered from the source 3D model. is the image rendered by the edited 3D model, M is the edit mask projected by the 3D edit box specified by the user, is a logical operator, are the parameters of the 3D model, image I e Generated from 3D model rendering.

[0024] Optionally, the above 3D model is an edited three-dimensional model.

[0025] In a second aspect, the present invention further provides a multimodal 3D model editing device, the device comprising: An acquisition module, used for acquiring a personalized image generation method and a source three-dimensional model of multi-modal fusion; A generation module, used to obtain a personalized image generation network corresponding to the source three-dimensional model based on the multimodal fusion personalized image generation method, and train the personalized image generation network; A processing module, configured to obtain multimodal embedding information by fusing a given reference image list and a text description corresponding to the reference image list; The generation module is further used to add multiple personalized learning modules using the trained personalized image generation network as a skeleton to obtain a multiple personalized image editing network; The editing module is used to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information using the multi-personalized image editing network to obtain a single-view edited image.

[0026] In a third aspect, the present invention provides an electronic device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the multimodal three-dimensional model editing method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0027] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the multimodal-based three-dimensional model editing method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0028] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions for causing a computer to execute the multimodal three-dimensional model editing method of the first aspect or any corresponding embodiment thereof.

[0029] In addition, the present invention also provides a computer program product, including computer instructions, which are used to enable a computer to execute the multimodal three-dimensional model editing method of the above-mentioned first aspect or any corresponding embodiment thereof.

[0030] The present invention provides a multi-modal 3D model editing method, device, equipment and medium, which allows users to input prompt information of multiple modes such as text, image, sketch, etc., and explores the complementarity of prompts of each mode, thereby improving the controllability of editing and realizing high-quality 3D model editing. In addition, it allows users to perform various editing operations, such as object insertion, object replacement, etc., to realize functional editing; at the same time, it allows users to edit multiple objects at a time to realize controllable editing of multiple objects.

[0031] The method provided by the present invention combines the attention-based local positioning loss to realize multi-personalized image editing. During reasoning, a forward process can complete personalized image editing of multiple objects. In addition, the proposed multi-personalized image editing method is applied to realize single-view image editing, and SDS and local reconstruction loss are used for iterative optimization to realize three-dimensional model editing. In addition, a multi-functional three-dimensional editing method is proposed, which can realize various editing such as object insertion and object replacement. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a schematic diagram of a controllable three-dimensional model editing reasoning based on multi-modality provided according to an embodiment of the present invention; Figure 2 is a flowchart of a multi-modal 3D model editing method provided according to an embodiment of the present invention; Figure 3is a flowchart of a method for generating personalized images based on multimodal fusion images according to an embodiment of the present invention; Figure 4 is a flowchart of a training image personalized generation network provided according to an embodiment of the present invention; Figure 5 is a flowchart of another multi-modal 3D model editing method provided according to an embodiment of the present invention; Figure 6 is a flowchart of a multi-personalized image editing method provided according to an embodiment of the present invention; Figure 7 is a flow chart of another multi-modal 3D model editing method provided according to an embodiment of the present invention; Figure 8 is a flowchart of another multi-modal based 3D model editing method provided according to an embodiment of the present invention; Fig. 9 is a schematic diagram showing a Web-based three-dimensional model editing platform provided according to an embodiment of the present invention; Fig.10 is a structural block diagram of a three-dimensional model editing device provided according to an embodiment of the present invention; Fig.11 It is a schematic diagram of the hardware structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0035] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0036] Furthermore, the terms “first” and “second” are used for descriptive purposes only and should not be understood as indicating or implying relative importance.

[0037] The technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0038] In order to solve the inherent limitations of text description in image editing and the lack of precise control over the specified appearance and position of the editing results, an embodiment of the present invention proposes a controllable three-dimensional model editing method based on multi-modality, allowing users to input prompt information of multiple modalities such as text, images, sketches, etc., and explore the complementarity of prompts of each modality, thereby improving the controllability of editing and achieving high-quality three-dimensional model editing.

[0039] In addition, users are allowed to perform various editing operations, such as object insertion, object replacement, etc., to realize functional editing; at the same time, users are allowed to edit multiple objects at a time to realize controllable editing of multiple objects.

[0040] Specifically, the 3D model editing method follows the process of completing image editing first and then upgrading to 3D to complete 3D editing. In the training stage: First, given the source 3D model (i.e., the 3D model to be edited) and its corresponding text prompt, a personalized image generation method based on multimodal fusion is used to learn the information of the source 3D model and obtain the personalized image generation network corresponding to the source 3D model. On this basis, according to the input multiple reference images or sketches and the corresponding text prompts, the multi-personalized image editing method is used to edit the image rendered by the source 3D model. Obtain a multi-personalized edited image of a single view. Finally, based on the multimodal 3D model editing method, the single-view image is edited in an iterative manner using the trained multi-personalized image editing network, and is used to update the 3D model, and the edited 3D model is obtained after multiple iterations.

[0041] In the inference stage, Figure 1 As shown, given a source 3D scene model, according to the input multimodal prompt, the source 3D scene model is edited using a trained multimodal 3D scene editing network to obtain a 3D scene model that meets the given prompt requirements.

[0042] The following is a detailed description of the multimodal 3D model editing method provided by an embodiment of the present invention.

[0043] An embodiment of the present invention provides an embodiment of a three-dimensional model editing method based on multimodality. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0044] In this embodiment, a multi-modal 3D model editing method is provided, which can be used in terminal devices, clients, servers and other devices. Figure 2 is a flowchart of a multimodal 3D model editing method according to an embodiment of the present invention, such as Figure 2 As shown, the method includes: Step S101: obtaining a multi-modal fusion image personalized generation method and a source three-dimensional model.

[0045] Among them, multimodality refers to two or more forms of expression of a three-dimensional image, such as through input text, image, sketch, etc. It should be understood that in this embodiment, in addition to the above-mentioned input text, image, sketch Figure 3 In addition to this mode, other modes may also be included, and this embodiment does not limit this.

[0046] The source 3D model refers to a model with 3D data constructed by using 3D production software in a virtual 3D space. These models are usually used in computer and video games, as well as special effects production for movies and TV programs, and even in the fields of engineering design, architectural design, and product manufacturing. In this embodiment, the source 3D model refers to a 3D environment image captured in a real scene.

[0047] In addition, a multimodal fused image personalized generation method is used to generate a personalized image generation network, which is used to learn the source three-dimensional scene and construct a multimodal and multi-personalized image editable network.

[0048] Step S102: Based on the multimodal fusion-based personalized image generation method, a personalized image generation network corresponding to the source three-dimensional model is obtained, and the personalized image generation network is trained.

[0049] Among them, a personalized image generation network is trained to learn the characteristics of the source 3D model.

[0050] One implementation method is as follows Figure 3 As shown in the figure, in order to ensure that the non-editable area in the source 3D model (that is, the area that will not be edited in the 3D scene) remains unchanged during the editing process, an image personalization learning method based on multimodal fusion is proposed. This method is used to fine-tune the existing pre-trained image generation diffusion model to generate the image corresponding to the source 3D model. The specific steps are as follows: First, using the given source 3D model, render the image corresponding to the source 3D model. Given the source 3D model represented by a 3D Gaussian ,in , , Indicates the location of the center point, is the i-th three-dimensional Gaussian point, represents the covariance matrix, -1 represents the inverse matrix, and the color of each point is determined by Indicates that transparency , then the three-dimensional Gaussian expression is:

[0051] in, Represents the query point and the center point distance.

[0052] Therefore, the color of a pixel in the single-view image of the source 3D model obtained by rendering is:

[0053]

[0054] The above two formulas give the calculation process of the color of any pixel. If you render an image, you can render all the pixels on the current image one by one according to the above formulas.

[0055] Then, a multimodal prompt fusion algorithm is proposed to fuse multimodal prompt information, thereby fully exploring the complementarity of multimodal information. Among them, multimodal prompt information can be generated by fusion of text embedding and image embedding.

[0056] Step S103: Based on the given reference image list and the text description corresponding to the reference image list, multimodal embedding information is obtained after fusion.

[0057] The fused multimodal embedding information can be a unified prompt information. For example, the text description information and the reference image information are combined and input into N MLP layers and the self-attention layer of the Transformer to obtain unified prompt information, and finally obtain multimodal embedding information.

[0058] Optionally, the multimodal embedding information can be represented as “C f ”.

[0059] MLP (Multilayer Perceptron). MLP is a feedforward artificial neural network model that maps multiple input data sets to a single output data set. In a feedforward neural network, data flows from the input layer to the output layer, passing through one or more hidden layers in between. Transformer: A deep learning network based on self-attention and cross-attention mechanisms. Transformer is a deep learning network based on self-attention and cross-attention mechanisms. Transformer consists of two parts: an encoder and a decoder. Both the encoder and the decoder are stacked with multiple identical layers, each of which contains a self-attention mechanism and a feedforward neural network.

[0060] Step S104: using the trained personalized image generation network as a skeleton, adding multiple personalized learning modules to obtain a multiple personalized image editing network.

[0061] The purpose of this step is to build a multi-personalized image editing network. One implementation method is that the multi-personalized image editing network uses the trained image personalization generation model of "step S102" as the backbone network, adds a multi-personalized learning module, and then generates it. Among them, the multi-personalized learning module is a LoRA layer added to the cross attention layer in the U-Net network.

[0062] The LoRA layer (Low-Rank Adaptation of Large Language Models) is a low-rank adaptation technique for fine-tuning large language models. In the application of image generation diffusion models, LoRA is used as a plug-in that allows users to train models with specific painting styles or character features using a small amount of data without modifying the source model.

[0063] Step S105: using a multi-personalized image editing network, the source three-dimensional scene image in the source three-dimensional model is edited according to the multi-modal embedding information to obtain a single-view edited image.

[0064] According to the input of multiple reference images or reference sketches and corresponding text prompts, a multi-personalized image editing network is used to edit the image rendered by the source three-dimensional model, so as to obtain a single-view multi-personalized edited image.

[0065] If a reference sketch of a three-dimensional scene is received, a conditional image generation model, such as ControlNet, is used to first convert the reference sketch into a reference image, and then step S105 is executed.

[0066] The method provided in this embodiment first obtains a personalized image generation network corresponding to a source three-dimensional model based on a personalized image generation method of multimodal fusion, and trains the personalized image generation network. Then, based on a given reference image and a text description corresponding to the reference image, multimodal embedding information is obtained after fusion. The multimodal embedding information allows a user to input prompt information of multiple modalities such as text, image, sketch, etc., and explores the complementarity of prompts of each modality, thereby improving the controllability of editing and achieving high-quality three-dimensional model editing.

[0067] In a possible implementation of this embodiment, as Figure 4 As shown, the above step S102, training the personalized image generation network, specifically includes: Step S201: extracting text embedding and image embedding of the source 3D model.

[0068] Step S202: Fuse the text embedding and the image embedding according to the multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and calculate the diffusion loss based on the multimodal fusion embedding.

[0069] Step S203: training the image personalized generation network using diffusion loss and category loss to obtain a trained image personalized generation network.

[0070] Specifically, the method of obtaining text embedding is as follows: Figure 5 As shown, including the following: First, based on the source 3D model, obtain the text description corresponding to the source 3D model, for example, a table "atable". Then, construct the text prompt Ts based on the text description. Given the text description "atable" corresponding to the source 3D model, construct the text prompt , or {A, V1, table}, and use the pre-trained CLIP text encoder Encode the text prompt Ts as a text embedding.

[0071] Among them, CLIP (Contrastive Language-Image Pre-Training, a text-image pre-training model using contrastive learning) can map images and texts into a shared vector space, so that it can be used for various vision and language tasks.

[0072] like Figure 5 As shown, a method for obtaining image embedding includes: firstly rendering the source 3D model using an image rendering method to obtain an image set, wherein the image set includes at least one image. Then, using the pre-trained CLIP text encoder Encode textual cues as text embeddings.

[0073] Furthermore, the image embedding method is as follows: given the source 3D model and the above multimodal fusion image personalized generation method, the image set obtained by rendering in different postures ,in, is the number of source images, using the pre-trained CLIP image encoder Encode images into image embeddings.

[0074] During training, a random image from the image set is selected and its image embedding is compared with the text prompt. The feature vectors corresponding to the text embedding of " are connected together and integrated into The MLP layer and the Transformer self-attention layer generate unified prompt information, and finally obtain multimodal fusion embedding , the specific calculation is as follows:

[0075] in, , is multimodal fusion embedding, is a text prompt, is the target image, For CLIP text encoder, is the CLIP image encoder, MLP is the multilayer perceptron, and ATT is the self-attention layer.

[0076] The above formula can fully integrate multimodal embedding through the MLP layer and the self-attention ATT layer. In particular, the self-attention layer can adaptively integrate the input image embedding with the text embedding. Establish a stronger connection with the table to better integrate the two modes of image and text.

[0077] Then, a personalized image generation network based on multimodal fusion is constructed and trained to generate an image corresponding to the source three-dimensional model. A specific implementation method is: Obtain diffusion models, such as DreamBooth, which is a personalized text-to-image diffusion model. It is a method for fine-tuning text-to-image diffusion models. Its biggest feature is Subject-Driven (topic-driven generation), that is, for a specific subject, it generates context images of the subject in different scenes, postures, and perspectives.

[0078] Using the diffusion model (DreamBooth) as the skeleton, the CLIP image encoder is added to construct a personalized image generation network. The structure of the personalized image generation network can be composed of: variational autoencoder , U-Net network, text encoder and image encoder It consists of at least one module or level.

[0079] Using a variational autoencoder E, the image is compressed into a smaller latent representation z, which is then perturbed by Gaussian noise ε in the forward diffusion process and denoised by predicting the noise using a U-Net parameterized by θ. This denoising process can be embedded in a multimodal way through a cross-attention mechanism As the condition. Among them, the specific calculation of diffusion loss is:

[0080] in, for Implicit encoding of moments, is the diffusion loss, z is the potential representation, Express expectations.

[0081] During the training process, in order to prevent category forgetting, a category loss function is added during the training of the image personalized generation network. The following formula is an expression of a category loss function:

[0082] in, is the category loss, A priori text hints for categories, such as “table”, are used to distinguish different tables.

[0083] The image personalized generation network is trained together with the category prior loss and the diffusion loss to obtain the trained image personalized generation network.

[0084] During the inference process, the input text description "a table", an image describing the source 3D model can be generated, that is, the trained image personalized generation model network can recognize a specific source 3D model and ensure that the non-editing area remains unchanged during the subsequent editing process.

[0085] After the above-mentioned construction and training of the image personalized generation model network, it also includes: using a multi-personalized image editing method, according to the multi-modal editing prompt, editing the source three-dimensional scene image, and obtaining a single-view edited image (ie, a 2D image).

[0086] like Figure 6 As shown, it specifically includes: integrating multimodal reference prompt information as a denoising condition for a multi-personalized image editing network. In the above step S103, given a reference image and a corresponding text description, taking two reference images as an example, an image list is constructed. , an example of a text prompt is as follows: "a"," ” ,”cake”,“and”,” ”,“apple”,”on”,” "table"}.

[0087] Given a reference image, obtain a multi-view reference image. In order for the multi-personalized image editing network to learn the multi-view information of the reference image, thereby improving the multi-view consistency of 3D model editing, for any given reference image, a pre-trained multi-view image generation model, such as MVDream, is used to generate a corresponding multi-view reference image set or reference image list.

[0088] Specifically, with reference image As an example, it is input into the pre-trained multi-view image generation model MVDream to obtain the corresponding reference image set , where M is the number of multi-view images in the reference image set.

[0089] Integrate multi-modal reference prompt information to build denoising conditions for multi-personalized image editing networks. During the training process, the reference image list For any reference image in , any view image in the reference image is randomly selected, and the selected image is used as the target image. Correspondingly, the embedding information of the target image is called target image embedding.

[0090] The selected target image and its corresponding text prompt are sent to the multimodal prompt fusion network in step S102 to obtain the fused multimodal embedding as the denoising condition of the multi-personalized image editing network.

[0091] Specifically, the text descriptor is fed into the text encoder to obtain text embedding, and then the reference image and The corresponding reference image of a certain perspective, such as the target image, is embedded in the target image through the image encoder and respectively and The corresponding text embedding connection is sent to multiple MLP layers and Transformer self-attention layers to finally obtain multimodal embedding information , the specific calculation is as follows:

[0092] in, is multimodal fusion embedding, or target image embedding, is a text prompt, and is the reference image, For CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0093] Construct a multi-personalized image editing network and train it using the proposed positioning loss. This process is similar to the method of constructing an image personalized generation network in the aforementioned embodiment. The multi-personalized image editing network uses the image personalized generation network trained in step S103 as the skeleton network, and adds a multi-personalized learning module. The multi-personalized learning module is a LoRA layer added to the cross-attention layer in the U-Net network. Corresponding to the aforementioned step S104.

[0094] During the training process, the weights of the image personalized generation network trained in step S102 are kept unchanged, and the proposed positioning loss is used to train the multi-personalized learning module.

[0095] Specifically, a positioning loss is constructed to train a multi-personalized image editing network. In order to enable the multi-personalized image editing network to distinguish different reference images and text prompts, a "positioning loss" is proposed, and the LoRA layer is locally positioned using the user-specified three-dimensional editing box to train the multi-personalized learning module. Ideally, different cross-attention maps should only focus on the corresponding reference image, rather than spreading across the entire image, to prevent confusion between multiple reference images. To achieve this goal, this embodiment proposes to use a user-specified three-dimensional bounding box to locally position the cross-attention map.

[0096] Specifically, the cross-attention layer outputs a cross-attention map , connects the latent pixels to the denoising conditional embedding at each layer, where represents the correlation from the kth conditional embedding to the (i, j)th potential pixel, The 3D region to be edited is specified by the user on the source 3D model and is represented by a 3D bounding box. , and Respectively represent the reference images and The corresponding editing area is projected to the specified viewing angle according to the given 3D bounding box during training to obtain the editing mask area at the specified viewing angle. , I A list of indices indicating which reference image each word in the text prompt corresponds to, represents the cross attention map of the i-th reference image, constraining the j-th cross attention map Infinitely close to the jth edit mask ,Right now , using L1 loss to minimize the distance between the cross attention map and the edit mask:

[0097] in, represents the positioning loss, m represents the number of edit masks, and mean operation represents the mean calculation.

[0098] In practice, a localization loss is applied to the up- and down-sampled cross-attention maps in the U-Net network structure, which are known to contain more semantic information. The localization loss enables the model to accurately assign attention to the reference image at test time, thus preventing confusion between multiple personalized reference images.

[0099] In addition, the positioning loss is also used in this embodiment , diffusion loss and category prior loss Train multiple personalized image editing networks together. An expression of diffusion loss is:

[0100] in, is the diffusion function, for Implicit encoding of moments, is the multimodal embedding information, ε is Gaussian noise, θ is a parameter, and E(x) is a variational autoencoder.

[0101] During the training process, in order to prevent category forgetting, category loss can also be added as follows:

[0102] in, Prior textual hints for categories, such as “a cake and an apple on a table", which translates to "a cake V2 and an apple V3 are placed on the table V1".

[0103] The method further includes: constructing a sketch-based reference image multimodal fusion method. A possible implementation method is: inputting a reference image in the form of a sketch, using the conditional image generation model ControlNet to convert the reference sketch into a reference image, and then performing the above operation.

[0104] The above step S105 specifically includes: based on the rendered image of the source three-dimensional scene and the corresponding multimodal editing prompt information, using the trained multi-personalized image editing network to perform editing and generate a single-view editing image that meets the editing prompt.

[0105] Specifically, given a rendered image of a source 3D scene and a corresponding multimodal editing prompt (prompt information C f ), using the trained multi-personalized image editing network, we can obtain single-view edited images that meet the editing prompts.

[0106] like Figure 7 As shown, based on the source 3D editing model, after text prompt 1: cake V2 and apple V3 are placed on table V1. Image prompt 1 includes an image / photo of a cake and an apple. After multi-personalized image editing processing, a single-view edited image is output.

[0107] The method provided in this embodiment also includes: using a multi-modal 3D model editing network to complete 3D scene editing that meets the multi-modal editing prompt requirements. Specifically, Figure 8 As shown, after the single-view edited image is obtained in the above step S105, the following further steps are included: Step S301: construct a multimodal 3D model editing network.

[0108] Specifically, based on the multimodal 3D editing model, a multi-personalized image editing network is used as the backbone network, and the pre-trained network parameters in the above steps are kept unchanged. The edited 3D model is represented by a 3D Gaussian. ,in , , Indicates the location of the center point, Represents the covariance matrix, and the color of each point is determined by Indicates that transparency , then the three-dimensional Gaussian representation of the edited three-dimensional model is:

[0109] in, Represents the query point and the center point distance.

[0110] The edited 3D model is initialized with a 3D Gaussian representation of the source 3D model.

[0111] Step S302: Perform three-dimensional editing using a multi-modal based three-dimensional editing method to obtain an edited three-dimensional model.

[0112] Step S303: changing the input multimodal editing prompt information, stylizing the edited three-dimensional model, changing the original style, and obtaining a three-dimensional model of the target style through iterative training using the trained multi-personalized image editing network.

[0113] Step S304: constructing a three-dimensional model editing platform, which includes an editable user interface for assisting the user to complete editing operations on the user interface.

[0114] In addition, the 3D model editing platform can also display one or more of the following functions: source 3D model input, multi-modal editing prompt input, 3D editing box specification, 3D model editing and result display functions.

[0115] Furthermore, step S302 specifically includes: obtaining a three-dimensional Gaussian representation of the edited three-dimensional model; initializing the three-dimensional Gaussian representation of the edited three-dimensional model to obtain an initialized edited three-dimensional model; rendering using the initialized edited three-dimensional model to obtain a rendered single-view edited image; inputting the rendered single-view edited image into a multi-personalized image editing network for denoising to obtain a clean single-view image.

[0116] Specifically, during the training process, according to the three-dimensional model, a rendering method based on Gaussian representation is used to obtain a rendered image, and the rendered image is input into a multi-personalized image editing network to obtain an edited single-view edited image; the multi-condition loss and background reconstruction loss are calculated, and the gradient is returned to the three-dimensional Gaussian representation of the edited three-dimensional model, and the training process is iteratively repeated until the preset number of training times is reached.

[0117] During the reasoning process, given the source 3D model and the corresponding multimodal editing prompts, the trained multimodal-based 3D model editing method is used to obtain the edited 3D model.

[0118] Among them, a constraint expression of multi-condition loss is:

[0119] in, Multi-conditional loss function, For input text prompt, is the image rendering pose parameter, is the rendering function of the three-dimensional Gaussian representation, Editing networks for multimodal 3D models, Represents optimizing the 3D Gaussian part contained in the 3D edit box specified by the user.

[0120] According to step S302, the edited three-dimensional model is used to render a single-view image The color of a pixel is:

[0121]

[0122] Get the rendered single-view image '.

[0123] In addition, during the training phase, it also includes: constructing a loss function based on background reconstruction loss and multi-conditional SDS (ScoreDistillation Sampling) loss, and training a multi-modal 3D editing model. During the training process, the weights of the multi-personalized image editing network are kept unchanged, and the reconstruction loss and SDS loss are used to optimize the 3D Gaussian representation of the edited 3D model. Specifically, the single-view image rendered by the edited 3D model is Input into the multi-personalized image editing model (network) for denoising, and the rendered single-view image 'Input into the multi-personalized image editing model (network) for denoising.

[0124] In this embodiment, the input text prompt , text prompt , Text prompt , three SDS losses will be generated, and the final multi-conditional SDS loss is:

[0125] At the same time, background reconstruction loss is added to ensure that the non-edited area remains unchanged. The reconstruction loss is:

[0126] in, is a multi-conditional loss function, 1, 2, 3 is the input text prompt, is the image rendering pose parameter, for Implicit encoding of time Editing networks for multimodal 3D models, is the true value of the noise, Express expectations, represents the weight, is the background reconstruction loss, The image is rendered from the source 3D model. is the image rendered by the edited 3D model, M is the edit mask projected by the 3D edit box specified by the user, is a logical operator. In logic, ⊙ usually represents the XNOR operation. are the parameters of the 3D model, image I e Generated from 3D model rendering.

[0127] Optionally, the above 3D model is an edited three-dimensional model.

[0128] During the training process, for the current edited 3D model, a rendering method based on Gaussian representation is used to obtain a rendered image, which is then input into a multi-personalized image editing network to obtain an edited single-view image and calculate the SDS loss. and background reconstruction loss , the gradient backpropagation optimizes the Gaussian representation of the edited 3D model, and the above process is iteratively repeated until a certain number of training times is reached.

[0129] During the reasoning process, given the source 3D model and the corresponding multimodal editing prompts, the edited 3D model can be obtained by using the trained multimodal-based 3D model editing method.

[0130] Among them, SDS loss, namely fractional distillation sampling loss, is a loss function used to optimize deep learning models. It is often used in image generation tasks, especially in Neural Radiance Fields (NeRF) and related technologies. SDS loss measures the difference between generated images and real images in a specific way, thereby guiding the model training process.

[0131] In this embodiment, the above method also includes: changing the input multi-modal editing prompt to complete multi-functional editing.

[0132] The above method can complete the addition and editing of the existing 3D model. By changing the added multi-modal editing prompt, the stylized editing of the existing 3D model can be completed to change its style.

[0133] For example, if you want to change the current 3D model to Van Gogh style, keep steps S101 to S102 unchanged. In step S103, given a reference image in Van Gogh style, the corresponding text description “a photo of table of style", and the user-specified 3D bounding box specifies the editing position, and retrains the multi-personalized image editing network, so that the source 3D model rendering image can be edited to become the Van Gogh style. The model in step S103 is replaced with the trained multi-personalized image editing network, and after iterative training, a Van Gogh style 3D model can be obtained.

[0134] like Fig. 9 As shown, the method provided in this embodiment also includes: constructing a Web-based 3D model editing platform, providing an operable UI interface, and assisting users in completing editing. To assist users in completing editing, a Web-based 3D model editing platform was built, providing an operable UI interface (such as Fig. 9), the main functions include: source 3D model input, multi-modal editing prompt input, 3D editing box specification, 3D model editing, and result display function. These functions can correspond to the following prompt words or prompt information: please enter the source 3D model and the corresponding text prompt, please enter the 3D editing box, please enter the multi-modal editing prompt, 3D model editing, etc.

[0135] Specifically, a user interaction process includes: When it is detected that the user clicks the control "Please enter the source 3D model and the corresponding text prompt", a 3D model and text prompt input box will pop up; after entering the text description, the background will automatically perform image personalization generation network training. After the training is completed, a prompt will pop up saying "Training completed, please continue to enter the 3D bounding box and multimodal editing prompt".

[0136] When it is detected that the user clicks the "Please enter the 3D bounding box" control, manually mark the editing position that needs to be added. Click "Please enter the reference image" and "Please enter the text prompt" of "Please enter the multi-modal editing prompt", enter the reference image and text prompt respectively, and the background will automatically train the multi-personalized image generation network. After the training is completed, a prompt "Training is completed, please click 3D model editing" will pop up. When it is detected that the user clicks "3D model editing", the background will automatically train the multi-modal based 3D model editing and output the edited 3D model.

[0137] The present invention provides a multi-modal 3D model editing method, which allows users to input prompt information of multiple modes such as text, image, sketch, etc., and explores the complementarity of prompts of each mode, thereby improving the controllability of editing and realizing high-quality 3D model editing. In addition, the user is allowed to perform various editing operations, such as object insertion, object replacement, etc., to realize functional editing; at the same time, the user is allowed to edit multiple objects at a time to realize controllable editing of multiple objects.

[0138] In addition, this method also fine-tunes the image generation diffusion model to output images that conform to multiple modal descriptions, improving the controllability of image generation and improving the consistency of non-editable areas during the editing process.

[0139] The embodiment of the present invention provides a multi-personalized image editing method, which applies the proposed multi-modal fusion method and combines it with the attention-based local positioning loss to achieve multi-personalized image editing. During reasoning, a forward process can complete personalized image editing of multiple objects.

[0140] In addition, the multimodal 3D model editing method of this embodiment realizes single-view image editing, and uses SDS and local reconstruction loss for iterative optimization to realize 3D model editing. In addition, a multifunctional 3D editing method is proposed, which can realize various editing such as object insertion and object replacement.

[0141] It should be noted that the method proposed in this embodiment can be used to efficiently edit existing 3D assets, thereby obtaining more diverse 3D assets and saving the modeling time and economic cost of 3D assets. It can also be used to support the product development of Metaverse servers, such as MetaEngine and AI servers, and their applications in the field of 3D content generation and editing.

[0142] The multimodal 3D model editing method proposed in the embodiment of the present invention can be added as a plug-in to existing 3D modeling software, thereby improving the efficiency and quality of modeling.

[0143] In this embodiment, a three-dimensional model editing device based on multimodality is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the device described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0144] This embodiment provides a three-dimensional model editing device, such as Fig.10 As shown, the device includes: an acquisition module 110, a generation module 120, a processing module 130 and an editing module 140. In addition, the device may also include other more or fewer modules, such as a transceiver module, a storage module, etc., which is not limited in this embodiment.

[0145] The acquisition module 110 is used to acquire a personalized image generation method and a source three-dimensional model of multi-modal fusion.

[0146] The generation module 120 is used to obtain a personalized image generation network corresponding to the source three-dimensional model based on a personalized image generation method of multimodal fusion, and train the personalized image generation network.

[0147] The processing module 130 is used to obtain multimodal embedding information based on a given reference image list and a text description corresponding to the reference image list after fusion.

[0148] The generation module 120 is also used to add multiple personalized learning modules using the trained personalized image generation network as a skeleton to obtain a multiple personalized image editing network.

[0149] The editing module 140 is used to edit the source 3D scene image in the source 3D model according to the multi-modal embedding information using a multi-personalized image editing network to obtain a single-view edited image.

[0150] In some optional embodiments, the generation module 120 is specifically used to extract text embedding and image embedding of the source three-dimensional model; fuse the text embedding and image embedding according to the multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and calculate the diffusion loss based on the multimodal fusion embedding; use the diffusion loss and category loss to train the image personalized generation network to obtain the trained image personalized generation network.

[0151] In other optional embodiments, the generation module 120 is further specifically used to obtain a text description corresponding to the source three-dimensional model based on the source three-dimensional model; construct a text prompt based on the text description; encode the text prompt using a pre-trained text encoder to generate a text embedding; render the source three-dimensional model using an image rendering method to obtain an image set, wherein the image set includes at least one image; encode at least one image using a pre-trained image encoder to generate an image embedding.

[0152] In some further optional embodiments, the generation module 120 is further used to connect the target image embedding with the text embedding, and input it into the self-attention layer of N multi-layer perceptrons and transformers, and output a multimodal fusion embedding.

[0153] The expression of multimodal fusion embedding is:

[0154] in, is multimodal fusion embedding, is a text prompt, is the target image, For CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

[0155] In some optional implementations, the acquisition module 110 is further used to acquire a diffusion model; the generation module 120 is further used to use the diffusion model as a skeleton, add a CLIP image encoder, and construct a personalized generation network for generating images.

[0156] Among them, the image personalized generation network includes: variational autoencoder, U-Net network, text encoder and image encoder One or more of .

[0157] In some other optional embodiments, the processing module 130 is specifically used to obtain a reference sketch and convert the reference sketch into a reference image, fuse multimodal reference prompt information based on a reference image list and a corresponding text description, and use the multimodal reference prompt information as a denoising condition for a multi-personalized image editing network.

[0158] The generation module 120 is also used to calculate the corresponding diffusion loss according to the denoising condition after obtaining the multi-personalized image editing network; use the diffusion loss to train the multi-personalized image editing network to obtain the trained multi-personalized image editing network; and construct a sketch-based reference image multimodal fusion method.

[0159] The editing module 140 is specifically used to perform editing based on the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt information using the trained multi-personalized image editing network to generate a single-view editing image that meets the editing prompts.

[0160] In some other optional embodiments, the acquisition module 110 is further used to acquire any reference image in the reference image list and select an image of any one perspective in the multi-perspective image set; the generation module 120 is further used to input the selected perspective image and the text prompt into the multi-modal prompt fusion network to obtain the fused multi-modal embedding information as a denoising condition of the multi-personalized image editing network; The acquisition module 110 is further used to acquire the positioning loss and the category prior loss before the generation module 120 trains the multi-personalized image editing network using the diffusion loss.

[0161] The generation module 120 is further used to train the multi-personalized image editing network using the positioning loss, the diffusion loss and the category prior loss to obtain the trained multi-personalized image editing network.

[0162] In some other optional embodiments, after the single-view editing image is obtained by using the editing module 140, the processing module 130 is also used to construct a multimodal-based 3D model editing network; perform 3D editing using a multimodal-based 3D editing method to obtain an edited 3D model; change the input multimodal editing prompt information, perform stylized editing on the edited 3D model to change the original style, and use the trained multi-personalized image editing network to obtain a 3D model of the target style through iterative training; and construct a 3D model editing platform.

[0163] The device also includes a display module for displaying a 3D model editing platform. The 3D model editing platform includes an editable user interface for assisting the user to complete editing operations on the user interface and displaying one or more of the following functions: source 3D model input, multi-modal editing prompt input, 3D editing box designation, 3D model editing, and result display functions.

[0164] In some other optional embodiments, the editing module 140 is specifically used to obtain a rendered image using a Gaussian representation-based rendering method according to the three-dimensional model during the training process, and input the rendered image into a multi-personalized image editing network to obtain an edited single-view image.

[0165] The processing module 130 is also used to obtain a three-dimensional Gaussian representation of the edited three-dimensional model; initialize the three-dimensional Gaussian representation of the edited three-dimensional model to obtain an initialized edited three-dimensional model; use the initialized edited three-dimensional model for rendering to obtain a rendered single-view edited image; input the rendered single-view edited image into the multi-personalized image editing network for denoising to obtain a clean single-view image.

[0166] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0167] The three-dimensional model editing device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0168] The 3D model editing device provided in this embodiment allows the user to input prompt information of multiple modes such as text, image, sketch, etc., and explores the complementarity of prompts of each mode, thereby improving the controllability of editing and realizing high-quality 3D model editing. In addition, it also allows the user to perform various editing operations, such as object insertion, object replacement, etc., to realize functional editing; at the same time, it allows the user to edit multiple objects at a time, realizing controllable editing of multiple objects.

[0169] The embodiment of the present invention also provides an electronic device having the above Fig.11 The three-dimensional model editing device shown.

[0170] See also Fig.11 , is a schematic diagram of the structure of an electronic device provided by an optional embodiment of the present invention, the electronic device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses for communication, and can be installed on a common mainboard or in other ways as needed. The processor can process instructions executed in the electronic device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface).

[0171] In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides part of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Fig.11 A processor 10 is taken as an example.

[0172] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0173] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the multi-modal three-dimensional model editing method shown in the above embodiment.

[0174] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0175] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0176] The electronic device further includes an input device 30 and an output device 40. The processor 10, the memory 20, the input device 30 and the output device 40 may be connected via a bus or other means. Fig.11 The example of connecting through bus is taken in the following.

[0177] The input device 30 can receive input digital or character information, and generate signal input related to the user settings and function control of the electronic device, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, an indicator rod, one or more mouse buttons, a trackball, a joystick, etc. The output device 40 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The above-mentioned display device includes but is not limited to a liquid crystal display, a light emitting diode, a display and a plasma display. In some optional embodiments, the display device may be a touch screen.

[0178] In addition, the electronic device also includes at least one communication interface, which is used for the device to communicate with other devices or a communication network.

[0179] An embodiment of the present invention also provides a computer-readable storage medium, and the above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented by downloading through a network and originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware.

[0180] The storage medium may be a disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid state drive, etc.; further, the storage medium may also include a combination of the above-mentioned types of memories. It is understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the multimodal-based three-dimensional model editing method shown in the above embodiment is implemented.

[0181] Embodiments of the present application may also provide a computer program product, including computer program instructions, which, when executed by a processor, cause the processor to perform the steps in the above method. Wherein, the computer program product may be written in any combination of one or more programming languages ​​to perform program codes for performing the operations of the disclosed embodiments, wherein the programming languages ​​include object-oriented programming languages, such as Java, C++, etc., and also include conventional procedural programming languages, such as "C" language or similar programming languages. The program code may be executed entirely on a user computing device, partially on a user device, as an independent software package, partially on a user computing device, partially on a remote computing device, or entirely on a remote computing device or server.

[0182] The above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention are described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some of the technical features therein by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional model editing method based on multimodality, characterized in that: The method comprises: Obtaining personalized image generation methods and source 3D models of multimodal fusion; Based on the multimodal fusion image personalized generation method, obtaining a personalized image generation network corresponding to the source three-dimensional model, and training the personalized image generation network; Based on a given reference image list and a text description corresponding to the reference image list, multimodal embedding information is obtained after fusion; Taking the trained personalized image generation network as the skeleton, add multiple personalized learning modules to obtain a multiple personalized image editing network; The multi-personalized image editing network is used to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image.

2. The method according to claim 1, characterized in that The training of the personalized image generation network comprises: extracting text embeddings and image embeddings of the source three-dimensional model; The text embedding and the image embedding are fused according to a multimodal prompt fusion algorithm to obtain a multimodal fusion embedding, and the diffusion loss is calculated based on the multimodal fusion embedding; The image personalized generation network is trained using the diffusion loss and the category loss to obtain a trained image personalized generation network.

3. The method according to claim 2, characterized in that The extracting text embedding and image embedding of the source three-dimensional model comprises: According to the source three-dimensional model, obtaining a text description corresponding to the source three-dimensional model; constructing a text prompt according to the text description; Encoding the text prompt using a pre-trained text encoder to generate the text embedding; Rendering the source three-dimensional model using an image rendering method to obtain an image set, wherein the image set includes at least one image; The at least one image is encoded using a pre-trained image encoder to generate the image embedding.

4. The method according to claim 3, characterized in that If the generated image embedding is a target image embedding, fusing the text embedding and the image embedding according to a multimodal prompt fusion algorithm to obtain a multimodal fusion embedding includes: The target image embedding is connected with the text embedding and input into the self-attention layer of N multi-layer perceptrons and transformers, and the multimodal fusion embedding is output; the expression of the multimodal fusion embedding is: in, is multimodal fusion embedding, is a text prompt, is the target image, For CLIP text encoder, is the CLIP image encoder, MLP is the multi-layer perceptron, and ATT is the self-attention layer.

5. The method according to claim 4, characterized in that The method for generating personalized images based on the multimodal fusion, obtaining a personalized image generation network corresponding to the source three-dimensional model, includes: Get the diffusion model; The diffusion model is used as a skeleton, a CLIP image encoder is added, and the image personalized generation network is constructed and generated; The image personalized generation network includes: variational autoencoder, U-Net network, text encoder and image encoder .

6. The method according to claim 1, characterized in that The step of fusing a given reference image list and a text description corresponding to the reference image list to obtain multimodal embedding information includes: Obtaining a reference sketch, and converting the reference sketch into a reference image; According to the reference image and the corresponding text description, multimodal reference prompt information is obtained by fusing, and the multimodal reference prompt information is used as a denoising condition of a multi-personalized image editing network; After obtaining the multi-personalized image editing network, the method further includes: Calculate the corresponding diffusion loss according to the denoising condition; Using the diffusion loss to train the multi-personalized image editing network to obtain a trained multi-personalized image editing network; Using the multi-personalized image editing network, editing the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information to obtain a single-view edited image, including: According to the rendered image of the source three-dimensional scene and the corresponding multi-modal editing prompt information, the trained multi-personalized image editing network is used for editing to generate a single-view editing image that meets the editing prompt.

7. The method according to claim 6, characterized in that Using the multimodal reference prompt information as a denoising condition for a multi-personalized image editing network includes: Using the pre-trained multi-view image generation model, generate a corresponding multi-view reference image list; Obtain any reference image in the reference image list, and select a viewing angle image in the reference image; Inputting the selected perspective image and text prompt into the multimodal prompt fusion network to obtain fused multimodal embedding information as a denoising condition for the multi-personalized image editing network; Before the diffusion loss is used to train the multi-personalized image editing network, the method further includes: Obtain positioning loss and category prior loss; The method of training the multi-personalized image editing network by using the positioning loss to obtain the trained multi-personalized image editing network includes: The multi-personalized image editing network is trained using the positioning loss, the diffusion loss, and the category prior loss to obtain a trained multi-personalized image editing network.

8. The method according to claim 7, characterized in that The expression of the positioning loss is: in, represents the positioning loss, m represents the number of edit masks, represents the cross attention map of the i-th reference image; M j Indicates the jth edit mask at the specified perspective. , mean operation represents mean calculation; The expression of the diffusion loss is: in, is the diffusion function, for Implicit encoding of moments, is the multimodal embedding information, ε is Gaussian noise, θ is a parameter, E(x) is a variational autoencoder, For expectations; The expression of the category prior loss is: in, represents the category prior loss, Represents category prior textual cues.

9. The method according to any one of claims 1 to 8, characterized in that After obtaining the single-view edited image, the method further includes: Construct a multimodal 3D model editing network; Performing three-dimensional editing using a multi-modality-based three-dimensional editing method to obtain an edited three-dimensional model; Changing the input multi-modal editing prompt information, stylizing the edited three-dimensional model, changing the original style, and obtaining a three-dimensional model of a target style through iterative training using a trained multi-personalized image editing network; Construct a three-dimensional model editing platform, which includes an editable user interface for assisting users in completing editing operations on the user interface and displaying one or more of the following functions: source three-dimensional model input, multimodal editing prompt input, three-dimensional editing box specification, three-dimensional model editing and result display functions.

10. The method according to claim 9, characterized in that The method of performing three-dimensional editing by using a multi-modal three-dimensional editing method to obtain an edited three-dimensional model further includes: Obtaining a three-dimensional Gaussian representation of the edited three-dimensional model; Initializing the three-dimensional Gaussian representation of the edited three-dimensional model to obtain an initialized edited three-dimensional model; Rendering is performed using the initialized edited three-dimensional model to obtain the rendered single-view edited image; The rendered single-view edited image is input into the multi-personalized image editing network for denoising to obtain a clean single-view image.

11. The method according to claim 10, characterized in that The denoising process of the rendered single-view edited image also includes: Obtain the constraints of the multi-conditional loss function and the background reconstruction loss; In the denoising process, the constraints of the multi-conditional loss function and the background reconstruction loss are used to constrain and obtain the clean single-view image; Among them, the constraints of the multi-conditional loss function are: The background reconstruction loss is: in, is a multi-conditional loss function, 1, 2, 3 is the input text prompt, is the image rendering pose parameter, for Implicit encoding of moments, Editing networks for multimodal 3D models, is the true value of the noise, Express expectations, represents the weight, is the background reconstruction loss, The image is rendered from the source 3D model. is the image rendered by the edited 3D model, M is the edit mask projected by the 3D edit box specified by the user, Is a logical operator.

12. A three-dimensional model editing device based on multimodality, characterized in that: The device comprises: An acquisition module, used for acquiring a personalized image generation method and a source three-dimensional model of multi-modal fusion; A generation module, used to obtain a personalized image generation network corresponding to the source three-dimensional model based on the multimodal fusion personalized image generation method, and train the personalized image generation network; A processing module, configured to obtain multimodal embedding information by fusing a given reference image list and a text description corresponding to the reference image list; The generation module is further used to add multiple personalized learning modules using the trained personalized image generation network as a skeleton to obtain a multiple personalized image editing network; The editing module is used to edit the source three-dimensional scene image in the source three-dimensional model according to the multi-modal embedding information using the multi-personalized image editing network to obtain a single-view edited image.

13. An electronic device, characterized in that: comprising a memory and a processor, wherein the memory and the processor are connected; The memory stores computer instructions, and the processor executes the multimodal-based three-dimensional model editing method according to any one of claims 1 to 11 by executing the computer instructions.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multimodal-based three-dimensional model editing method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The invention comprises computer instructions, wherein the computer instructions are used to make a computer execute the multimodal based three-dimensional model editing method according to any one of claims 1 to 11.

Images