Multi-view conversion method based on multi-view consistency diffusion model

Abstract

The invention relates to the technical field of image conversion, and discloses a multi-view conversion method based on a multi-view consistency diffusion model, and the multi-view consistency diffusion model is trained through a unified multi-view-to-multi-view conversion framework based on the diffusion model. The conversion framework has a new self-supervised training strategy and a consistency and antagonism supervision strategy, and the strategies utilize an existing 2D single-view image converter and a 3D Gaussian splash technology to generate pseudo-truth values as supervision signals, so that view consistency and high-frequency details are enhanced; the multi-view consistency diffusion model trained by the conversion framework further comprises a potential multi-view consistency module, and the potential 3DGS module is used as a 3D representation of a bottom layer to promote information exchange across multiple views, so that 3D priori is introduced into the diffusion model to ensure view consistency. Finally, a joint optimization strategy is provided, and better balance between consistency and authenticity can be realized.

Description

Technical Field

[0001] The present invention relates to the field of image conversion technology, and in particular to a method based on multi-view Figure 1 A multi-view transformation method for consistent diffusion model. Background Art

[0002] Image translation is a long-standing problem in computer graphics and computer vision. Many problems can be considered as image-to-image translation. The purpose of image translation is to translate an image from one domain to another. It takes the input image as a condition and translates the image from the source domain to the target domain while preserving its content, including image super-resolution, deblurring, denoising, editing tasks, etc.

[0003] Due to the good generative ability of diffusion models, single-view image conversion methods based on diffusion models have recently achieved realistic results. However, most existing research focuses on single-view image conversion tasks, and there are still two major problems in directly applying diffusion models to multi-view image conversion tasks:

[0004] First: lack of multi-view training data;

[0005] Second: Multi-view Figure 1 Consistency is difficult to guarantee.

[0006] For image super-resolution tasks, DiffBIR and StableSR use the generative power of latent diffusion models to generate realistic high-resolution images. For image deblurring tasks, HiDiff implements diffusion models to generate prior features in latent space for deblurring and restores beautiful images. For image editing tasks, InstructPix2Pix uses large pre-trained models in language and image domains to generate diffusion models for training data pairs and training text editing.

[0007] Although these methods have produced good results for single-view image processing in their respective tasks, they have encountered difficulties when applying them to multi-view image conversion tasks. Due to the lack of multi-view training data for targeted training, these models cannot be well applied to multi-view conversion tasks. And simply performing frame-by-frame image conversion will cause multi-view image conversion errors. Figure 1 The problem of difficulty in ensuring consistency leads to inconsistencies in geometry and appearance between images from different perspectives. Summary of the invention

[0008] In order to solve the above technical problems, the present invention provides a multi-view Figure 1Through a unified multi-view to multi-view conversion framework based on diffusion model (Multi-View Image Translation via View-Consistent Diffusion Models, MV2MV), a multi-view Figure 1 View-Consistent Diffusion Models (VCDM).

[0009] First, the conversion framework has a new self-supervised training strategy, the consistency and adversarial supervision strategy, which utilizes the existing 2D single-view image converter and 3D Gaussian splashing (3DGS) to generate pseudo-ground-truth as a supervision signal to enhance the visual Figure 1 The first problem mentioned above is solved by optimizing the image quality and high-frequency details.

[0010] Second, the multi-view transform framework is trained Figure 1 The consistent diffusion model also includes potential multi-view Figure 1 The latent 3DGS module is used as the underlying 3D representation to promote information exchange across multiple views, thereby introducing 3D priors into the diffusion model to ensure that the view Figure 1 This solves the second problem mentioned above.

[0011] Finally, a joint optimization strategy is proposed, which can achieve a better trade-off between consistency and authenticity.

[0012] Under the above scheme, our method performs extensive experiments on various image conversion tasks and shows that MV2MV outperforms the state-of-the-art methods both quantitatively and qualitatively.

[0013] The present invention specifically adopts the following technical solutions:

[0014] A multi-view based Figure 1 The multi-view conversion method of the consistent diffusion model can complete the training of multi-view Figure 1 Image conversion based on consistent diffusion model; multi-view Figure 1 The training process of the consistent diffusion model specifically includes:

[0015] Step 1: Construct a multi-view image to multi-view image conversion framework based on the diffusion model. The conversion framework is used to train the multi-view image. Figure 1 Conformal diffusion model; multi-view Figure 1 The consistent diffusion model includes potential multi-view Figure 1 Consistency module, VAE encoder, first ControlNet module and denoising Unet network; potential multi-view Figure 1The consistency module includes a potential 3DGS module and a second ControlNet module;

[0016] Step 2: Convert the multi-view image to a single view to obtain a converted image; render the converted image through a 3D Gaussian splash model to obtain a rendered image; add noise to the rendered image to convert it into a noisy rendered image; input the multi-view image to the VAE encoder to obtain a conditional feature map, and input the conditional feature map and the noisy rendered image to the first ControlNet module; input the consistency feature map output by the potential 3DGS module and the noisy rendered image to the second ControlNet module; input the text prompt into the denoising Unet network after passing through the text encoder; Figure 1 The denoising Unet network output in the consistent diffusion model generates an image;

[0017] Step 3: The converted image and the rendered image are regarded as pseudo-true values, and the multi-view image is generated based on the discriminator and the adversarial loss function in the adversarial generative network. Figure 1 The consistent diffusion model is trained through supervision.

[0018] Furthermore, the denoising UNet network includes an encoder, a decoder and an intermediate layer, the encoder includes multiple encoding layers, and the decoder includes multiple decoding layers; the first ControlNet module and the second ControlNet module both include multiple encoding layers, multiple zero convolutions and intermediate layers; the number of encoding layers of the first ControlNet module and the second ControlNet module is the same as the number of encoding layers of the encoder, and the number of zero convolutions of the first ControlNet module and the second ControlNet module is the same as the number of decoding layers of the decoder; the latent 3DGS module is based on a 3D Gaussian splash algorithm and uses Gaussian spheres to store latent features.

[0019] Furthermore, in step 2, the multi-view image is converted into a single view to obtain a converted image, and the converted image is rendered using a 3D Gaussian splash model to obtain a rendered image, which specifically includes:

[0020] The collection of multi-view images is denoted as I i represents the i-th image, N represents the number of images, and the set of multi-view images is input into the 2D single-view image converter to generate a set of converted images h i is the i-th transformed image; the 3D Gaussian splash model is directly optimized using the transformed image to enforce a set of geometrically consistent rendered images x i Render the i-th image.

[0021] Furthermore, step three specifically includes:

[0022] A collection of rendered images consistent with a given geometry In training multi-view Figure 1 When using the consistent diffusion model, the generated image and x are directly minimized at each time step. i The difference between them is as follows: randomly select a time step t, add noise, and change x i Convert to a noisy rendered image The consistency loss function for:

[0023]

[0024] in, The generated image is output by the denoising UNet network according to the predicted noise at time step t; the consistency loss function makes the multi-view Figure 1 The consistent diffusion model can generate multi-view Figure 1 the results; The coefficient representing the duration t is used to control the noise level; ∈ θ (·) represents the denoising UNet network;

[0025] Convert the image As the true sample R, and the multi-view Figure 1 Generated image of the output of the consistent diffusion model As a fake sample F; supervise the multi-view through the discriminator and adversarial loss function in the adversarial generation network Figure 1 Conformal diffusion model.

[0026] Furthermore, the multi-view supervised learning is performed by the discriminator and the adversarial loss function in the adversarial generative network. Figure 1 The consistent diffusion model includes:

[0027] Multi-view Figure 1 The adversarial loss function between the consistent diffusion model and the discriminator is:

[0028]

[0029]

[0030] Represents the generation loss, used to train multi-view Figure 1 Conformal diffusion model; Represents the discriminative loss, which is used to train the discriminator; E R and E F Both represent expectations, and D(·) represents the discriminator.

[0031] Furthermore, in order to introduce geometric constraints to make multi-view Figure 1Generated image of the output of the consistent diffusion model With converted image With geometric similarity, the perceptual loss function is introduced For training multi-view Figure 1 Conformal diffusion model:

[0032]

[0033] Here, LPIPS(·,·) denotes learned perceptual patch similarity.

[0034] Furthermore, a joint optimization strategy is also included to simultaneously optimize multiple views. Figure 1 Conformal diffusion model and 3D Gaussian splash model, including:

[0035] Multi-view Figure 1 The generated image output by the consistent diffusion model is input into the denoising diffusion implicit model to obtain the optimized image Then, using the rendered image Through consistency loss Figure 1 The consistent diffusion model is supervised and optimized through the following 3DGS loss function Optimize 3D Gaussian splash model:

[0036]

[0037] λ represents the weight coefficient, and SSIM(·,·) refers to the structural similarity index.

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] 1. This paper proposes a new consistency and adversarial supervision strategy. It can effectively circumvent the requirement for real training data, and utilizes the prior knowledge of existing diffusion model-based single-view image converters that are well trained on large 2D image datasets. It generates pseudo-truths and guarantees multi-view accuracy through consistency loss and adversarial loss. Figure 1 Multi-view Figure 1 The problem of lack of multi-view training data is solved.

[0040] 2. The present invention proposes a potential multi-view Figure 1 Consistency module (LAConsistNet). It makes multi-view Figure 1 The consistent diffusion model can explicitly enforce multi-view geometric consistency. Compared with previous methods that construct 3D voxels as the underlying 3D representation, the method of the present invention proposes to use the latent 3DGS module to generate a consistent feature map and combine it with the multi-view Figure 1The method of the present invention is memory-friendly and efficiency-friendly, can handle large-scale scenes, and is suitable for different multi-view image conversion tasks. Figure 1 Problems with consistency

[0041] 3. The present invention proposes a multi-view Figure 1 The joint optimization strategy of the 3D Gaussian splash model and the uniform diffusion model is proposed. Figure 1 The consistent diffusion model can produce results with high-frequency details, but it cannot guarantee the consistency of these details. Figure 1 The uniform diffusion model and the 3D Gaussian splash model are used to Figure 1 Consistency attributes are used to obtain coherent details across views and transfer them to multi-view Figure 1 The consistent diffusion model ensures consistent generation of high-frequency details. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of MV2MV provided according to an embodiment of the present invention;

[0043] Figure 2 A flow chart of a consistency and adversarial supervision strategy provided according to an embodiment of the present invention;

[0044] Figure 3 Potential multi-view according to an embodiment of the present invention Figure 1 Consistency module structure diagram;

[0045] Figure 4 A flow chart of a joint optimization strategy provided according to an embodiment of the present invention;

[0046] Figure 5 A qualitative comparison diagram of an image super-resolution task provided according to an embodiment of the present invention;

[0047] Figure 6 A qualitative comparison diagram of text-driven editing tasks provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] A preferred embodiment of the present invention is described in detail below with reference to the accompanying drawings.

[0049] The structure of the multi-view to multi-view conversion framework based on the diffusion model of the present invention is as follows: Figure 1 As shown, the purpose of the conversion framework is to train multi-view Figure 1To circumvent the requirement for real training data, we first propose a new self-supervised training strategy, called consistency and adversarial supervision, which leverages the prior knowledge of existing single-view image converters that are well trained on large 2D image datasets. Specifically, we process multi-view images separately to generate high-quality converted images, which are then fed into a 3D Gaussian splatter model (3DGS) to obtain perspective-consistent rendered images. We regard the converted and rendered images as ground-truth and supervise the multi-view using consistency loss and adversarial loss. Figure 1 Secondly, the potential multi-view Figure 1 The consistency module uses the latent 3DGS module (Latent-3DGS) as the underlying 3D representation to achieve cross-view information exchange. Specifically, the consistency feature map output by the latent 3DGS module is used as the input of the ControlNet module, and the ControlNet module is combined with the multi-view Figure 1 The corresponding decoder layers of the denoising UNet network in the consistent diffusion model are combined to enhance consistency. Finally, the present invention proposes a joint optimization strategy to train multiple viewpoints at the same time. Figure 1 The consistent diffusion model and 3D Gaussian splash model are used to further enhance the consistency and realism of the generated images.

[0050] The VAE encoder, ControlNet module and denoising UNet network used in the present invention are prior arts. The innovation of the present invention is to use these technologies to construct a multi-view Figure 1 The consistent diffusion model does not involve improvements to the VAE encoder, ControlNet module, and denoising UNet network, so I will not go into details.

[0051] In addition, 3D Gaussian splash is an existing technology, which is a rasterization technology for describing 3D Gaussian distribution for real-time radiation field rendering.

[0052] The specific technical solutions of the present invention are introduced in several parts below.

[0053] 1. Consistency and Adversarial Supervision Strategies

[0054] 1.1 Pseudo-truth value generation:

[0055] In recent years, diffusion models have achieved success in 2D image conversion tasks due to their strong prior knowledge and appearance generation capabilities. Therefore, we propose a self-supervised framework that leverages the success of existing diffusion-based single-view image converters and 3D Gaussian splash models to generate pseudo-truth as a supervisory signal, circumventing the requirement for real training data. Although pseudo-truth can also be obtained using non-diffusion-based methods, in practice we prioritize methods that can generate high-quality and realistic texture details. We use several multi-view image conversion tasks to demonstrate the adaptability of MV2MV: super-resolution, denoising, deblurring, and text-driven editing.

[0056] Given an input multi-view image set The present invention first generates a set of converted images through an existing 2D single-view image converter. Next, the present invention uses these transformed images to directly optimize the 3D Gaussian splash model to enforce the production of a set of geometrically consistent rendered images Due to the generative nature of the image converter, Although the quality is good, there are obvious visual inconsistencies. Show good vision Figure 1 Then, the present invention focuses on obtaining the consistency in the visual Figure 1 The result is a good balance between consistency and high quality. Specifically, the present invention proposes consistency loss and adversarial loss to achieve these two optimization goals. The former ensures that the multi-view Figure 1 Multi-view Figure 1 The latter is used to recover high-frequency details.

[0057] 1.2 Consistency loss:

[0058] A set of geometrically consistent rendered images from multiple perspectives In training multi-view Figure 1 When the consistent diffusion model is used, the present invention directly minimizes the generated result and x at each time step. i Specifically, the present invention randomly selects a time step t and adds noise to the rendered image x. i Convert to a noisy rendered image Consistency loss for:

[0059]

[0060] in, is the image generated by the denoising UNet network based on the predicted noise at time step t. The consistency loss ensures that the multi-view Figure 1The consistent diffusion model can generate multi-view Figure 1 The result.

[0061] 1.3 Adversarial Loss:

[0062] because The geometric and appearance inconsistencies between them make them unsuitable for direct use in supervising the multi-view Figure 1 Therefore, the goal of the present invention is to restore high-frequency details by adopting an adversarial training strategy. Previous studies have shown that adversarial training is effective in preventing rendering blur caused by inconsistent viewpoint conflicts during supervision from different viewpoints.

[0063] Specifically, the present invention proposes to minimize the generation of multi-view Figure 1 Preliminary generation image of the consistent diffusion model Convert image with high quality h i The distribution difference between Figure 1 The present invention transforms the set of images into As the true sample R, and the multi-view Figure 1 A collection of generated images for the consistent diffusion model As a fake sample F. Multi-view supervision through the discriminator and adversarial loss in the adversarial generation network Figure 1 The consistent diffusion model. For the adversarial generative network, see “Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems 27 (2014).”

[0064] Multi-view Figure 1 The adversarial loss function between the consistent diffusion model and the discriminator is:

[0065]

[0066]

[0067] In addition, in order to introduce geometric constraints to promote multi-view Figure 1 Generate images of the consistent diffusion model With high quality converted image h i With geometric similarity, the present invention introduces the perceptual loss function

[0068]

[0069] Here, LPIPS(·,·) represents Learned Perceptual Image Patch Similarity (LPIPS).

[0070] The process of consistency and adversarial supervision strategies can be found in Figure 2 .

[0071] 2. Potential Multi-View Figure 1 Consistency Module

[0072] In order to multi-view Figure 1 Visualization in the consistent diffusion model Figure 1 Consistency, inspired by the ConsistNet module, this paper proposes a potential multi-view Figure 1 The consistency module injects 3D priors into the neural network block as additional conditions, such as Figure 3 Unlike the ConsistNet module that uses the View Aggregation and Ray Aggregation modules to aggregate multi-view information, the present invention uses the latent 3DGS module as the underlying 3D representation to model multi-view geometric representation, making multi-view Figure 1 The consistent diffusion model can complete the multi-view conversion task more simply and effectively. The latent 3DGS module is based on the 3D Gaussian splash algorithm, which uses Gaussian spheres to store latent features and is regarded as a trainable parameter.

[0073] Potential Multi-View Figure 1 The consistency module is built on the ControlNet module and the pre-trained Stable Diffusion. The 3D consistency feature map rendered by the latent 3DGS module is used as the input of the ControlNet module and inserted into each encoding layer of the denoising UNet network to enforce 3D consistency. Specifically, the denoising UNet network includes an encoder and a decoder, where the encoder and decoder both have 12 layers and 1 intermediate layer. Similar to the ControlNet module, the present invention creates trainable copies of 12 encoding layers and 1 intermediate layer, and uses a zero convolution layer (i.e., a 1×1 zero-initialized convolution layer) to append its output to each decoding layer of the denoising UNet network.

[0074] In order to achieve fast training on the pre-trained Stable Diffusion, the trainable structure of the ControlNet module is consistent with the literature "Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala." Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE / CVFInternational Conference on Computer Vision. 2023." In addition, the present invention uses a pre-trained 3D Gaussian splash model to initialize the potential 3DGS module as the initial 3D prior.

[0075] 3. Joint Optimization Strategy

[0076] In the framework of the present invention, a potential multi-view Figure 1 Consistency module and consistent loss to ensure multi-view Figure 1 Multi-view Figure 1 However, the generation of details is accompanied by randomness, which leads to variations in high-frequency details generated by multi-view images, which has a negative impact on training stability and visual quality. Figure 1 This prompts the present invention to propose a joint optimization strategy to optimize multiple views at the same time. Figure 1 The uniform diffusion model and the 3D Gaussian splash model are used to Figure 1 The consistency attribute is used to capture the coherent details between views. Figure 1 The high-frequency details of the consistent constraints are retained, while the inconsistent high-frequency details are eliminated. The results generated by using the 3D Gaussian splash model are used as the multi-view Figure 1 Generate multi-view images using consistent diffusion model Figure 1 Guidance on the finest details.

[0077] like Figure 4 As shown, multi-view Figure 1 The consistent diffusion model uses the denoising diffusion implicit model (DDIM) method to obtain the diffusion generation result. And used to optimize the 3D Gaussian splash model. Then, a collection of rendered images using the 3D Gaussian splash model And use consistency loss to Figure 1 The consistent diffusion model is used for supervision to enforce multi-view Figure 1 The present invention uses the following 3DGS loss function to generate consistent high-frequency details. Optimize 3D Gaussian splash model:

[0078]

[0079] Here SSIM(·,·) refers to the Structural Similarity Index measure (SSIM). It is worth noting that only a few time steps are used during inference to minimize the additional processing time. This approach is similar to the distillation operation and significantly improves the generation results and inference efficiency without compromising efficiency.

[0080] The diffusion model has demonstrated powerful generative capabilities in the field of two-dimensional image processing, and a series of general image conversion methods have emerged, achieving good conversion effects. The present invention aims to use the diffusion model to achieve general multi-view image translation. Implementing multi-view image conversion directly in the image domain can better utilize existing two-dimensional generative priors, such as the Stable Diffusion model, to achieve more flexible processing and obtain more realistic results. The challenges are as follows: The first is the lack of multi-view training data. Using real-world data for supervised learning will greatly enhance the generalization ability of the model and can effectively adapt to the complexity and variability of real scenes. However, collecting high-quality or low-quality multi-view image pairs from the real world is usually very expensive or unavailable. The second challenge is visual Figure 1 The generative characteristics of the diffusion model lead to the inevitable generation of inconsistent content for different perspectives when processing multi-view images separately. The existing technology cannot handle these two challenges well. This paper proposes a consistency and adversarial supervision strategy, potential multi-view Figure 1 To address the above challenges, this paper proposes a unified diffusion model-based multi-view image to multi-view image conversion framework, called MV2MV, for various multi-view image conversion tasks such as super-resolution, denoising, deblurring, and text-driven editing. Figure 1 The consistent diffusion model (VCDM) is used to accomplish the above tasks.

[0081] First, the present invention introduces a new self-supervised training strategy, called the consistency and adversarial supervision strategy. Specifically, the present invention first uses the existing single-view image converter to process the multi-view images separately to obtain a set of high-quality outputs, and then inputs them into the 3D Gaussian splash model to obtain perspective-consistent outputs. These two sets of output results are used as pseudo-true values ​​as supervision signals, and consistency loss and adversarial loss are introduced, which effectively combines the advantages of the two sets of pseudo-true values ​​to ensure consistency and high quality. The problem of lack of multi-view training data is solved by generating pseudo-true values, avoiding the need for multi-view training data.

[0082] Secondly, the present invention proposes a potential multi-view Figure 1 Consistency module for building multi-view Figure 1 Specifically, the potential multi-view Figure 1 The consistency module uses the potential 3DGS module as the underlying 3D representation to ensure the information exchange between multi-view images, thereby ensuring the consistency of multi-view generation and effectively solving the problem of visual Figure 1 The problem of difficult to ensure consistency.

[0083] Finally, this paper introduces a joint optimization strategy to train multiple viewpoints simultaneously. Figure 1 The consistent diffusion model and the 3D Gaussian splash model are used to ensure the consistency of the high-frequency details guided by the adversarial loss, thereby better balancing image consistency and realism.

[0084] In summary, the contributions of the present invention are as follows:

[0085] (1) The present invention constructs a multi-view Figure 1 The VCDM is a novel consistent diffusion model that can handle various translation tasks in the image domain, such as image super-resolution, deblurring, denoising, and editing tasks.

[0086] (2) This paper proposes a consistency and adversarial supervision strategy to supervise the multi-view by generating pseudo-truths with the help of the existing single-view image converter and the 3D Gaussian splash model. Figure 1 The consistent diffusion model solves the problem of lack of multi-view training data and circumvents the need for multi-view training data.

[0087] (3) The present invention proposes a potential multi-view Figure 1 The consistency module effectively solves the visual Figure 1 The problem of difficult to ensure consistency.

[0088] (4) This paper introduces a joint optimization strategy by simultaneously training multiple view Figure 1 The consistent diffusion model and the 3D Gaussian splash model are used to ensure the consistency of the high-frequency details guided by the adversarial loss, thereby better balancing image consistency and realism.

[0089] (5) Extensive experiments are conducted in both qualitative and quantitative aspects to verify the effectiveness of the proposed method. Experimental results show that the proposed method has advantages in various multi-view image translation tasks such as super-resolution, denoising, deblurring, and text-driven editing.

[0090] 4. Examples

[0091] 4.1 Datasets and Indicators:

[0092] The present invention utilizes different types of data sets to evaluate the multi-view Figure 1 The performance of the consistent diffusion model in different image conversion tasks. For image super-resolution, the present invention conducts experiments on the complex real-world dataset Mip-Nerf360, which contains 9 unbounded indoor and outdoor scenes. For text-driven editing tasks, the present invention conducts experiments on two positive scenes, Face and Fangzhou, and the Garden scene of the 360-degree dataset.

[0093] In the quantitative experiments, the present invention uses reference and non-reference indicators to comprehensively evaluate each method. Figure 1 Due to the generation characteristics of the consistent diffusion model, the details of the results may not faithfully follow the ground-truth. The present invention adopts indicators that are closely related to human visual perception. LPIPS and DISTS are used as reference-based metrics to evaluate the perceptual quality of the generated results relative to the ground-truth. NIQE, MANIQA and MUSIQ are non-reference image quality assessment indicators for evaluating image fidelity, which are closer to human perception. In addition, in order to better evaluate the performance of text-driven editing tasks, the present invention also uses CLIP Text Image Directional Similarity and CLIP Direction Consistency as evaluation indicators, which measure the alignment of the performed edits with text instructions and the consistency of the edits performed across views.

[0094] 4.2 Image super-resolution task:

[0095] On the Mip-Nerf360 dataset, the present invention uses image data that has been reduced by 8 times as Ground-truth. In addition, in order to adapt to the ×4 times super-resolution task, the present invention uses 4 times bicubic interpolation to generate low-resolution images. In this experiment, CCSR is used as a 2D image converter in the framework of the present invention. The present invention compares MV2MV with several state-of-the-art methods, including NeRF-based methods: NeRF-SR, and 2D image super-resolution methods: SwinIR, BSRGAN, DiffBIR, CCSR, StableSR. Bicubic interpolation is included in the comparison of the present invention as a baseline. Please note that the present invention uses a 2D image super-resolution method for view-by-view processing and is directly integrated with a 3D Gaussian splash model.

[0096] The quantitative results are shown in Table 1. Due to its stronger generation ability, MV2MV achieves the highest scores in NIQE, MANIQA and MUSIQ compared to all baselines, indicating that it is more consistent with human visual perception. In addition, MV2MV achieves comparable scores with other methods in LPIPS and DISTS, which indicates that the perceptual quality of the ground-truth is better. Figure 5 As shown in Figure 2. Although diffusion-based methods such as DiffBIR, CCSR, and StableSR are good at generating realistic details when processing images individually, multi-view inconsistencies can lead to different degrees of blurry reconstruction. In contrast, the MV2MV of the present invention is able to generate multi-view Figure 1 Realistic details.

[0097] Table 1. Quantitative evaluation results of image super-resolution tasks

[0098]

[0099] 4.3 Text-driven editing tasks

[0100] We implement text-driven editing by using InstructPix2Pix as an image converter in our framework. We compare MV2MV with two recent state-of-the-art methods, Instruct-NeRF2NeRF and Instruct-GS2GS, which use iterative update mechanisms to resolve 3D inconsistencies between different edits.

[0101] Although the editing task is a subjective task, we evaluate two quantitative indicators, CLIP Text--Image DirectionSimilarity and CLIP Direction Consistency, and report non-reference image quality assessment indicators to evaluate the quality of text-driven editing results. The quantitative results are shown in Table 2. The MV2MV of our method outperforms other methods in all indicators, proving its effectiveness in text-driven editing tasks. Qualitative comparisons are shown in Table 2. Figure 6 The method of the present invention produces more consistent and realistic results, while other methods produce ambiguous results due to inconsistent editing.

[0102] Table 2. Quantitative evaluation results on text-driven editing tasks

[0103]

[0104] It is obvious to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present invention is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present invention, and any reference numerals in the claims should not be regarded as limiting the claims involved.

[0105] In addition, it should be understood that although the present specification is described in terms of implementation modes, not every implementation mode contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. A multi-view conversion method based on a multi-view consistent diffusion model, characterized in that: Ability to achieve image transformation by completing the trained multi-view consistency diffusion model; The training process of the multi-view consistent diffusion model specifically includes: Step 1: construct a multi-view image to multi-view image conversion framework based on a diffusion model, and the conversion framework is used to train a multi-view consistency diffusion model; the multi-view consistency diffusion model includes a latent multi-view consistency module, a VAE encoder, a first ControlNet module and a denoising Unet network; the latent multi-view consistency module includes a latent 3DGS module and a second ControlNet module; Step 2: Convert the multi-view image to a single view to obtain a converted image; render the converted image through a 3D Gaussian splash model to obtain a rendered image; add noise to the rendered image to convert it into a noisy rendered image; input the multi-view image to the VAE encoder to obtain a conditional feature map, and input the conditional feature map and the noisy rendered image to the first ControlNet module; input the consistency feature map output by the potential 3DGS module and the noisy rendered image to the second ControlNet module; input the text prompt into the denoising Unet network after passing through the text encoder; the denoising Unet network in the multi-view consistency diffusion model generates an image; Step three, regarding the converted image and the rendered image as pseudo-true values, and based on the generated image, supervised training is performed on the multi-view consistency diffusion model through the discriminator and the adversarial loss function in the adversarial generative network.

2. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 1, characterized in that: The denoising UNet network includes an encoder, a decoder and an intermediate layer. The encoder includes multiple encoding layers, and the decoder includes multiple decoding layers. The first ControlNet module and the second ControlNet module both include multiple encoding layers, multiple zero convolutions and intermediate layers. The number of encoding layers of the first ControlNet module and the second ControlNet module is the same as the number of encoding layers of the encoder, and the number of zero convolutions of the first ControlNet module and the second ControlNet module is the same as the number of decoding layers of the decoder. The latent 3DGS module is based on the 3D Gaussian splash algorithm and uses Gaussian spheres to store latent features.

3. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 1, characterized in that: In step 2, the multi-view image is converted into a single view to obtain a converted image, and the converted image is rendered using a 3D Gaussian splash model to obtain a rendered image, which specifically includes: The collection of multi-view images is denoted as I i represents the i-th image, N represents the number of images, and the set of multi-view images is input into the 2D single-view image converter to generate a set of converted images h i is the i-th transformed image; the 3D Gaussian splash model is directly optimized using the transformed image to enforce a set of geometrically consistent rendered images x i Render the i-th image.

4. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 3 is characterized in that: Step three specifically includes: A collection of rendered images consistent with a given geometry When training the multi-view consistent diffusion model, we directly minimize the difference between the generated image and x at each time step. i The difference between them is as follows: randomly select a time step t, add noise, and change x i Convert to a noisy rendered image The consistency loss function for: in, To generate an image based on the predicted noise output through the denoising UNet network at time step t; the consistency loss function enables the multi-view consistency diffusion model to generate multi-view consistent results; The coefficient representing the duration t is used to control the noise level; ∈ θ (·) represents the denoising UNet network; Convert the image As the true sample R, the generated image output by the multi-view consistency diffusion model As a fake sample F; supervise the multi-view consistency diffusion model through the discriminator in the adversarial generation network and the adversarial loss function.

5. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 4 is characterized in that: The multi-view consistency diffusion model is supervised by the discriminator and the adversarial loss function in the adversarial generative network, specifically including: The adversarial loss function between the multi-view consistency diffusion model and the discriminator is: Represents the generation loss, used to train the multi-view consistent diffusion model; Represents the discriminative loss, which is used to train the discriminator; E R and E F Both represent expectations, and D(·) represents the discriminator.

6. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 4 is characterized in that: In order to introduce geometric constraints to make the generated images output by the multi-view consistent diffusion model With the converted image With geometric similarity, the perceptual loss function is introduced For training multi-view consistent diffusion models: Here, LPIPS(·,·) denotes learning-perceptual patch similarity.

7. The multi-view conversion method based on the multi-view consistent diffusion model according to claim 3 is characterized in that: It also includes a joint optimization strategy, which simultaneously optimizes the multi-view consistent diffusion model and the 3D Gaussian splash model, including: The generated image output by the multi-view consistent diffusion model is input into the denoising diffusion implicit model to obtain the optimized image Then, using the rendered image The multi-view consistency diffusion model is supervised and optimized through consistency loss, using the following 3DGS loss function Optimize 3D Gaussian splash model: λ represents the weight coefficient, and SSIM(·,·) refers to the structural similarity index.

Images