Multi-view conversion method based on multi-view consistency diffusion model

Abstract

The present invention relates to the field of image conversion technology, and discloses a multi-view conversion method based on a multi-view consistency diffusion model. A multi-view consistency diffusion model is trained through a unified multi-view to multi-view conversion framework based on a diffusion model. The conversion framework has a new self-supervised training strategy, a consistency and adversarial supervision strategy, which utilizes the existing 2D single-view image converter and 3D Gaussian splash technology to generate pseudo-true values ​​as supervision signals, thereby enhancing view consistency and high-frequency details; the multi-view consistency diffusion model trained by the conversion framework also includes a latent multi-view consistency module, which utilizes the latent 3DGS module as the underlying 3D representation to promote information exchange across multiple views, thereby introducing 3D priors into the diffusion model to ensure view consistency. Finally, a joint optimization strategy is proposed, which can achieve a better trade-off between consistency and authenticity.

Description

Technical Field

[0001] The present invention relates to the field of image conversion technology, and in particular to a method based on multi-view Figure 1 A multi-view transformation method for consistent diffusion model. Background Art

[0002] Image translation is a long-standing problem in computer graphics and computer vision. Many problems can be considered image-to-image translation. Image translation aims to convert an image from one domain to another. It conditions the input image and transforms the image from the source domain to the target domain while preserving its content. These include image super-resolution, deblurring, denoising, editing tasks, and many more.

[0003] Due to the good generative ability of diffusion models, recent single-view image translation methods based on diffusion models have achieved realistic results. However, most existing research focuses on single-view image translation tasks, and there are still two major problems in directly applying diffusion models to multi-view image translation tasks:

[0004] First: lack of multi-view training data;

[0005] Second: Multi-view Figure 1 Consistency is difficult to guarantee.

[0006] For image super-resolution, DiffBIR and StableSR leverage the generative power of latent diffusion models to generate realistic high-resolution images. For image deblurring, HiDiff implements diffusion models to generate prior features in the latent space for deblurring and restoring beautiful images. For image editing, InstructPix2Pix leverages large pre-trained models in language and image domains to generate training data pairs for diffusion models used in text editing.

[0007] Although these methods have produced good results for single-view image processing in their respective tasks, they have encountered difficulties when applying them to multi-view image conversion tasks. Due to the lack of multi-view training data for targeted training, these models cannot be well applied to multi-view conversion tasks. And simply performing frame-by-frame image conversion will cause multi-view image conversion errors. Figure 1 The problem of difficulty in ensuring consistency leads to inconsistencies in geometry and appearance between images from different perspectives. Summary of the Invention

[0008] In order to solve the above technical problems, the present invention provides a multi-view Figure 1The multi-view image translation method based on the consistent diffusion model is trained through a unified multi-view to multi-view conversion framework (Multi-View Image Translation via View-Consistent Diffusion Models, MV2MV) based on the diffusion model. Figure 1 View-Consistent Diffusion Models (VCDM).

[0009] First, the conversion framework has a new self-supervised training strategy - consistency and adversarial supervision strategy, which utilizes the existing 2D single-view image converter and 3D Gaussian splashing technology (3DGS) to generate pseudo-truth as a supervision signal to enhance the visual Figure 1 The first problem mentioned above is solved by improving the consistency and high-frequency details.

[0010] Secondly, the multi-view Figure 1 The consistent diffusion model also includes potential multi-view Figure 1 The consistency module uses the potential 3DGS module as the underlying 3D representation to promote information exchange across multiple views, thereby introducing 3D priors into the diffusion model to ensure the visual Figure 1 This solves the second problem mentioned above.

[0011] Finally, a joint optimization strategy is proposed to achieve a better trade-off between consistency and authenticity.

[0012] Under the above scheme, our method outperforms the state-of-the-art methods both quantitatively and qualitatively in extensive experiments conducted on various image conversion tasks.

[0013] The present invention specifically adopts the following technical solutions:

[0014] A multi-view based Figure 1 The multi-view conversion method of the consistent diffusion model can complete the training of multi-view Figure 1 Image conversion based on the consistent diffusion model; multi-view Figure 1 The training process of the consistent diffusion model specifically includes:

[0015] Step 1: Construct a multi-view image to multi-view image conversion framework based on the diffusion model. The conversion framework is used to train the multi-view image. Figure 1 Conformal diffusion model; multi-view Figure 1 The consistent diffusion model includes potential multi-view Figure 1 Consistency module, VAE encoder, first ControlNet module and denoising Unet network; potential multi-view Figure 1The consistency module includes a potential 3DGS module and a second ControlNet module;

[0016] Step 2: Perform single-view conversion on the multi-view image to obtain a converted image; render the converted image through a 3D Gaussian splash model to obtain a rendered image; add noise to the rendered image to convert it into a noisy rendered image; input the multi-view image into the VAE encoder to obtain a conditional feature map, and input the conditional feature map and the noisy rendered image into the first ControlNet module; input the consistency feature map output by the potential 3DGS module and the noisy rendered image into the second ControlNet module; input the text prompt into the denoising Unet network after passing through the text encoder; multi-view Figure 1 The denoising Unet network output in the consistent diffusion model generates an image;

[0017] Step 3: The converted image and the rendered image are regarded as pseudo-true values, and the multi-view image is generated based on the discriminator and the adversarial loss function in the adversarial generation network. Figure 1 The consistent diffusion model is trained for supervision.

[0018] Furthermore, the denoising UNet network includes an encoder, a decoder and an intermediate layer, the encoder includes multiple encoding layers, and the decoder includes multiple decoding layers; the first ControlNet module and the second ControlNet module both include multiple encoding layers, multiple zero convolutions and intermediate layers; the number of encoding layers of the first ControlNet module and the second ControlNet module is the same as the number of encoding layers of the encoder, and the number of zero convolutions of the first ControlNet module and the second ControlNet module is the same as the number of decoding layers of the decoder; the latent 3DGS module is based on the 3D Gaussian splash algorithm and uses Gaussian spheres to store latent features.

[0019] Furthermore, in step 2, performing single-view conversion on the multi-view image to obtain a converted image, and rendering the converted image using a 3D Gaussian splash model to obtain a rendered image specifically includes:

[0020] The set of multi-view images is denoted as I i Represents the i-th image, N represents the number of images, and the set of multi-view images is input into the 2D single-view image converter to generate a set of converted images h i is the i-th transformed image; the 3D Gaussian splash model is directly optimized using the transformed image to enforce a set of geometrically consistent rendered images x i Render the i-th image.

[0021] Furthermore, step three specifically includes:

[0022] A set of geometrically consistent rendered images for a given In training multi-view Figure 1 When using the consistent diffusion model, the generated image and x are directly minimized at each time step. i The difference between them is as follows: randomly select a time step t, add noise, and change x i Convert to a noisy rendered image The consistency loss function for:

[0023]

[0024] in, To generate an image based on the predicted noise output by the denoising UNet network at time step t; the consistency loss function makes the multi-view Figure 1 The consistent diffusion model can generate multi-view Figure 1 the result; The coefficient representing the duration t, used to control the noise level; ∈ θ (·) represents the denoising UNet network;

[0025] The image will be converted As the true sample R, and the multi-view Figure 1 Generated image of the output of the consistent diffusion model As a fake sample F; supervise the multi-view through the discriminator and adversarial loss function in the adversarial generation network Figure 1 Causative diffusion model.

[0026] Furthermore, the discriminator and the adversarial loss function in the adversarial generative network supervise the multi-view Figure 1 The consistent diffusion model includes:

[0027] Multi-view Figure 1 The adversarial loss function between the consistent diffusion model and the discriminator is:

[0028]

[0029]

[0030] Represents the generation loss for training multi-view Figure 1 Causative diffusion model; Represents the discriminative loss, used to train the discriminator; E R and E F Both represent expectations, and D(·) represents the discriminator.

[0031] Furthermore, in order to introduce geometric constraints to make multi-view Figure 1Generated image of the output of the consistent diffusion model With converted image With geometric similarity, the perceptual loss function is introduced For training multi-view Figure 1 Causative diffusion model:

[0032]

[0033] Here, LPIPS(·,·) denotes the learned perceptual patch similarity.

[0034] Furthermore, it also includes a joint optimization strategy, which optimizes multiple views at the same time. Figure 1 Conformal diffusion model and 3D Gaussian splash model, including:

[0035] Multi-view Figure 1 The generated image output by the consistent diffusion model is input into the denoising diffusion implicit model to obtain the optimized image Then, using the rendered image Multi-view via consistency loss Figure 1 The consistent diffusion model is supervised and optimized through the following 3DGS loss function Optimize 3D Gaussian splash model:

[0036]

[0037] λ represents the weight coefficient, and SSIM(·,·) refers to the structural similarity index.

[0038] Compared with the prior art, the beneficial technical effects of the present invention are:

[0039] 1. This paper proposes a new consistency and adversarial supervision strategy. It can effectively circumvent the requirement of real training data, and utilizes the prior knowledge of existing single-view image converters based on diffusion models that are well trained on large 2D image datasets. It generates pseudo-true values ​​and guarantees the consistency loss and adversarial loss of multi-view images. Figure 1 Multi-view Figure 1 The results are consistent and have high-frequency details. It solves the problem of lack of multi-view training data.

[0040] 2. The present invention proposes a potential multi-view Figure 1 Consistency module (LAConsistNet). It makes multi-view Figure 1 The consistent diffusion model can explicitly enforce multi-view geometric consistency. Compared with previous methods that construct 3D voxels as the underlying 3D representation, the method of the present invention proposes to use the latent 3DGS module to generate a consistent feature map and combine it with the multi-view Figure 1The method of the present invention is memory-friendly and efficiency-friendly, can handle large-scale scenes, and is applicable to different multi-view image conversion tasks. Figure 1 Problems with ensuring consistency

[0041] 3. The present invention proposes a multi-view Figure 1 The joint optimization strategy of the uniform diffusion model and the 3D Gaussian splash model is proposed. Figure 1 The consistent diffusion model can produce high-frequency details, but it cannot guarantee the consistency of these details. Figure 1 The uniform diffusion model and 3D Gaussian splash model are used to Figure 1 Consistency attributes are used to obtain coherent details across views and transfer them to multi-view Figure 1 The consistent diffusion model ensures the generation of consistent high-frequency details. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A schematic diagram of the structure of MV2MV provided according to an embodiment of the present invention;

[0043] Figure 2 A flow chart of a consistency and adversarial supervision strategy provided according to an embodiment of the present invention;

[0044] Figure 3 The potential multi-viewing provided by the embodiment of the present invention Figure 1 Consistent module structure diagram;

[0045] Figure 4 A flow chart of a joint optimization strategy provided according to an embodiment of the present invention;

[0046] Figure 5 A qualitative comparison diagram of image super-resolution tasks provided according to an embodiment of the present invention;

[0047] Figure 6 A qualitative comparison diagram of text-driven editing tasks provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] A preferred embodiment of the present invention will be described in detail below with reference to the accompanying drawings.

[0049] The structure of the multi-view to multi-view conversion framework based on the diffusion model of the present invention is as follows: Figure 1 As shown, the purpose of the conversion framework is to train multi-view Figure 1To circumvent the requirement for real training data, we first propose a new self-supervised training strategy, called consistency and adversarial supervision strategy, which leverages the prior knowledge of existing single-view image converters that are well trained on large 2D image datasets. Specifically, we process multi-view images separately to generate high-quality converted images, which are then fed into a 3D Gaussian splatter model (3DGS) to obtain perspective-consistent rendered images. We regard the converted and rendered images as ground-truth and supervise the multi-view training with consistency loss and adversarial loss. Figure 1 Secondly, the potential multi-view Figure 1 The consistency module uses the latent 3DGS module (Latent-3DGS) as the underlying 3D representation to achieve cross-view information exchange. Specifically, the consistency feature map output by the latent 3DGS module is used as the input of the ControlNet module, and the ControlNet module is combined with the multi-view Figure 1 The corresponding decoder layers of the denoising UNet network in the consistent diffusion model are combined to enhance consistency. Finally, the present invention proposes a joint optimization strategy to train multiple viewpoints simultaneously. Figure 1 A consistent diffusion model and a 3D Gaussian splash model are used to further enhance the consistency and realism of the generated images.

[0050] The VAE encoder, ControlNet module and denoising UNet network used in this invention are existing technologies. The innovation of this invention is to use these technologies to construct a multi-view Figure 1 The consistent diffusion model does not involve improvements to the VAE encoder, ControlNet module, and denoising UNet network, so I will not go into details.

[0051] In addition, 3D Gaussian splatter is an existing technology, which is a rasterization technology for describing 3D Gaussian distribution for real-time radiation field rendering.

[0052] The specific technical solutions of the present invention are introduced in several parts below.

[0053] 1. Consistency and Adversarial Supervision Strategies

[0054] 1.1 Pseudo-truth value generation:

[0055] In recent years, diffusion models have achieved success in 2D image conversion tasks due to their strong prior knowledge and appearance generation capabilities. Therefore, we propose a self-supervised framework that leverages the success of existing diffusion-based single-view image converters and 3D Gaussian splash models to generate pseudo-truths as supervision signals, circumventing the requirement for real training data. Although pseudo-truths can also be obtained using non-diffusion-based methods, in practice, we prioritize methods that can generate high-quality and realistic texture details. We use several multi-view image conversion tasks to demonstrate the adaptability of MV2MV: super-resolution, denoising, deblurring, and text-driven editing.

[0056] Given an input multi-view image set The present invention first generates a set of converted images through an existing 2D single-view image converter. Next, the present invention uses these transformed images to directly optimize the 3D Gaussian splash model to enforce the production of a set of geometrically consistent rendered images Due to the generative nature of image converters, Although the quality is good, there are obvious visual inconsistencies. Show good vision Figure 1 Then, the present invention focuses on obtaining the best performance in visual field by proposing the consistency and adversarial supervision strategies. Figure 1 The results are well balanced between consistency and high quality. Specifically, the present invention proposes consistency loss and adversarial loss to achieve these two optimization goals. The former ensures the multi-view Figure 1 Multi-view Figure 1 The latter is used to recover high-frequency details.

[0057] 1.2 Consistency loss:

[0058] A set of rendered images from multiple perspectives given geometric consistency In training multi-view Figure 1 When the consistent diffusion model is used, the present invention directly minimizes the generated result and x at each time step. i Specifically, the present invention randomly selects a time step t and adds noise to the rendered image x i Convert to a noisy rendered image Consistency loss for:

[0059]

[0060] in, is the image generated by the denoising UNet network based on the predicted noise at time step t. The consistency loss ensures that the multi-view Figure 1The consistent diffusion model can generate multi-view Figure 1 The result.

[0061] 1.3 Adversarial Loss

[0062] because The geometric and appearance inconsistencies between them make them unsuitable for direct use in supervising the multi-view Figure 1 Therefore, the goal of this paper is to recover high-frequency details by adopting an adversarial training strategy. Previous studies have shown that adversarial training is effective in preventing rendering blur caused by viewpoint inconsistency conflicts during different viewpoint supervision processes.

[0063] Specifically, the present invention proposes to minimize the generation of multi-view Figure 1 Preliminary generated image of the consistent diffusion model Convert image with high quality h i The distribution difference between Figure 1 The present invention transforms the set of images into As the true sample R, and the multi-view Figure 1 A collection of generated images for the consistent diffusion model As a fake sample F. Multi-view supervision through the discriminator and adversarial loss in the adversarial generation network Figure 1 For generative adversarial networks, see “Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. Advances in Neural Information Processing Systems 27 (2014).”

[0064] Multi-view Figure 1 The adversarial loss function between the consistent diffusion model and the discriminator is:

[0065]

[0066]

[0067] In addition, in order to introduce geometric constraints to promote multi-view Figure 1 Generative image of the consistent diffusion model With high quality converted image h i With geometric similarity, the present invention introduces the perceptual loss function

[0068]

[0069] Here, LPIPS(·,·) represents the learned perceptual image patch similarity (LPIPS).

[0070] The process of consistency and adversarial supervision strategy can be found in Figure 2 .

[0071] 2. Potential Multi-View Figure 1 Consistency Module

[0072] In order to multi-view Figure 1 Realizing visual Figure 1 Consistency, inspired by the ConsistNet module, this paper proposes a potential multi-view Figure 1 The consistency module injects the 3D prior into the neural network block as an additional condition, such as Figure 3 Unlike the ConsistNet module that uses View Aggregation and Ray Aggregation modules to aggregate multi-view information, the present invention uses the latent 3DGS module as the underlying 3D representation to model multi-view geometric representation, making multi-view Figure 1 The consistent diffusion model can more simply and effectively complete the multi-view conversion task. The latent 3DGS module is based on the 3D Gaussian splash algorithm, which uses Gaussian spheres to store latent features and treats them as trainable parameters.

[0073] Potential multi-viewing Figure 1 The consistency module is built on the ControlNet module and the pre-trained Stable Diffusion. The 3D consistency feature map rendered by the latent 3DGS module is used as the input of the ControlNet module and inserted into each encoding layer of the denoising UNet network to enforce 3D consistency. Specifically, the denoising UNet network includes an encoder and a decoder, where the encoder and decoder both have 12 layers and 1 intermediate layer. Similar to the ControlNet module, the present invention creates trainable copies of 12 encoding layers and 1 intermediate layer, and uses a zero convolution layer (i.e., a 1×1 zero-initialized convolution layer) to append its output to each decoding layer of the denoising UNet network.

[0074] To achieve fast training on pre-trained Stable Diffusion, the trainable structure of the ControlNet module is consistent with the literature "Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023. In addition, the present invention uses a pre-trained 3D Gaussian splash model to initialize the latent 3DGS module as the initial 3D prior.

[0075] 3. Joint Optimization Strategy

[0076] In the framework of the present invention, a potential multi-view Figure 1 Consistency module and consistent loss to ensure multi-view Figure 1 Multi-view Figure 1 However, the generation of details is accompanied by randomness, which leads to variations in high-frequency details generated by multi-view images, which has a negative impact on training stability and visual quality. Figure 1 This prompts the present invention to propose a joint optimization strategy to optimize multiple views at the same time. Figure 1 The uniform diffusion model and 3D Gaussian splash model are used to Figure 1 The consistency property is used to capture the coherent details between views. Figure 1 The high-frequency details of the consistent constraints are retained, while the inconsistent high-frequency details are eliminated. The results generated by the 3D Gaussian splash model are used as the multi-view Figure 1 Conformal Diffusion Model Generates Multi-View Figure 1 Guidance on the finest details.

[0077] like Figure 4 As shown, multi-view Figure 1 The consistent diffusion model uses the denoising diffusion implicit model (DDIM) method to obtain the diffusion generation result. And used to optimize the 3D Gaussian splash model. Then, the collection of rendered images using the 3D Gaussian splash model And through the consistency loss Figure 1 The consistent diffusion model is used for supervision to enforce multi-view Figure 1 The present invention uses the following 3DGS loss function to generate consistent high-frequency details. Optimize 3D Gaussian splash model:

[0078]

[0079] Here, SSIM(·,·) refers to the Structural Similarity Index measure (SSIM). Notably, only a few time steps are used during inference to minimize additional processing time. This approach, similar to a distillation operation, significantly improves both generation and inference efficiency without sacrificing performance.

[0080] The diffusion model has demonstrated powerful generative capabilities in the field of two-dimensional image processing, and a series of general image conversion methods have emerged, achieving good conversion effects. The present invention aims to use the diffusion model to achieve general multi-view image translation. Implementing multi-view image conversion directly in the image domain can better utilize existing two-dimensional generative priors, such as the Stable Diffusion model, to achieve more flexible processing and obtain more realistic results. The challenges are as follows: The first is the lack of multi-view training data. Supervised learning using real-world data will greatly enhance the generalization ability of the model and can effectively adapt to the complexity and variability of real scenes. However, collecting high-quality or low-quality multi-view image pairs from the real world is usually very expensive or unavailable. The second challenge is visual Figure 1 The generative characteristics of the diffusion model lead to the inevitable generation of inconsistent content for different viewpoints when processing multi-view images separately. The existing technology cannot handle these two challenges well. This paper proposes a consistency and adversarial supervision strategy, a potential multi-view Figure 1 To address the above challenges, this paper proposes a unified multi-view image to multi-view image conversion framework based on diffusion model, called MV2MV, for various multi-view image conversion tasks such as super-resolution, denoising, deblurring and text-driven editing. Specifically, this paper constructs a multi-view image to multi-view image conversion framework based on diffusion model, called MV2MV. Figure 1 The consistent diffusion model (VCDM) is used to accomplish the above tasks.

[0081] First, the present invention introduces a new self-supervised training strategy, called the consistency and adversarial supervision strategy. Specifically, the present invention first uses the existing single-view image converter to process the multi-view images separately to obtain a set of high-quality outputs, and then inputs them into the 3D Gaussian splash model to obtain perspective-consistent outputs. These two sets of output results are used as pseudo-true values ​​as supervision signals, and consistency loss and adversarial loss are introduced, which effectively combines the advantages of the two sets of pseudo-true values ​​to ensure consistency and high quality. By generating pseudo-true values, the problem of lack of multi-view training data is solved, and the need for multi-view training data is circumvented.

[0082] Secondly, the present invention proposes a potential multi-view Figure 1 Consistency module for building multi-view Figure 1 Specifically, the latent multi-view Figure 1 The consistency module uses the potential 3DGS module as the underlying 3D representation to ensure the information exchange between multi-view images, thereby ensuring the consistency of multi-view generation and effectively solving the problem of visual Figure 1 The problem of difficulty in ensuring consistency.

[0083] Finally, this paper introduces a joint optimization strategy to train multiple viewpoints simultaneously. Figure 1 The consistent diffusion model and 3D Gaussian splash model are used to ensure the consistency of high-frequency details guided by the adversarial loss, thereby better balancing image consistency and realism.

[0084] In summary, the contributions of the present invention are as follows:

[0085] (1) The present invention constructs a multi-view Figure 1 The consistent diffusion model (VCDM) is able to handle various translation tasks in the image domain, such as image super-resolution, deblurring, denoising, and editing tasks.

[0086] (2) This paper proposes a consistency and adversarial supervision strategy to supervise multi-view by generating pseudo-truths with the help of the existing single-view image converter and 3D Gaussian splash model. Figure 1 The consistent diffusion model solves the problem of lack of multi-view training data and circumvents the need for multi-view training data.

[0087] (3) The present invention proposes a potential multi-view Figure 1 The consistency module effectively solves the visual Figure 1 The problem of difficulty in ensuring consistency.

[0088] (4) This paper introduces a joint optimization strategy by simultaneously training multiple Figure 1 The consistent diffusion model and 3D Gaussian splash model are used to ensure the consistency of high-frequency details guided by the adversarial loss, thereby better balancing image consistency and realism.

[0089] (5) Extensive qualitative and quantitative experiments are conducted to verify the effectiveness of the proposed method. The experimental results show that the proposed method has advantages in various multi-view image translation tasks such as super-resolution, denoising, deblurring, and text-driven editing.

[0090] 4. Examples

[0091] 4.1 Datasets and Metrics

[0092] The present invention utilizes different types of data sets to evaluate the multi-view Figure 1 The performance of the consistent diffusion model in different image transformation tasks was investigated. For image super-resolution, experiments were conducted on the complex real-world dataset Mip-Nerf360, which contains nine unbounded indoor and outdoor scenes. For text-driven editing tasks, experiments were conducted on two forward-looking scenes, Face and Fangzhou, as well as the Garden scene from the 360-degree dataset.

[0093] In the quantitative experiments, the present invention uses reference and non-reference indicators to comprehensively evaluate each method. Figure 1 Due to the generative characteristics of the consistent diffusion model, the details of the results may not faithfully follow the ground-truth. The present invention adopts indicators that are closely related to human visual perception. LPIPS and DISTS are used as reference-based metrics to evaluate the perceptual quality of the generated results relative to the ground-truth. NIQE, MANIQA and MUSIQ are non-reference image quality assessment indicators for evaluating image fidelity, which are closer to human perception. In addition, in order to better evaluate the performance of text-driven editing tasks, the present invention also uses CLIP Text Image Directional Similarity and CLIP Direction Consistency as evaluation indicators, which measure the alignment of the performed edits with text instructions and the consistency of the edits performed across views.

[0094] 4.2 Image super-resolution task:

[0095] On the Mip-Nerf360 dataset, the present invention uses image data that has been reduced by 8 times as ground-truth. In addition, in order to adapt to the ×4-fold super-resolution task, the present invention uses 4-fold bicubic interpolation to generate low-resolution images. In this experiment, CCSR is used as a 2D image converter in the framework of the present invention. The present invention compares MV2MV with several state-of-the-art methods, including NeRF-based methods: NeRF-SR, and 2D image super-resolution methods: SwinIR, BSRGAN, DiffBIR, CCSR, StableSR. Bicubic interpolation is included in the comparison of the present invention as a baseline. Please note that the present invention uses a 2D image super-resolution method for view-by-view processing and is directly integrated with a 3D Gaussian splash model.

[0096] The quantitative results are shown in Table 1. Due to its stronger generation ability, MV2MV achieves the highest scores in NIQE, MANIQA, and MUSIQ compared to all baselines, indicating that it is more consistent with human visual perception. In addition, MV2MV achieves comparable scores with other methods in LPIPS and DISTS, which indicates that the perceptual quality of the ground-truth is better. Figure 5 As shown in Figure 2. Although diffusion-based methods such as DiffBIR, CCSR, and StableSR are good at generating realistic details when processing images individually, multi-view inconsistencies can lead to varying degrees of blurry reconstruction. In contrast, the MV2MV of our invention is able to generate multi-view Figure 1 Realistic details.

[0097] Table 1. Quantitative evaluation results of image super-resolution tasks

[0098]

[0099] 4.3 Text-driven editing tasks

[0100] We implement text-driven editing by using InstructPix2Pix as an image converter in our framework. We compare MV2MV with two recent state-of-the-art methods, Instruct-NeRF2NeRF and Instruct-GS2GS, which use iterative update mechanisms to resolve 3D inconsistencies between different edits.

[0101] Although editing is a subjective task, we evaluate two quantitative metrics, CLIP Text-Image DirectionSimilarity and CLIP Direction Consistency, and report non-reference image quality assessment metrics to evaluate the quality of text-driven editing results. The quantitative results are shown in Table 2. The MV2MV method of our invention outperforms other methods in all metrics, demonstrating its effectiveness in text-driven editing tasks. Qualitative comparisons are shown in Table 2. Figure 6 The method of the present invention produces more consistent and realistic results, while other methods produce ambiguous results due to inconsistent editing.

[0102] Table 2. Quantitative evaluation results on text-driven editing tasks

[0103]

[0104] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above and that the invention can be embodied in other specific forms without departing from the spirit or essential characteristics of the invention. Therefore, the embodiments should be considered in all respects as illustrative and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. It is intended that all variations within the meaning and range of equivalents of the claims be embraced herein, and any reference signs in the claims should not be construed as limiting the claims to which they relate.

[0105] In addition, it should be understood that although this specification is described in terms of implementation methods, not every implementation method contains only one independent technical solution. This narrative method of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

Claims

1. A multi-view conversion method based on a multi-view consistency diffusion model, characterized in that: Ability to achieve image transformation by completing the trained multi-view consistency diffusion model; The training process of the multi-view consistent diffusion model specifically includes: Step 1: Build a multi-view image to multi-view image conversion framework based on a diffusion model. The conversion framework is used to train a multi-view consistency diffusion model. The multi-view consistency diffusion model includes a latent multi-view consistency module, a VAE encoder, a first ControlNet module, and a denoising Unet network. The latent multi-view consistency module includes a latent 3DGS module and a second ControlNet module. Step 2: convert the multi-view image into a single view to obtain a converted image; render the converted image using a 3D Gaussian splash model to obtain a rendered image, specifically including: the set of multi-view images is recorded as , Indicates the images, where N represents the number of images. The set of multi-view images is input into the 2D single-view image converter to generate a set of converted images. , For the A transformed image; the 3D Gaussian splatter model is directly optimized using the transformed image to enforce a set of geometrically consistent rendered images , For the A rendered image; Adding noise to the rendered image converts it into a noisy rendered image; inputting the multi-view image into the VAE encoder to obtain a conditional feature map, and inputting the conditional feature map and the noisy rendered image into the first ControlNet module; inputting the consistency feature map output by the latent 3DGS module and the noisy rendered image into the second ControlNet module; the text prompt is input into the denoising Unet network after passing it through the text encoder; the denoising Unet network in the multi-view consistency diffusion model outputs the generated image; Step 3: Consider the converted image and the rendered image as pseudo-true values, and supervise the multi-view consistency diffusion model based on the generated image and the discriminator and adversarial loss function in the adversarial generative network, specifically including: for a given set of geometrically consistent rendered images , when training the multi-view consistent diffusion model, at each time step, the difference between the generated image and The differences between the two include: randomly selecting a time step , and add noise, Convert to a noisy rendered image , consistency loss function for: ;in, For a time step of The denoising UNet network is used to generate images based on the predicted noise output; the consistency loss function enables the multi-view consistency diffusion model to generate consistent results for multiple views; Indicates duration The coefficient of is used to control the noise level; Represents the denoising UNet network; the image will be converted As a true sample , and the generated image output by the multi-view consistent diffusion model As a fake sample ; The multi-view consistent diffusion model is supervised by the discriminator and the adversarial loss function in the adversarial generative network. Specifically, the adversarial loss function between the multi-view consistent diffusion model and the discriminator is: ; ; Represents the generation loss, used to train the multi-view consistent diffusion model; Represents the discriminative loss, used to train the discriminator; and All expressed expectations, represents the discriminator.

2. The multi-view conversion method based on the multi-view consistency diffusion model according to claim 1 is characterized in that: The denoising UNet network includes an encoder, a decoder and an intermediate layer. The encoder includes multiple encoding layers, and the decoder includes multiple decoding layers. The first ControlNet module and the second ControlNet module both include multiple encoding layers, multiple zero convolutions and intermediate layers. The number of encoding layers in the first and second ControlNet modules is the same as that of the encoder, and the number of zero convolutions in the first and second ControlNet modules is the same as that of the decoder. The latent 3DGS module is based on the 3D Gaussian splash algorithm and uses Gaussian spheres to store latent features.

3. The multi-view conversion method based on the multi-view consistency diffusion model according to claim 1, characterized in that: In order to introduce geometric constraints to make the generated images output by the multi-view consistent diffusion model With converted image With geometric similarity, the perceptual loss function is introduced , used to train the multi-view consistent diffusion model: ； here, Represents learning to perceive image patch similarity.

4. The multi-view conversion method based on the multi-view consistency diffusion model according to claim 1, characterized in that: It also includes a joint optimization strategy, which simultaneously optimizes the multi-view consistent diffusion model and the 3D Gaussian splash model. Specifically, it includes: The generated image output by the multi-view consistent diffusion model is input into the denoising diffusion implicit model to obtain the optimized image , then, using the rendered image The multi-view consistency diffusion model is supervised and optimized through consistency loss, using the following 3DGS loss function Optimize 3D Gaussian splash model: ； represents the weight coefficient, Refers to the structural similarity index.

Images