Multi-view image synthesis method, storage medium, electronic equipment and device

Abstract

The invention relates to a multi-view image synthesis method, a storage medium, electronic equipment and a device, and relates to the technical field of image synthesis, and the method comprises the steps: obtaining a corresponding potential space representation based on an input view image and a plurality of target multi-view images; random noise with the same scale as the target multi-view image is added into the potential space representation corresponding to the target multi-view image, and the potential space representation after diffusion is obtained; obtaining a corresponding reference condition based on the input visual angle image and internal and external parameters of the camera; combining the potential space representation of the input view angle image and the diffused potential space representation in a channel dimension, and inputting the combined potential space representation and the diffused potential space representation into a preset prediction noise model in combination with a reference condition to obtain a multi-view angle image; and performing 3D model synthesis based on the multi-view image. According to the method, 3D model synthesis is performed based on the input view angle image and the target multi-view angle image in combination with the random noise and the prediction noise model, so that the consistency and the generation efficiency of multi-view generation are effectively improved.

Description

Technical Field

[0001] The present application relates to the technical field of image synthesis, and in particular to a multi-view image synthesis method, storage medium, electronic equipment and device. Background Art

[0002] The current technical means have technical defects when generating multi-view images, such as: insufficient view consistency. When generating images of multiple viewpoints from a single viewpoint, existing methods often find it difficult to maintain a high degree of view consistency, resulting in the generated images being visually incoherent and lacking in realism. The generation efficiency is low. The synthesis of multi-view images often takes a long time, which becomes a significant bottleneck in application scenarios that require rapid image generation. Overfitting to a specific viewpoint. Some existing technologies perform well when generating images of a specific viewpoint, but perform poorly when generating images with a wide distribution of viewpoints, which limits their application in complex scenarios.

[0003] Therefore, in order to solve the above technical problems, a multi-view image synthesis technology is now provided. Summary of the invention

[0004] The present application provides a multi-view image synthesis method, storage medium, electronic device and apparatus, which performs 3D model synthesis based on input view images and target multi-view images in combination with random noise and predicted noise models, effectively improving the consistency and generation efficiency of multi-view generation.

[0005] In a first aspect, the present application provides a multi-view image synthesis method, the method comprising the following steps: Based on the input view image and multiple target multi-view images, the corresponding latent space representation is obtained; Adding random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain a diffused latent space representation; Based on the input viewing angle image and the internal and external parameters of the camera, a corresponding reference condition is obtained; Merging the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combining the reference condition and inputting them into a preset prediction noise model to obtain a multi-view image; Based on the multi-view images, 3D model synthesis is performed.

[0006] On the basis of the above technical solution, the input view image is passed through a preset pre-trained image-text feature alignment model to obtain CLIP embedding features; The CLIP embedded features are combined with the camera internal and external parameters to obtain corresponding reference conditions.

[0007] On the basis of the above technical solution, the noise predicted by the prediction noise model is obtained, and the distance between the noise and the random noise is calculated to obtain a loss value; The prediction noise model is back-propagated and optimized based on the loss value.

[0008] On the basis of the above technical solution, the camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

[0009] In a second aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a multi-view image synthesis method mentioned in the first aspect.

[0010] In a third aspect, the present application provides an electronic device including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, a multi-view image synthesis method mentioned in the first aspect is implemented.

[0011] In a fourth aspect, the present application provides a multi-view image synthesis device, the device comprising: A latent space representation module, which is used to obtain a corresponding latent space representation based on an input view image and multiple target multi-view images; A diffusion module, which is used to add random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain a diffused latent space representation; A reference condition generation module, which is used to obtain corresponding reference conditions based on the input perspective image and camera internal and external parameters; A multi-view image generation module, which is used to merge the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combine the reference condition and input them into a preset prediction noise model to obtain a multi-view image; The 3D model synthesis module is used to synthesize the 3D model based on the multi-view images.

[0012] On the basis of the above technical solution, the reference condition generation module is further used to obtain CLIP embedding features by subjecting the input view image to a preset pre-trained image-text feature alignment model; The reference condition generation module is further used to merge the CLIP embedding feature with the camera internal and external parameters to obtain corresponding reference conditions.

[0013] On the basis of the above technical solution, the device further comprises: A model optimization module, which is used to obtain the noise predicted by the prediction noise model, perform distance calculation with the random noise, and obtain a loss value; The model optimization module is also used to perform back-propagation optimization on the prediction noise model based on the loss value.

[0014] On the basis of the above technical solution, the camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

[0015] The beneficial effects of the technical solution provided by this application include: This application performs 3D model synthesis based on input view images and target multi-view images, combined with random noise and predicted noise models, effectively improving the consistency and generation efficiency of multi-view generation. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 A schematic diagram of the principle of the multi-view image synthesis method provided in an embodiment of the present application; Figure 2 A schematic diagram of the principle of epipolar geometric relationship in the multi-view image synthesis method provided in an embodiment of the present application; Figure 3 Schematic diagram of the principle of the extreme constrained attention block in the multi-view image synthesis method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0019] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0020] The embodiments of the present application provide a multi-view image synthesis method, storage medium, electronic device and apparatus, which perform 3D model synthesis based on input view images and target multi-view images in combination with random noise and predicted noise models, thereby effectively improving the consistency and generation efficiency of multi-view generation.

[0021] In order to achieve the above technical effects, the overall idea of ​​this application is as follows: A multi-view image synthesis method, the method comprising the following steps: S1, based on the input perspective image and multiple target multi-perspective images, obtain the corresponding latent space representation; S2, adding random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain the latent space representation after diffusion; S3, obtaining corresponding reference conditions based on the input perspective image and the internal and external parameters of the camera; S4, merging the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combining with the reference condition, inputting them into a preset prediction noise model to obtain a multi-view image; S5. Perform 3D model synthesis based on multi-view images.

[0022] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0023] First, see Figures 1 to 3 As shown, the embodiment of the present application provides a multi-view image synthesis method, the method comprising the following steps: S1, based on the input perspective image and multiple target multi-perspective images, obtain the corresponding latent space representation; S2, adding random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain the latent space representation after diffusion; S3, obtaining corresponding reference conditions based on the input perspective image and the internal and external parameters of the camera; S4, merging the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combining with the reference condition, inputting them into a preset prediction noise model to obtain a multi-view image; S5. Perform 3D model synthesis based on multi-view images.

[0024] It should be noted that 3D reconstruction from a single image is an important and challenging task in fields such as augmented reality, virtual reality, robotics simulation, and gaming. The complexity of this task lies in the need to accurately interpret spatial structures and details from a 2D perspective, which requires extensive knowledge of the 3D world.

[0025] Recently, the field of 3D generation has developed rapidly with the introduction of diffusion models. A series of studies have attempted to train 3D diffusion models from scratch. However, due to the limited size of publicly available 3D datasets, these methods show poor generalization ability. Therefore, another group of works proposed to use the prior knowledge of 2D diffusion models to create 3D models from text or images through score distillation sampling (SDS). Although these methods have achieved remarkable results, they require time-consuming optimization for each scene.

[0026] More recent work generates new perspective images by fine-tuning a pre-trained 2D image diffusion model on a 3D dataset, which can be used for 3D shape and appearance recovery. This feed-forward process not only retains the generalization ability of the image diffusion model, but also allows for faster synthesis of multi-perspective images. The Zero123 method proposes a viewpoint-conditioned diffusion model that embeds viewpoint features and camera information, but the multi-perspective consistency of the generated image is limited. In order to improve multi-perspective consistency, more recent work, which is also the closest prior art to the embodiments of the present application, uses a diffusion model based on a global explicit three-dimensional representation to generate multi-perspective images. This type of method introduces a 3D feature volume constructed from global multi-perspective images, so that the model can obtain projection features in three-dimensional space, thereby improving perspective consistency. However, this heavy global feature volume modeling not only slows down the generation process, but also easily overfits to a specific viewpoint, resulting in distorted and blurred images.

[0027] Existing 3D generation techniques, such as diffusion models that directly generate 3D assets (such as point clouds, meshes, neural radiation fields), face the challenge of relatively small publicly accessible 3D datasets, resulting in limited generalization capabilities. These methods can usually only generate shapes of specific categories and have difficulties in adapting to more complex input conditions. On the other hand, some studies guide 3D generation through pre-trained 2D image diffusion models, such as using scene optimization processes to generate text- or image-based 3D representations. However, this per-scene optimization method has efficiency issues, and generating a single scene may take tens of minutes or even hours.

[0028] In addition, in terms of generating multi-view images from a single view, although some methods integrate 3D representations for multi-view synthesis, they lack design considerations for 3D reconstruction. For example, Zero123 introduced relative viewpoint conditions into the 2D diffusion model, allowing new views to be generated from a single image, but due to limited consistency, 3D reconstruction was achieved using SDS loss. Recent work has introduced 3D global feature volumes in the multi-view generation process to ensure consistency, but heavy global 3D representation modeling not only slows down the generation speed, but also tends to produce low-quality images.

[0029] The purpose of the embodiments of the present application is to solve the above-mentioned problems by quickly generating consistent and high-quality multi-view images through a local modeling three-dimensional representation method, thereby improving the quality and efficiency of multi-view image synthesis while ensuring good perspective consistency.

[0030] The novel method proposed in the embodiment of the present application is based on the epipolar constrained attention block (ECA Block, ECA stands for Efficient Channel Attention), which solves the shortcomings of the above-mentioned prior art and improves the efficiency and quality of multi-view image synthesis.

[0031] In the embodiment of the present application, 3D model synthesis is performed based on the input view image and the target multi-view image in combination with random noise and predicted noise models, effectively improving the consistency and generation efficiency of multi-view generation.

[0032] Further, the input view image is passed through a preset pre-trained image-text feature alignment model to obtain CLIP embedding features; The CLIP embedded features are combined with the camera internal and external parameters to obtain corresponding reference conditions.

[0033] Further, the noise predicted by the prediction noise model is obtained, and a distance calculation is performed between the noise and the random noise to obtain a loss value; The prediction noise model is back-propagated and optimized based on the loss value.

[0034] Furthermore, the camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

[0035] Based on the technical solution of the embodiment of the present application, a specific description is given: Given an input view of an object, the technical goal is to generate multi-view images of the object. The goal of the embodiment of the present application is to learn a model that accepts input views and synthesizes multi-view images under specified camera transformations.

[0036] The specific operation process of the embodiment of the present application is as follows Figure 1 The 3D model training data sample of the embodiment of the present application includes the input view image of the model (Input view ), N target multi-view images (Target views ), and the camera internal and external parameters for rendering these views.

[0037] In the training phase, first, the input view image and N target multi-view images are input to the encoder module of the variational autoencoder ( ), and their latent space representations are obtained respectively. and ; Then sample random noise of the same scale , added to the latent space representation of the target multi-view image In the process, this process is called diffusion. Then, the potential representation of the input view image The target multi-view potential representation after adding noise is merged in the channel dimension and input into the prediction noise model UNet module; At the same time, as an input condition, the input view image ) also uses the pre-trained image-text feature alignment model CLIP (Contrastive Language-Image Pre-training) to obtain the CLIP embedding feature of the image, which is consistent with the camera's internal and external parameters ( ) are merged and injected into the prediction noise model UNet through the cross attention mechanism as a reference condition; finally, the distance between the noise predicted by the model and the added noise is calculated as the loss, and the model is back-propagated and optimized.

[0038] In the inference stage, unlike the training stage, there is no true value of the target multi-view image. Instead, random variables of the same scale as the multi-view image are directly sampled and input into the noise prediction model. After T steps of predicted noise, the potential representation of the image is gradually obtained from the random variables, and then the multi-view image is obtained through the decoder module of the variational autoencoder.

[0039] To improve the consistency, quality and efficiency of multi-view generation, a novel noise prediction model is designed, and a local interactive multi-view diffusion scheme based on a novel view synthesis base model is introduced. A lightweight attention block is designed in the original feature space of the diffusion model to establish nearby view associations to model 3D representations.

[0040] The technical solution of the embodiment of the present application involves the interaction process between the intermediate features of multi-view images, thereby extending single-view synthesis to multi-view synthesis. Specifically, a 3D modeling module is designed based on the attention mechanism, named the epipolar constraint attention module (ECA Block), and inserted into the basic diffusion model. The ECA block promotes the information exchange between the potential features of adjacent views and utilizes the epipolar constraints in stereo vision to enhance the consistency and quality of multi-view images. ECA Block is lightweight and highly adaptable, and is suitable for various basic diffusion models.

[0041] As shown in the accompanying drawings of the embodiments of the present application Figure 1As shown, the technical solution of the embodiment of the present application is based on the basic new perspective image synthesis model, and a module for modeling 3D consistency is designed, which is inserted into the midsampling and upsampling stages of UNet. The module is constructed using an attention network, which aims to learn generalized epipolar geometry and is called the epipolar constraint attention block (ECA Block). During training, only the parameters of the ECA block are updated, thereby preserving the feature space of the basic model and encouraging the module to extract 3D priors.

[0042] Specifically, the original noise prediction model only learns and models information in a single image through the residual network layer and the self-attention network layer. For the multi-view generation of multi-view consistency that this task hopes to solve, the association between multi-view information is not established. In order to add interaction between multiple views in the noise prediction model, a multi-view interaction module is added after the self-attention network layer, which is the ECA module (ECA Block) mentioned above. After the multi-view features or random variable features are input into the noise prediction model, the ECA module will build a geometric space association between the neighboring view features of the multi-image features, and use the cross-attention mechanism (Near-Views Cross-Attn) and the ray self-attention mechanism (Ray Self-Attn) at the relevant positions to realize the feature interaction of multi-view images in three-dimensional space. The features output by the multi-view features after the ECA module will be the features after the interaction between the images, that is, the features with spatial association. After the feature goes through the same steps as the original noise prediction model, the noise predicted will be the noise associated with multiple views, which plays an important role in the multi-view consistency between the multi-view images obtained in the end.

[0043] The technical solution of the embodiment of the present application has the following technical points: First, a diffusion model for synthesizing multi-view images: This diffusion model describes the diffusion model used to synthesize viewpoint images and establish associations between multiple images. Corresponding to the interactive multi-view diffusion scheme described in the previous question, it explains the framework of how to model multi-view consistency from a holistic perspective.

[0044] This multi-view diffusion model is based on the image diffusion model, especially the latent diffusion model (LDM). The latent diffusion model LDM is a stable image generation model, which is structurally composed of a variational autoencoder and a noise prediction model. Consistent with other generative learning methods, it is divided into two stages: perceptual compression and semantic compression. In the perceptual compression learning stage, the learning method encapsulates the data into an abstract representation by removing high-frequency details and projects high-dimensional redundant data from the pixel space to a hyperspace called the latent space. Specifically, the autoencoder (AE) structure is a structure that captures perceptual compression. The encoder in the AE projects the high-dimensional data into the latent space, and the decoder recovers the image from the latent space; in the semantic compression stage, it captures the semantic structure present in the data. This concept and semantic structure preserve the context and mutual relationship of various objects in the image. The above is an overview of the general generative learning method process. For the latent diffusion model LDM, its perceptual compression is accomplished by a variational autoencoder. The autoencoder in the LDM captures the perceptual structure of the data by projecting the data into the latent space. The proposer uses a special loss function to train this autoencoder, called "perceptual loss". This loss function ensures that the reconstruction is confined to the image shape and reduces the blur that may occur when using pixel space loss; then, it learns the data distribution by gradually removing the noise in the normally distributed variables. LDM uses the UNet network structure as a noise prediction model to predict the noise gradually added to the latent image and gradually denoise it, thereby restoring the image. In general, LDM performs a denoising process in the latent space of the autoencoder. During the training phase, the input image is first mapped to the latent space through a fixed encoder and then perturbed through a predefined Markov process.

[0045] More relevant to the above functions is the Zero123 model. As the basic model of Novel View Synthesis (NVS), Zero123 is also based on the latent diffusion model. On its basis, it adds input view and camera parameters as conditions, and extends single image synthesis to the task of generating new view images with controllable camera view. This model is built on the pre-trained image diffusion model and adds camera control to optimize specific goals.

[0046] Since the above-mentioned latent diffusion model and Zero123 new perspective image generation diffusion model all generate single images without establishing the association between multiple images, the generation of three-dimensional models and three-dimensional scenes requires the establishment of multi-perspective consistency between multiple views in order to synthesize reasonable multi-perspective images. Therefore, based on the latent diffusion model of a single image or a single view, a multi-perspective diffusion model framework is proposed. This framework is based on the new perspective synthesis diffusion model and promotes the spatial interaction of images at the latent feature level. The 3D geometric meaning is incorporated into the gradual denoising process to generate multi-perspective images with 3D consistency.

[0047] It is hoped that the noise prediction model UNet network can be improved by adding a feature interaction module between views to achieve modeling of multi-view images. Specifically, in addition to the module where the features of multiple images learn their own semantics, a module is added to allow multiple images to learn interactively. This module uses the three-dimensional spatial geometric relationship as a guide, uses the three-dimensional geometric association between the pixels of the known multi-view images, and uses the attention mechanism to make them learn from each other and model interactive associations. This module is called the three-dimensional consistency modeling module. During the training phase, a set of multi-view images and their related camera extrinsics in the dataset are selected for fine-tuning to learn three-dimensional consistency. A general 3D consistency module is introduced in the UNet of the base model, which is inserted into the middle and each upsampling block of the UNet, allowing multi-view image feature maps to interact in 3D space and create consistent connections. This integration transforms the process of generating multiple images from an independent process to a synchronous interactive process. The final denoising network is trained on a dataset containing multi-view images and camera extrinsics.

[0048] By freezing the base model and training only the 3D modeling module, the original generative capability of the base model is maintained. This additional module, which models multi-view associations, enhances the ability of the base diffusion model to generate images with multi-view consistency. The designed multi-view diffusion framework is applicable to various base diffusion models.

[0049] After training the newly inserted 3D consistency modeling module, the multi-view features or multi-view random variable features are input into the novel UNet network. In the noise prediction model, these features not only learn the semantics of the image itself, but also model the perspective association between multiple images. The multi-view noise output later will be a feature with multi-view consistency association. Finally, through step-by-step denoising, a multi-view image with multi-view consistency will be obtained.

[0050] Second, the polar constraint attention mechanism: This mechanism explains in detail the 3D consistency modeling module designed in the noise prediction model UNet network. This module corresponds to the epipolar constraint attention module (ECA Block) mentioned above.

[0051] In general, a new cross-view interaction module based on epipolar geometry principle, the epipolar constraint attention block (ECA Block), is inserted into the original noise prediction model UNet network. The ECA Block aims to identify the association between the potential features of the current view and the corresponding features along the epipolar lines in other views.

[0052] The technical background of epipolar geometry (also called epipolar geometry) is described as follows: Epipolar geometry is an effective and condensed consistent spatial association established for multi-view images, as shown in the accompanying drawings of the specification. Figure 2 As shown, i1, i2 are the imaging surfaces of the left and right cameras; P is called the polar plane, which is the plane composed of C1C2X; L1 and L2 are called epipolar lines, which are the projections of rays R1 and R2 on the imaging surfaces of each other's cameras. They can also be called epipolar lines, or they can be understood as the intersection of the epipolar planes and the imaging surfaces of the two cameras. e1 and e2 are called poles, which are the projections of the cameras on each other’s imaging planes. They can also be understood as the pole planes and the imaging planes of the two cameras. B is called the baseline, which is the distance between the two cameras.

[0053] From the attached drawings of the specification Figure 2 It can be seen that the epipolar planes intersect at the baseline, and all the epipolar lines intersect at the poles. These constitute epipolar geometry / epipedal geometry. The core of epipolar geometry is to have epipolar constraints: given a feature point on the left image, the matching point of this point on the right image must be on the corresponding epipolar line. The epipolar constraint gives important constraints on corresponding points. It compresses the matching of corresponding points from searching in the entire image to searching on a line, greatly reducing the search range and greatly improving the matching efficiency.

[0054] From the above description of epipolar geometric association, we can see that if we want to establish associations between multiple views, then, specifically, after the epipolar constraint is introduced, what is established is the association between the pixel block of view i1 and the pixel block on its epipolar line in view i2. Therefore, according to this principle, the ECA Block is designed to be established as a near-view cross-attention module and a ray self-attention module.

[0055] Nearby View Cross-Attention Module: For the target view, the processing method relies on generating an output feature map from the features of the nearest K views out of a total of N views. Using the camera parameters, rays are derived from patches on the feature map of the target view and the corresponding epipolar lines on the feature maps of the neighboring view images are calculated. Then, S points are uniformly sampled along each epipolar line and the feature map on the reference view is performed using bilinear interpolation to produce features in 3D space.

[0056] Ray Self-Attention Module: To simulate the contribution of spatial points to pixel-level features, self-attention is used to implement the ray aggregation module. After obtaining the feature representation of the target view through near-view cross-attention, self-attention is applied to the features along the ray. This step is crucial to reveal the deep relationship between the sampled points along the ray and their spatial correlation.

[0057] In addition, in order to enhance the model's understanding of perspective relationships and cognition of positions, two position encoding techniques are introduced.

[0058] Position encoding: To enhance the module's ability to understand perspective relationships, position encoding technology from the Light Field Network (LFN) is introduced. Each pixel is associated with a ray, which is represented by Plücker coordinates. A strategy from GPNR (General Purpose Neural Radiance Fields) is also adopted to establish a ray relative positioning system through the corresponding camera external parameters to ensure that the rays are consistently represented relative to the common reference system.

[0059] The above scheme design constitutes the ECA module. In general, as shown in the attached drawings of the specification Figure 3 As shown in Figure 1, for the current view feature (Target view), it is hoped to build the association between views from its neighboring K-1 adjacent view features (K-1 Reference views).

[0060] First, select a pixel block feature in the current view feature, send a ray from the sampling camera position, and get the imaging ray, project the ray onto the adjacent view to get the epipolar line on the adjacent view. First, establish a cross-attention mechanism for the S spatial point features sampled on the ray and the epipolar pixel features on the adjacent view.

[0061] Specifically, for each point s, the feature is obtained through position encoding and its features projected from neighboring views .Will As query features, the attention mechanism is implemented using the following formula :

[0062] In the synthesis process of the target view feature map, the features of nearby views that have a 3D geometric relationship with the target view are integrated, as shown in the above formula. This method effectively captures the feature correlation between views, thereby effectively enhancing the multi-view Figure 1 Then, after obtaining the features of each sample point, self-attention is applied along the ray of dimension S, from Query, key, and value features are derived from the 3D image. This step is crucial to reveal the depth relationship between sample points along the ray and capture their spatial correlation. Finally, a weighted sum is performed to fuse the 3D features to the target patch. 2D features.

[0063] Figure 3 In

[15] , latent features of multi-view images are associated in 3D space through two key attention blocks. The near-view cross-attention module initially aggregates features from nearby views onto ray points of the target view under the guidance of epipolar geometry. Subsequently, the ray self-attention module models the depth information to fuse ray features into 2D feature maps. The ECA block promotes multi-view interaction by effectively utilizing spatial geometry.

[0064] This design precisely aligns features within four resolution levels of the UNet latent space, maintaining the feature integrity of the base model using epipolar projection. The two-module division of the 3D attention process further allows for efficient modeling of multi-view consistency.

[0065] In summary, the key technical points of the embodiments of the present application are: (1) Integration of 3D modeling module: EpiDiff combines the 3D modeling module with a fixed diffusion model to maintain the original feature distribution and the compatibility of multiple diffusion models. This integration enhances the generalization and flexibility of the model, enabling it to adapt to different types of 3D assets and scenes.

[0066] (2) Epipolar Constrained Attention Block (ECA Block): ECA Block uses epipolar geometry to effectively learn the correlation between adjacent views. Its lightweight design and targeted optimization improve multi-view consistency while reducing the computational burden and optimizing the generation process.

[0067] (3) Superiority of the multi-view diffusion model: Experiments show that EpiDiff is significantly superior to baseline methods in terms of generation speed and image quality (PSNR, SSIM, LPIPS). It generates multiple high-quality views in a short time, enhancing the accuracy and reliability of 3D reconstruction.

[0068] Among them, PSNR, Peak Signal-to-Noise Ratio, peak signal-to-noise ratio; SSIM, Structural Similarity, structural similarity; LPIPS, Learned Perceptual Image Patch Similarity, learns perceptual image patch similarity.

[0069] (4) Integration and optimization of existing technologies: EpiDiff improves the efficiency and quality of the generation from text to 3D models by integrating and optimizing existing technologies, such as latent diffusion models and 3D generation methods.

[0070] (5) Overall technical optimization of generating three-dimensional models based on text description feedforward: This invention optimizes the process of directly generating multi-view images and three-dimensional models from text descriptions, enhances the generalization and application scope of the model, and provides strong support for solving efficiency, generalization and quality issues.

[0071] These key innovations constitute the core of the technical solution of the embodiment of the present application, and provide strong technical support for solving the efficiency, generalization and quality issues of generating multiple views from text descriptions and then synthesizing three-dimensional models.

[0072] Compared with the prior art, its main advantage is that it can more efficiently generate high-quality multi-view images with strong perspective consistency. The innovative application of the epipolar constraint attention block (ECA Block) and the effective use of epipolar geometric constraints in the technology proposed in the embodiment of the present application significantly improve the speed and quality of image generation. In addition, the technical solution of the embodiment of the present application can handle a wider range of perspective distributions, and is particularly suitable for complex 3D shape reconstruction. In terms of generation speed, image quality (such as PSNR, SSIM, LPIPS and other indicators) and degrees of freedom of perspective distribution, the embodiments of the present application have demonstrated significant technical advantages, making it have outstanding technical superiority in the field of multi-view image synthesis.

[0073] In addition, alternatives to the technical solution of the embodiment of the present application may include adopting different types of basic diffusion models or adjusting the structure of the ECA Block. In addition, it is also feasible to explore the use of other types of attention mechanisms or geometric constraint methods, which may provide different ways to improve model performance and applicability. These alternatives can provide diverse optimization paths based on specific application requirements or technical challenges. For example, you can consider adjusting the model structure for specific types of 3D assets or scenes, or exploring new geometric constraint techniques to enhance the generalization and adaptability of the model.

[0074] In a second aspect, the present application provides a storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements a multi-view image synthesis method mentioned in the first aspect.

[0075] In a third aspect, the present application provides an electronic device including a memory and a processor, wherein the memory stores a computer program running on the processor, and when the processor executes the computer program, a multi-view image synthesis method mentioned in the first aspect is implemented.

[0076] In a fourth aspect, an embodiment of the present application provides a multi-view image synthesis device, the device comprising: A latent space representation module, which is used to obtain a corresponding latent space representation based on an input view image and multiple target multi-view images; A diffusion module, which is used to add random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain a diffused latent space representation; A reference condition generation module, which is used to obtain corresponding reference conditions based on the input perspective image and camera internal and external parameters; A multi-view image generation module, which is used to merge the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combine the reference condition and input them into a preset prediction noise model to obtain a multi-view image; A 3D synthesis module is used to synthesize a 3D model based on the multi-view images.

[0077] In the embodiment of the present application, 3D model synthesis is performed based on the input view image and the target multi-view image in combination with random noise and predicted noise models, effectively improving the consistency and generation efficiency of multi-view generation.

[0078] Furthermore, the reference condition generation module is further used to pass the input view image through a preset pre-trained image-text feature alignment model to obtain CLIP embedding features; The reference condition generation module is further used to merge the CLIP embedding feature with the camera internal and external parameters to obtain corresponding reference conditions.

[0079] Furthermore, the device also includes: A model optimization module, which is used to obtain the noise predicted by the prediction noise model, perform distance calculation with the random noise, and obtain a loss value; The model optimization module is also used to perform back-propagation optimization on the prediction noise model based on the loss value.

[0080] Furthermore, the camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

[0081] It should be noted that the multi-view image synthesis device provided in the embodiment of the present application, and its corresponding technical problems, technical means and technical effects are similar to the principles of the multi-view image synthesis method in terms of principle.

[0082] It should be noted that, in this application, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0083] The above is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A multi-view image synthesis method, characterized in that: The method comprises the following steps: Based on the input view image and multiple target multi-view images, the corresponding latent space representation is obtained; Adding random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain a diffused latent space representation; Based on the input viewing angle image and the internal and external parameters of the camera, a corresponding reference condition is obtained; Merging the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combining the reference condition and inputting them into a preset prediction noise model to obtain a multi-view image; Based on the multi-view images, 3D model synthesis is performed.

2. The multi-view image synthesis method according to claim 1, wherein: Passing the input view image through a preset pre-trained image-text feature alignment model to obtain CLIP embedding features; The CLIP embedded features are combined with the camera internal and external parameters to obtain corresponding reference conditions.

3. The multi-view image synthesis method according to claim 1, wherein: Obtain the noise predicted by the prediction noise model, perform distance calculation on the noise and the random noise, and obtain a loss value; The prediction noise model is back-propagated and optimized based on the loss value.

4. The multi-view image synthesis method according to claim 1, wherein: The camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

5. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

6. An electronic device comprising a memory and a processor, wherein the memory stores a computer program that runs on the processor, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 4 is implemented.

7. A multi-view image synthesis device, characterized in that: The device comprises: A latent space representation module, which is used to obtain a corresponding latent space representation based on an input view image and multiple target multi-view images; A diffusion module, which is used to add random noise of the same scale as the target multi-view image to the latent space representation corresponding to the target multi-view image to obtain a diffused latent space representation; A reference condition generation module, which is used to obtain corresponding reference conditions based on the input perspective image and camera internal and external parameters; A multi-view image generation module, which is used to merge the latent space representation of the input view image and the latent space representation after diffusion in the channel dimension, and then combine the reference condition and input them into a preset prediction noise model to obtain a multi-view image; The 3D model synthesis module is used to synthesize the 3D model based on the multi-view images.

8. The multi-view image synthesis device according to claim 7, wherein: The reference condition generation module is also used to pass the input view image through a preset pre-trained image-text feature alignment model to obtain CLIP embedding features; The reference condition generation module is further used to merge the CLIP embedding feature with the camera internal and external parameters to obtain corresponding reference conditions.

9. The multi-view image synthesis device according to claim 7, characterized in that: The device also includes: A model optimization module, which is used to obtain the noise predicted by the prediction noise model, perform distance calculation with the random noise, and obtain a loss value; The model optimization module is also used to perform back-propagation optimization on the prediction noise model based on the loss value.

10. The multi-view image synthesis device according to claim 7, wherein: The camera intrinsic and extrinsic parameters are intrinsic and extrinsic parameters of a camera for obtaining the target multi-view image.

Images