Multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraint

Abstract

This invention specifically relates to a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints, belonging to the field of image generation technology. This invention takes a single-view image as input, retrieves reference images from a 2D image library to construct retrieval enhancement conditions, and supplements prior details. It constructs a cross-view mapping relationship through monocular depth estimation and camera parameters, converting reprojection errors into geometric consistency biases and injecting cross-view attention. Multi-view latent variables are generated in parallel during diffusion-based inverse denoising, and multi-view consistent images are obtained through feature interaction and decoding. This invention improves multi-view structure and texture consistency within a unified framework, alleviates detail drift and invisible region collapse, and is suitable for scenarios such as 3D reconstruction. The generated quality and stability are superior to existing methods.

Description

Technical Field

[0001] This invention belongs to the field of image generation technology, specifically relating to a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints. Background Technology

[0002] Currently, most multi-view image generation methods employ a diffusion model as the generation backbone, and improve consistency and detail quality through cross-view information interaction or external condition enhancement. On the one hand, some works introduce cross-view feature interaction during diffusion denoising, enabling multi-view samples to share information at the same denoising time step, thereby alleviating the problem of appearance inconsistency between viewpoints. For example, the paper "L. Hölleine et al. ViewDiff: 3D-Consistent Image Generation with Text-to-Image Models. in Proceedings of the 2024 IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 5043-5052." integrates an interaction module in U-Net to improve multi-view consistency, but when the proportion of occlusion, repetitive textures, and invisible regions increases, detail instability and cross-view texture drift may still occur. On the other hand, retrieval-enhanced generation provides external visual evidence for the generation process by retrieving reference images related to the input from a two-dimensional image library, thereby improving the generation reliability of long-tail concepts, rare materials, and complex textures. The paper "A. Blattmann et al. Retrieval-augmented diffusion models. in Proceedings of the 36th International Conference on Neural Information Processing Systems, 2022, pp. 15309-15324" proposes using the retrieval of neighboring samples as a condition to supplement local content details and enhance generation confidence. However, when directly using retrieved details for multi-view generation, the lack of cross-view constraints provided by camera geometry and depth / plane priors can easily lead to inconsistent interpretations of the same structure from different perspectives, thereby amplifying geometric inconsistencies and the risk of content collapse in invisible areas. Therefore, a unified solution is urgently needed: integrating retrieval enhancement and cross-view attention interaction within a diffusion joint denoising framework, and further utilizing camera geometry and depth / plane priors to impose interpretable constraints on cross-view information convergence, so as to achieve stable consistency in structure, texture, and identity information in multi-view outputs.

[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0004] This invention provides a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints, aiming to solve the problems of weak geometric consistency between viewpoints, detail and texture drift, and content collapse in invisible areas in existing multi-view diffusion generation.

[0005] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.

[0006] According to a first aspect of the present invention, a method for generating multi-view consistent images based on retrieval enhancement and cross-view attention constraints is provided, the method comprising: Step 1: Obtain the input image Obtain the target camera set ,in, For the number of target viewpoints, each target camera Includes external parameters and internal reference Input image Corresponding source camera Its external parameters are Internal reference is recorded as ; Step 2: Based on the input image From a two-dimensional image library retrieval Zhang related reference images ,in To determine the number of retrievals; local visual features are extracted from the reference image and mapped to a fixed-length feature sequence. And perform weighted fusion to form search enhancement conditions. , as the external condition input of the diffusion denoising network; Step 3: Process the input image Perform monocular depth estimation to obtain a depth map. Combined with source camera With target camera set Cross-view mapping relationships are constructed using backprojection, pose transformation, and reprojection operations. and based on Obtain cross-view mapping relationships between target viewpoints ; Step 4: Position each target camera Encoding as camera conditional embedding ; Step 5: Joint denoising multi-view diffusion generation, generated in parallel during the same diffusion reverse denoising process. Latent variables from a target perspective Furthermore, a cross-view attention module is introduced within the diffusion denoising network to enable information interaction between intermediate features from multiple perspectives. The conditions of the diffusion denoising network include the input image... Encoding features, camera conditions and enhanced search conditions ; Step 6: Geometric consistency bias construction and injection. Based on the cross-view mapping relationship constructed in Step 3, calculate the geometric consistency bias based on the reprojection error, and inject the geometric consistency bias into the cross-view attention calculation process to suppress feature convergence in non-geometrically corresponding regions. Step 7: During the training phase, the diffusion denoising network is optimized using a joint objective function; during the inference phase, the final latent variables are... By decoding with a decoder, a set of consistent images from multiple perspectives is obtained. .

[0007] In some exemplary implementations, the retrieval in step 2 employs a weighted approach of global embedding similarity and importance, with the retrieval weight defined as:

[0008] in, For global encoder, For similarity function, This is the temperature coefficient.

[0009] In some exemplary embodiments, the local features of the reference image in step 2 are resampled and mapped to a fixed-length sequence, satisfying:

[0010] in, For local feature extractors, For resampling mapping operators, This indicates splicing and merging.

[0011] In some exemplary embodiments, the cross-view mapping in step 3 is constructed by backprojection, pose transformation, and reprojection for the source view pixels. Defined as:

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] in, For the back projection operator, pixel position Depth value at that location, Source camera Internal reference, and Source camera The rotation matrix and translation vector; , , The first target camera The intrinsic parameters, rotation matrix, and translation vector. , , The first The intrinsic parameters, rotation matrix, and translation vector of each target camera; The three-dimensional point corresponding to the source viewpoint pixel. and The three-dimensional points are respectively in the 3D point at ... The target camera and the first The coordinates of the target camera in the coordinate system For perspective projection operators, and The three-dimensional points are respectively in the 3D point at ... The first target perspective and the first Reprojection pixel position in the target viewpoint Indicates by the first From the perspective of the first target Cross-view mapping of target perspectives.

[0018] In some exemplary embodiments, the diffusion inverse denoising in step 5 is updated in parallel across all viewpoints as follows:

[0019] in, For the first The first noise reduction moment Latent variables corresponding to each target perspective For noise prediction networks, For the input image encoder, For target camera Conditional embedding, For the first The noise scheduling coefficient at each moment. It is standard Gaussian noise.

[0020] In some exemplary embodiments, the cross-view attention module in step 6 is calculated in the following form:

[0021]

[0022] in, This represents the intermediate features from the corresponding perspective. It is a linear mapping matrix. For channel dimension, This is a geometric consistency bias.

[0023] In some exemplary embodiments, the geometric consistency bias is constructed from the reprojection error:

[0024] in, This represents the corresponding position obtained through cross-view mapping. The weighting coefficients for geometric consistency bias.

[0025] In some exemplary implementations, a joint objective function is used for optimization during the training phase, satisfying:

[0026] in:

[0027]

[0028]

[0029] in, To predict the loss for diffused noise, For multi-view consistency loss, To enhance the constraint loss for retrieval; and For feature extraction networks, and The first The first target perspective and the first Generate images from a target viewpoint. Indicates the diffusion time step With Gaussian noise Expectations and These are the weight parameters.

[0030] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints described in the first aspect.

[0031] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints described in the first aspect above.

[0032] The multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints provided by the embodiments of the present invention introduces retrieval enhancement conditions to provide external visual references for the multi-view diffusion generation process, effectively improving the stability of complex textures and details in multi-view generation; it introduces a cross-view attention mechanism in the diffusion reverse denoising process, enabling multiple views to share intermediate feature information at the same time step, reducing appearance inconsistencies and texture drift between views; it constructs cross-view mappings based on camera geometry and depth or plane priors, and transforms reprojection errors into attention bias injection feature interaction processes, thereby enhancing the geometric consistency and interpretability of multi-view generation results from a mechanistic perspective; and it collaboratively models retrieval enhancement and geometric consistency constraints under a unified diffusion framework, enabling the method to have good stability and generalization ability in multi-view consistent image generation tasks.

[0033] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description

[0034] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0035] Figure 1 This is a schematic diagram of the structure of the multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints of the present invention.

[0036] Figure 2 This is a schematic diagram illustrating the construction of enhanced search conditions according to the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the cross-view attention and geometric consistency constraints of the present invention.

[0038] Figure 4 This is a schematic diagram of the multi-view consistent image generation output result of the present invention. Detailed Implementation

[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0040] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0041] To overcome the shortcomings of existing multi-view diffusion generation methods in terms of cross-view consistency and detail stability, this invention proposes a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints. This method enhances the expression of long-tailed details and complex materials by retrieving reference images related to the input from a 2D image library and constructing a retrieval feature sequence. By introducing a cross-view attention module during diffusion inverse denoising, multiple views share appearance and identity information at the same time step, thus suppressing identity and texture drift between views. Furthermore, a cross-view mapping is constructed based on camera geometry and depth or plane assumptions, transforming reprojection errors into attention biases or weights, imposing interpretable geometric consistency constraints on cross-view feature interactions. Camera scale uniformity processing reduces the impact of monocular depth scale ambiguity on cross-view deformation. Combined with joint training constraints and consistency modulation strategies, this method can still achieve structurally and texturally stable multi-view generation results even under weak supervision or small-sample multi-view data conditions.

[0042] This invention provides a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints. Using a single-view input image as a condition, it integrates retrieval enhancement features and a cross-view attention mechanism of camera geometric consistency modulation during diffusion inverse denoising, achieving stable and consistent generation of multi-target view images at the appearance, structure, and texture levels. (Reference) Figure 1-4 As shown, the specific steps include the following: Step 1: Obtain input image and target viewpoint information. Obtain the target camera set ,in, For the number of target viewpoints, each target camera Includes external parameters and internal reference The input image Corresponding source camera Its external parameters are Internal reference is recorded as ; Step 2: Constructing enhanced search criteria based on the input image From a two-dimensional image library retrieval Zhang related reference images ,in To determine the number of retrievals; local visual features are extracted from the reference image and mapped into a fixed-length feature sequence. And perform weighted fusion to form search enhancement conditions. , as the external condition input of the diffusion denoising network; The retrieval weight is calculated as follows:

[0043] in, For global encoder, For similarity function, Let be the temperature coefficient. Local visual features are extracted from the reference image and mapped to a fixed-length feature sequence, satisfying:

[0044] in, For local feature extractors, For resampling mapping operators, This indicates splicing and merging.

[0045] Step 3: Geometric priors and cross-viewpoint mapping construction for the input image. Perform monocular depth estimation to obtain a depth map. For source view pixels definition:

[0046]

[0047]

[0048]

[0049]

[0050]

[0051] in, For the back projection operator, pixel position Depth value at that location, Source camera Internal reference, and Source camera The rotation matrix and translation vector; , , The first target camera The intrinsic parameters, rotation matrix, and translation vector. , , The first The intrinsic parameters, rotation matrix, and translation vector of each target camera; The three-dimensional point corresponding to the source viewpoint pixel. and The three-dimensional points are respectively in the 3D point at ... The target camera and the first The coordinates of the target camera in the coordinate system For perspective projection operators, and The three-dimensional points are respectively in the 3D point at ... The first target perspective and the first Reprojection pixel position in the target viewpoint Indicates by the first From the perspective of the first target Cross-view mapping of target perspectives.

[0052] Step 4: Camera condition embedding and scale consistency, embedding each target camera Encoding as camera conditional embedding ; Step 5: Joint denoising multi-view diffusion generation, during the diffusion-based inverse denoising process, updating the latent variables of each viewpoint in parallel:

[0053] in, For the first The first noise reduction moment Latent variables corresponding to each target perspective For noise prediction networks, For the input image encoder, For target camera Conditional embedding, For the first The noise scheduling coefficient at each moment. It is standard Gaussian noise.

[0054] Step 6: Cross-view attention and geometric consistency modulation for cross-view intermediate features Constructing attention computation:

[0055]

[0056] in, It is a linear mapping matrix. For channel dimension, The geometric consistency bias is constructed from the reprojection error:

[0057] Step 7: Optimize the model using a joint objective function during the training phase:

[0058] in:

[0059]

[0060]

[0061] in, To predict the loss for diffused noise, For multi-view consistency loss, To enhance the constraint loss for retrieval; and For feature extraction networks, and The first The first target perspective and the first Generate images from a target viewpoint. Indicates the diffusion time step With Gaussian noise Expectations and These are the weight parameters.

[0062] In the reasoning stage, the final latent variables will be... By decoding, a set of consistent images from multiple perspectives is obtained. .

[0063] Example 1 Step 1: Obtain the input image Obtain the target camera set ,in, For the number of target viewpoints, each target camera Includes external parameters and internal reference The input image Corresponding source camera Its external parameters are Internal reference is recorded as In one specific embodiment, a configuration is set. The target viewpoint covers the front view, side view, rear view, and another side view, which is used to form a consistent multi-view generation result.

[0064] Step 2: Based on the input image From a two-dimensional image library Search results Zhang related reference images In one specific embodiment, the following is set Global embeddings are extracted from the input image and the reference image, and their similarity is calculated. The retrieval weight is then calculated based on a weighted approach that combines global embedding similarity with importance. Defined as:

[0065] in, For global encoder, For similarity function, The temperature coefficient is then used. Local visual features are subsequently extracted from each reference image and resampled and mapped to a fixed-length feature sequence to form retrieval enhancement conditions. ,satisfy:

[0066] in, For local feature extractors, For resampling mapping operators, This indicates splicing and merging.

[0067] Step 3: Constructing geometric priors and implicit cross-view correspondences for the input image. Perform monocular depth estimation to obtain a depth map. For source view pixels definition:

[0068]

[0069]

[0070]

[0071]

[0072]

[0073] in, For the back projection operator, pixel position Depth value at that location, Source camera Internal reference, and Source camera The rotation matrix and translation vector; , , The first target camera The intrinsic parameters, rotation matrix, and translation vector. , , The first The intrinsic parameters, rotation matrix, and translation vector of each target camera; The three-dimensional point corresponding to the source viewpoint pixel. and The three-dimensional points are respectively in the 3D point at ... The target camera and the first The coordinates of the target camera in the coordinate system For perspective projection operators, and The three-dimensional points are respectively in the 3D point at ... The first target perspective and the first Reprojection pixel position in the target viewpoint Indicates by the first From the perspective of the first target Cross-view mapping of target perspectives.

[0074] Step 4: Camera condition embedding and scale consistency, embedding each target camera Encoding as camera conditional embedding .

[0075] Step 5: Generate in parallel during the same diffusion reverse denoising process Latent variables from a target perspective The back-diffusion and parallel updates from each perspective are as follows:

[0076] in, For the first The first noise reduction moment Latent variables corresponding to each target perspective For noise prediction networks, For the input image encoder, For target camera Conditional embedding, For the first The noise scheduling coefficient at each moment. The noise is standard Gaussian noise. In one specific embodiment, the number of diffusion inference steps is set to 50.

[0077] Step 6: Cross-view attention and geometric consistency modulation for cross-view intermediate features Constructing attention computation:

[0078]

[0079] in, It is a linear mapping matrix. For channel dimension, The geometric consistency bias is constructed from the reprojection error:

[0080] Step 7: Optimize the model using a joint objective function during the training phase:

[0081] Among them, the diffusion noise prediction loss, the multi-view consistency constraint, and the retrieval consistency constraint are as follows:

[0082]

[0083]

[0084] In the reasoning stage, the final latent variables will be... By decoding, a set of consistent images from multiple perspectives is obtained. . Figure 4 Several examples of inputs and multi-view consistent output generation are given.

[0085] This invention discloses a multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints, addressing the problems of weak geometric consistency between viewpoints, detail and texture drift, and content collapse in invisible areas in existing multi-view diffusion generation methods. This method takes a single-view input image as a condition, first retrieving relevant reference images from a 2D image library and constructing a retrieval feature sequence to form retrieval enhancement conditions to supplement detail priors; then, performing monocular depth estimation on the input image and constructing a cross-view mapping relationship in conjunction with camera parameters, converting reprojection errors into geometric consistency biases and injecting them into cross-view attention calculations; during diffusion inverse denoising, multiple target viewpoint latent variables are generated in parallel, and multi-view feature interaction is achieved through cross-view attention, ultimately decoding to obtain a set of multi-view consistent images. Compared with existing technologies, this invention simultaneously improves multi-view structural consistency and texture stability within a single framework, reduces the risk of content collapse in invisible areas, and is suitable for applications such as 3D reconstruction.

[0086] The method of the present invention is further illustrated below through simulation: 1. Simulation conditions The method of this invention is a simulation performed using Anaconda software on a CPU with an Intel Core i7-9750H CPU, 32GB of memory, an Nvidia RTX3090 graphics card, and a Windows 10 operating system.

[0087] 2. Simulation Content The simulation uses publicly available ABO and GSO datasets. A single-view image is selected as input for each sample, and the following settings are configured: Each target perspective is output, and the output resolution is standardized to ensure fairness in comparing different methods.

[0088] To demonstrate the effectiveness of the invention method, Zero123, SyncDreamer, ViewFusion, Realfusion, Wonder3D, Era3D, and MV-AR were selected as comparison algorithms. Zero123 was proposed in the paper “R. Liu, R. Wu, B. Van Hoorick, P. Tokmakov, S. Zakharov, and C. Vondrick. Zero-1-to-3: Zero-shot One Image to 3D Object. in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2023, pp. 9264-9275.”; SyncDreamer was proposed in the paper “Y. Liu, C. Lin, Z. Zeng, X. Long, L. Liu, T. Komura, and W. Wang. SyncDreamer: Generating Multiview-consistent Images from a Single-view Image. in Proceedings of the International Conference on Representation Learning, 2024, pp. 27676-27697.”; and ViewFusion was proposed in the paper “X. Yang, Y. Zuo, S. Ramasinghe, L. Bazzani, G. Avraham, and A. van den Hengel. ViewFusion:Towards Multi-View Consistency via Interpolated Denoising. in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2024, pp.9870-9880." Proposed; RealFusion is in the document "L. Melas-Kyriazi, I. Laina, C.Rupprecht, and A. Vedaldi. RealFusion 360° Reconstruction of Any Object from a Single Image.in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2023, pp. 8446-8455." Wonder3D was proposed in the document "X. Longet al. Wonder3D: Single Image to 3D Using Cross-Domain Diffusion. 9970-9980."; Era3D was proposed in the literature "P. Li, Y. Liu, Processing Systems, 2024, pp. The method described in this invention is described in 1-26. MV-AR was proposed in the paper "JK Hu, YX Yang, JL Liu, JB Wu, C. Zhao, and YY Lu. Auto-Regressively Generating Multi-View Consistent Images. in Proceedings of the IEEE / CVF International Conference on Computer Vision, 2025.". Ours represents the results obtained using the method described in this invention. PSNR, SSIM, and LPIPS are image quality evaluation metrics, and the comparison results are shown in Table 1.

[0089] As shown in Table 1, on the two datasets, the multi-view consistency generation performance of the present invention is significantly better than other comparative algorithms. The higher PSNR, SSIM and LPIPS evaluation results indicate that the results generated by Ours have better multi-view consistency.

[0090] Table 1

[0091] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0092] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.

[0093] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.

Claims

1. A multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints, characterized in that, The method includes: Step 1: Obtain the input image Obtain the target camera set ,in, For the number of target viewpoints, each target camera Includes external parameters and internal reference Input image Corresponding source camera Its external parameters are Internal reference is recorded as ; Step 2: Based on the input image From a two-dimensional image library retrieval Zhang related reference images ,in To determine the number of retrievals; local visual features are extracted from the reference image and mapped to a fixed-length feature sequence. And perform weighted fusion to form search enhancement conditions. , as the external condition input of the diffusion denoising network; Step 3: Process the input image Perform monocular depth estimation to obtain a depth map. Combined with source camera With target camera set The input image is constructed using backprojection, pose transformation, and reprojection operations. The pixel correspondences of each target viewpoint are determined, and the cross-viewpoint mapping relationship between target viewpoints is obtained based on the pixel correspondences. ;in, Indicates the first A target perspective index Indicates the relationship with the first Index of another target perspective, different from the target perspective. Number of target viewpoints; Step 4: Position each target camera Encoding as camera conditional embedding ; Step 5: Joint denoising multi-view diffusion generation, generated in parallel during the same diffusion reverse denoising process. Latent variables from a target perspective Furthermore, a cross-view attention module is introduced within the diffusion denoising network to enable information interaction between intermediate features from multiple perspectives. The conditions of the diffusion denoising network include the input image... Encoding features, camera conditions and enhanced search conditions ; Step 6: Geometric consistency bias construction and injection. Based on the cross-view mapping relationship constructed in Step 3, calculate the geometric consistency bias based on the reprojection error, and inject the geometric consistency bias into the cross-view attention calculation process to suppress feature convergence in non-geometrically corresponding regions. Step 7: During the training phase, the diffusion denoising network is optimized using a joint objective function; during the inference phase, the final latent variables are... By decoding with a decoder, a set of consistent images from multiple perspectives is obtained. .

2. The method according to claim 1, characterized in that, Step 2's retrieval uses a weighted approach of global embedding similarity and importance, with the retrieval weight defined as: in, For global encoder, For similarity function, This is the temperature coefficient.

3. The method according to claim 1, characterized in that, In step 2, the local features of the reference image are resampled and mapped into a fixed-length sequence, satisfying: in, For local feature extractors, For resampling mapping operators, This indicates splicing and merging.

4. The method according to claim 1, characterized in that, Step 3, the cross-view mapping, is constructed by backprojection, pose transformation, and reprojection for the source view pixels. Defined as: in, For the back projection operator, pixel position Depth value at that location, Source camera Internal reference, and Source camera The rotation matrix and translation vector; , , The first target camera The intrinsic parameters, rotation matrix, and translation vector. , , The first The intrinsic parameters, rotation matrix, and translation vector of each target camera; The three-dimensional point corresponding to the source viewpoint pixel. and The three-dimensional points are respectively in the 3D point at ... The target camera and the first The coordinates of the target camera in the coordinate system For perspective projection operators, and The three-dimensional points are respectively in the 3D point at ... The first target perspective and the first Reprojection pixel position in the target viewpoint Indicates by the first From the perspective of the first target Cross-view mapping of target perspectives.

5. The method according to claim 1, characterized in that, The diffusion-backward denoising process in step 5 is updated in parallel across all viewpoints as follows: in, For the first The first noise reduction moment Latent variables corresponding to each target perspective For noise prediction networks, For the input image encoder, For target camera Conditional embedding, For the first The noise scheduling coefficient at each moment. It is standard Gaussian noise.

6. The method according to claim 1, characterized in that, The cross-perspective attention module in step 6 is calculated in the following form: in, This represents the intermediate features from the corresponding perspective. It is a linear mapping matrix. For channel dimension, This is a geometric consistency bias.

7. The method according to claim 6, characterized in that, The geometric consistency bias is constructed from the reprojection error: in, This represents the corresponding position obtained through cross-view mapping. The weighting coefficients for geometric consistency bias.

8. The method according to claim 1, characterized in that, During the training phase, a joint objective function is used for optimization, satisfying: in: in, To predict the loss for diffused noise, For multi-view consistency loss, To enhance the constraint loss for retrieval; and For feature extraction networks, and The first The first target perspective and the first Generate images from a target viewpoint. Indicates the diffusion time step With Gaussian noise Expectations and These are the weight parameters.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints as described in any one of claims 1 to 8.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the multi-view consistent image generation method based on retrieval enhancement and cross-view attention constraints as described in any one of claims 1 to 8.

Images