Three-dimensional scene reconstruction method under sparse view angle based on multi-modal feature guidance

Through a multimodal guidance method that integrates image and text features from sparse perspectives, the problem that the prior art is difficult to achieve high-quality three-dimensional scene reconstruction under sparse perspectives is solved, and an efficient and robust three-dimensional scene reconstruction effect is achieved.

CN119672213AActive Publication Date: 2025-03-21GUANGDONG LAB OF ARTIFICIAL INTELLIGENCE & DIGITAL ECONOMY (SZ)
3 Cites 1 Cited by

Patent Information

Application Number
CN202411701347.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-26
Publication Date
2025-03-21
Estimated Expiration
2044-11-26

AI Technical Summary

Technical Problem

Existing deep learning-based three-dimensional reconstruction algorithms are difficult to achieve high-quality three-dimensional scene reconstruction under sparse perspective conditions, especially when objects that are incomplete or large-area occluded in the image exist.

Method used

Using a multimodal feature-guided method, by freezing the parameters of the initial image segmentation model and text encoder, the sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor are trained to obtain the preprocessing results and text description of the sparse view image, fuse the image and text features, and output the multimodal fusion features to reconstruct the three-dimensional scene.

Benefits of technology

High-quality three-dimensional scene reconstruction at sparse perspectives is realized, and the generated new perspective images have excellent visual quality and improve reconstruction efficiency and robustness.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention discloses a three-dimensional scene reconstruction method under a sparse view angle based on multi-modal feature guidance. The method comprises the following steps: training an initial sparse view angle image encoder, an initial multi-modal guidance module and an initial three-dimensional Gaussian parameter predictor; respectively inputting the sparse view angle image into a trained sparse view angle image encoder and an image segmentation model of the frozen parameters to obtain shallow image features and segmentation features, and inputting the sparse view angle text description into a text encoder of the frozen parameters to obtain text features; in the trained multi-modal guide module, optimizing and refining the shallow image features by using the segmentation features and the text features to obtain multi-modal fusion features; and reconstructing a three-dimensional scene according to the multi-modal fusion features by using the trained three-dimensional Gaussian parameter predictor. According to the method, the features of image and text modals are fused, under the guidance of the multi-modal features, high-quality three-dimensional scene reconstruction is realized by using the sparse view angle image, and the robustness is relatively high.
Need to check novelty before this filing date? Find Prior Art