A multi-modal feature guided sparse view three-dimensional scene reconstruction method

Abstract

The application discloses a kind of sparse view angle under three-dimensional scene reconstruction method based on multi-modal feature guide, the method includes: training initial sparse view image encoder, initial multi-modal guide module and initial three-dimensional Gaussian parameter predictor;Sparse view image is respectively input into the trained sparse view image encoder and the image segmentation model of frozen parameter, and shallow image feature and segmentation feature are obtained, sparse view text description is input into the text encoder of frozen parameter, and text feature is obtained;In the trained multi-modal guide module, shallow image feature is optimized and refined using segmentation feature and text feature, and multi-modal fusion feature is obtained;Using the trained three-dimensional Gaussian parameter predictor, according to multi-modal fusion feature, three-dimensional scene is reconstructed.The application fuses the features of image and text mode, under the guidance of multi-modal feature, using sparse view image realizes high-quality three-dimensional scene reconstruction, and has higher robustness.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of three-dimensional reconstruction, in particular to a three-dimensional scene reconstruction method and system under sparse view based on multi-modal feature guidance, a terminal and a computer readable storage medium. BACKGROUND

[0002] Three-dimensional scene reconstruction is an important research direction in the field of computer vision, and has wide application value in many fields such as automatic driving, virtual reality, and industrial manufacturing. The algorithms used for three-dimensional scene reconstruction can be divided into two categories according to technical methods, namely three-dimensional reconstruction algorithms based on traditional multi-view geometry and three-dimensional reconstruction algorithms based on deep learning. In recent years, with the rapid development of deep learning technology, more and more researchers are committed to developing more efficient neural network models to improve the effect and performance of three-dimensional scene reconstruction.

[0003] However, the current three-dimensional reconstruction algorithm based on deep learning has weak extrapolation ability for incomplete or large-area occluded objects in the image when performing three-dimensional scene reconstruction under more challenging sparse view conditions, making it difficult to achieve high-quality three-dimensional scene reconstruction.

[0004] Therefore, the prior art still needs to be improved and developed. SUMMARY

[0005] The main purpose of the present application is to provide a three-dimensional scene reconstruction method and system under sparse view based on multi-modal feature guidance, a terminal and a computer readable storage medium, which aims to solve the problem that the current three-dimensional reconstruction algorithm based on deep learning is difficult to achieve high-quality three-dimensional scene reconstruction under more challenging sparse view conditions.

[0006] To achieve the above-mentioned application purposes, the present application provides a three-dimensional scene reconstruction method under sparse view based on multi-modal feature guidance, which comprises:

[0007] Freezing the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters, training the initial sparse view image encoder, the initial multi-modal guidance module and the initial three-dimensional Gaussian parameter predictor to obtain the trained sparse view image encoder, the multi-modal guidance module and the three-dimensional Gaussian parameter predictor;

[0008] Obtaining a sparse view image, and pre-processing the sparse view image to obtain a sparse view text description corresponding to the sparse view image;

[0009] inputting the sparse view images into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, inputting the sparse view text description into the text encoder, and outputting text features;

[0010] inputting the shallow image features, the segmentation features, and the text features into the multi-modal guiding module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting multi-modal fusion features;

[0011] inputting the multi-modal fusion features into the three-dimensional Gaussian parameter predictor, outputting three-dimensional Gaussian parameters, and reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new view image.

[0012] Optionally, the initial image segmentation model, the initial text encoder, the initial sparse view image encoder, the initial multi-modal guiding module, and the initial three-dimensional Gaussian parameter predictor form an initial three-dimensional scene reconstruction model.

[0013] parameters of the initial image segmentation model and the initial text encoder are frozen to obtain an image segmentation model and a text encoder with frozen parameters, the initial sparse view image encoder, the initial multi-modal guiding module, and the initial three-dimensional Gaussian parameter predictor are trained to obtain trained sparse view image encoders, multi-modal guiding modules, and three-dimensional Gaussian parameter predictors, and the method specifically comprises the following steps:

[0014] obtaining a data set, dividing the data set into a training set and a test set, and pre-processing the training set and the test set respectively to obtain pre-processed training sets and test sets;

[0015] parameters of the initial image segmentation model and the initial text encoder are frozen to obtain an image segmentation model and a text encoder with frozen parameters.

[0016] the initial sparse view image encoder, the initial multi-modal guiding module, and the initial three-dimensional Gaussian parameter predictor are trained end to end by using the pre-processed training set, parameters of the initial three-dimensional scene reconstruction model are optimized by a gradient back propagation mechanism to minimize a loss function, and trained sparse view image encoders, multi-modal guiding modules, and three-dimensional Gaussian parameter predictors are obtained.

[0017] the image segmentation model and the text encoder with frozen parameters are combined with the trained sparse view image encoders, multi-modal guiding modules, and three-dimensional Gaussian parameter predictors to obtain a trained three-dimensional scene reconstruction model.

[0018] Test the trained three-dimensional scene reconstruction model by using the preprocessed test set, and if the test result meets the requirements, the trained three-dimensional scene reconstruction model is used as the final three-dimensional scene reconstruction model.

[0019] The loss function includes a learnable perceptual image block similarity and a mean square error.

[0020] Optionally, the data set is obtained, the data set is divided into a training set and a test set, and the training set and the test set are preprocessed respectively to obtain a preprocessed training set and a preprocessed test set, specifically including:

[0021] A data set is obtained, and the data set includes a plurality of videos and a plurality of camera parameters corresponding to each video frame in the plurality of videos;

[0022] The data set is randomly divided into a training set and a test set according to a preset division manner;

[0023] Each video in the training set is divided into a set number of sampling groups, and the first frame of each sampling group is processed by a multi-modal large model to generate a text description corresponding to the first frame of the sampling group. All video frames in the same sampling group use the text description corresponding to the first frame of the sampling group to obtain a preprocessed training set.

[0024] All video frames in each video in the test set are processed by a multi-modal large model to generate a text description corresponding to all video frames in the test set, and a preprocessed test set is obtained.

[0025] Optionally, the sparse view image is obtained, and the sparse view image is preprocessed to obtain a sparse view text description corresponding to the sparse view image, specifically including:

[0026] A sparse view image and a camera parameter corresponding to the sparse view image are obtained.

[0027] The sparse view image is processed by a multi-modal large model to generate a sparse view text description corresponding to the sparse view image.

[0028] Optionally, the sparse view image is input into the sparse view image encoder and the image segmentation model respectively, and the sparse view image is output as a shallow image feature and a segmentation feature. The sparse view text description is input into the text encoder, and the text feature is output, specifically including:

[0029] The sparse view image is denoted as I, and the sparse view text description corresponding to the sparse view image is denoted as T:

[0030]

[0031] wherein i denotes a sparse view, K denotes a number of sparse views, I i denotes a sparse view image corresponding to the sparse view i, h and w denote a length and a width of the sparse view image respectively, R denotes a real matrix, T i denotes a sparse view text description corresponding to the sparse view i, l denotes a length of the sparse view text description.

[0032] inputting the sparse view image I into a sparse view image encoder E i , and outputting a shallow image feature F W :

[0033] F W = E i (I);

[0034]

[0035] wherein c denotes a number of channels of the shallow image feature, F W i denotes a shallow image feature corresponding to the sparse view i.

[0036] inputting the sparse view image I into an image segmentation model S i , and outputting a segmentation feature F S :

[0037] F S = S i (I);

[0038]

[0039] wherein F S i denotes a segmentation feature corresponding to the sparse view i.

[0040] inputting the sparse view text description T into a text encoder E t , and outputting a text feature F E :

[0041] F E = E t (T);

[0042]

[0043] wherein d denotes a dimension of a hidden state vector of each wordpiece, F E i denotes a text feature corresponding to the sparse view i.

[0044] Optionally, the multi-modal guiding module comprises a cross-modal cross-attention module, a 1x1 convolutional layer and a visual cross-attention module.

[0045] The shallow image features, the segmentation features and the text features are input into the multi-modal guidance module, the shallow image features are optimized and refined by using the segmentation features and the text features, and multi-modal fusion features are output, specifically including:

[0046] Firstly, the cross-modal cross-attention module f ca is used to fuse the segmentation features F S and the text features F E , and then the 1x1 convolution layer f ch is used to compress the channel dimension to obtain the cross-modal guidance features F C :

[0047] F C =f ch (f ca (F S ,F E ));

[0048]

[0049] Wherein, F C i represents the cross-modal guidance features corresponding to the sparse view angle i;

[0050] The visual cross-attention module f v is used to fuse the cross-modal guidance features F C and the shallow image features F W to obtain the multi-modal fusion features M:

[0051] M=f v (F C ,F W );

[0052]

[0053] Wherein, M i represents the multi-modal fusion features corresponding to the sparse view angle i.

[0054] Optionally, the three-dimensional Gaussian parameter predictor includes a plane scanning module, a lightweight UNet network, a convolutional neural network-based up-sampler, a depth prediction module and a three-dimensional Gaussian prediction head;

[0055] The multi-modal fusion features are input into the three-dimensional Gaussian parameter predictor, and three-dimensional Gaussian parameters are output, and a new view image is obtained by reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters, specifically including:

[0056] Through the plane scanning module, a cost volume V corresponding to the multi-modal fusion features M is constructed in a three-dimensional space:

[0057]

[0058] where P(·) denotes the feature projection operation, m j denotes the multimodal fusion feature of another sparse view j with respect to sparse view i, D i and D j denote the projection matrix under sparse view i and sparse view j, respectively, d m denotes the m-th depth candidate region uniformly sampled on the inverse depth domain, denotes the projection feature obtained by projecting sparse view j to sparse view i, denotes the depth candidate region d m and the cost volume corresponding to sparse view i, denotes the depth candidate region d b and the cost volume corresponding to sparse view i, b denotes the number of depth candidate regions, V i denotes the cost volume corresponding to sparse view i;

[0059] by a lightweight UNet network U and a convolutional neural network-based up-sampler f up , the cost volume V is refined in feature and aligned in channel to obtain a refined cost volume V C :

[0060] V C = f up (V+U(M,V));

[0061]

[0062] where V C i denotes the refined cost volume corresponding to sparse view i, and U(·) denotes a lightweight UNet network;

[0063] The refined cost volume V C and the depth candidate region G are input into a depth prediction module to obtain a depth estimation map N of the sparse view image:

[0064] N = U(f s (V C )·G);

[0065] G = [d1, d2, …, d b ] ∈ R h×w ;

[0066]

[0067] where N i denotes the depth estimation map of the sparse view image corresponding to sparse view i, and fs (·) represents a Softmax function;

[0068] the camera parameters p corresponding to the sparse view image pair are input into a three-dimensional Gaussian prediction head P' to obtain three-dimensional Gaussian parameters p required for three-dimensional scene reconstruction c g :

[0069] p g = P'(N, p c );

[0070] reconstruct a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new view image.

[0071] To achieve the above-mentioned purposes, the application further provides a sparse view three-dimensional scene reconstruction system based on multi-modal feature guidance, which comprises:

[0072] a model training module: used for freezing the parameters of an initial image segmentation model and an initial text encoder to obtain a frozen-parameter image segmentation model and a text encoder, training an initial sparse view image encoder, an initial multi-modal guidance module and an initial three-dimensional Gaussian parameter predictor to obtain a trained sparse view image encoder, multi-modal guidance module and three-dimensional Gaussian parameter predictor;

[0073] a data acquisition module: used for acquiring sparse view images and pre-processing the sparse view images to obtain sparse view text descriptions corresponding to the sparse view images;

[0074] a feature extraction module: used for inputting the sparse view images into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, inputting the sparse view text descriptions into the text encoder, and outputting text features;

[0075] a feature fusion module: used for inputting the shallow image features, the segmentation features and the text features into the multi-modal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting multi-modal fusion features;

[0076] a three-dimensional scene reconstruction module: used for inputting the multi-modal fusion features into the three-dimensional Gaussian parameter predictor, outputting three-dimensional Gaussian parameters, reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters, and obtaining a new view image.

[0077] ​To achieve the above-mentioned purposes of the application, the application further provides a terminal, which comprises a memory, a processor and a multi-modal feature guided sparse-view three-dimensional scene reconstruction program stored in the memory and executable on the processor, and the multi-modal feature guided sparse-view three-dimensional scene reconstruction program realizes the steps of the multi-modal feature guided sparse-view three-dimensional scene reconstruction method as described above when executed by the processor.

[0078] To achieve the above-mentioned purposes of the application, the application further provides a computer readable storage medium, which stores a multi-modal feature guided sparse-view three-dimensional scene reconstruction program, and the multi-modal feature guided sparse-view three-dimensional scene reconstruction program realizes the steps of the multi-modal feature guided sparse-view three-dimensional scene reconstruction method as described above when executed by a processor.

[0079] In the application, the parameters of the initial image segmentation model and the initial text encoder are frozen to obtain an image segmentation model and a text encoder with frozen parameters, the initial sparse-view image encoder, the initial multi-modal guiding module and the initial three-dimensional Gaussian parameter predictor are trained to obtain a trained sparse-view image encoder, a multi-modal guiding module and a three-dimensional Gaussian parameter predictor, a sparse-view image is acquired and preprocessed to obtain a sparse-view text description corresponding to the sparse-view image, the sparse-view image is input into the sparse-view image encoder and the image segmentation model respectively to output shallow image features and segmentation features, the sparse-view text description is input into the text encoder to output text features, the shallow image features, the segmentation features and the text features are input into the multi-modal guiding module, the shallow image features are optimized and refined by using the segmentation features and the text features, and multi-modal fusion features are output, the multi-modal fusion features are input into the three-dimensional Gaussian parameter predictor to output three-dimensional Gaussian parameters, and a three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new-view image. The application optimizes the three-dimensional scene reconstruction model, introduces segmentation features and text features as supplementary information for the shallow image features extracted from the sparse-view image, fuses the features of the image modal and the text modal to obtain multi-modal features with strong extrapolation capability, uses the sparse-view image to realize high-quality three-dimensional scene reconstruction under the guidance of the multi-modal features, the generated new-view image has excellent visual quality and is generated with high efficiency, and the optimized three-dimensional scene reconstruction model has high robustness and can ensure high-quality three-dimensional scene reconstruction in different environments. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 is a flowchart of a preferred embodiment of the multi-modal feature guided sparse-view three-dimensional scene reconstruction method of the application.

[0081] Figure 2 is a structural diagram of a three-dimensional scene reconstruction model of the present application;

[0082] Figure 3 is a structural diagram of a multi-modal guidance module of the present application;

[0083] Figure 4 is a structural diagram of a preferred embodiment of a three-dimensional scene reconstruction system based on multi-modal feature guidance in a sparse view of the present application;

[0084] Figure 5 is a structural diagram of a preferred embodiment of a terminal of the present application. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solutions and advantages of the present application clearer and more explicit, the present application will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0086] Three-dimensional scene reconstruction is an important research direction in the field of computer vision, and has wide application value in many fields such as automatic driving, virtual reality, industrial manufacturing, etc. The algorithms used for three-dimensional scene reconstruction can be divided into two categories according to technical methods, which are three-dimensional reconstruction algorithms based on traditional multi-view geometry and three-dimensional reconstruction algorithms based on deep learning. In recent years, with the rapid development of deep learning technology, more and more researchers have devoted to developing more efficient neural network models to improve the effect and performance of three-dimensional scene reconstruction.

[0087] At present, the mainstream three-dimensional reconstruction algorithm based on deep learning mainly uses neural radiance field and other neural implicit representation methods to describe the three-dimensional scene by fusing the position and viewpoint conditions of the neural network. However, this process is usually computationally intensive, requiring a large amount of training time and rendering resources. The 3D Gaussian Splatting technology models the scene through a set of Gaussian ellipsoids and rasterizes the Gaussian ellipsoid into an image to achieve efficient rendering. Due to its explicit representation and highly parallel workflow, this method significantly reduces the training cost and improves the real-time performance. In addition, the application of 3D Gaussian Splatting technology promotes the development of a series of downstream tasks such as three-dimensional reconstruction, three-dimensional editing and three-dimensional generation.

[0088] However, the current 3D Gaussian Splatting-based method still has some deficiencies when reconstructing a three-dimensional scene under more challenging sparse view conditions. For example, although convolutional neural networks and visual Transformers can effectively extract visual information from images with sparse views and complete three-dimensional reconstruction in regular scenes, the extrapolation ability of these methods is weak for objects in the image that are incomplete or have large-area occlusions, making it difficult to achieve high-quality three-dimensional scene reconstruction.

[0089] To solve the above technical problems, the present application provides a three-dimensional scene reconstruction method under sparse view based on multi-modal feature guidance. The parameters of the initial image segmentation model and the initial text encoder are frozen to obtain the image segmentation model and the text encoder with frozen parameters. The initial sparse view image encoder, the initial multi-modal guidance module, and the initial three-dimensional Gaussian parameter predictor are trained to obtain the trained sparse view image encoder, the multi-modal guidance module, and the three-dimensional Gaussian parameter predictor. The sparse view image is obtained and preprocessed to obtain the sparse view text description corresponding to the sparse view image. The sparse view image is input into the sparse view image encoder and the image segmentation model respectively, and the shallow image features and the segmentation features are output. The text feature is output by inputting the sparse view text description into the text encoder. The shallow image features, the segmentation features, and the text features are input into the multi-modal guidance module, and the shallow image features are optimized and refined using the segmentation features and the text features, and the multi-modal fusion features are output. The multi-modal fusion features are input into the three-dimensional Gaussian parameter predictor to output the three-dimensional Gaussian parameters, and the three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain the new view image. The present application optimizes the three-dimensional scene reconstruction model, introduces the supplementary information of segmentation features and text features for the shallow image features extracted from the sparse view image, fuses the features of image modalities and text modalities to obtain multi-modal features with strong extrapolation ability, and uses the sparse view image to realize high-quality three-dimensional scene reconstruction under the guidance of multi-modal features. The generated new view image has excellent visual quality and high efficiency. The optimized three-dimensional scene reconstruction model has high robustness, ensuring high-quality three-dimensional scene reconstruction in different environments.

[0090] The application content will be further described by describing the embodiments in conjunction with the drawings.

[0091] The preferred embodiment of the three-dimensional scene reconstruction method under sparse view based on multi-modal feature guidance of the present application, as shown in Figure 1 and Figure 2 , specifically includes:

[0092] S1: freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters, train the initial sparse view image encoder, the initial multi-modal guiding module and the initial three-dimensional Gaussian parameter predictor to obtain the trained sparse view image encoder, the multi-modal guiding module and the three-dimensional Gaussian parameter predictor.

[0093] In one implementation manner of the embodiment, the initial image segmentation model, the initial text encoder, the initial sparse view image encoder, the initial multi-modal guiding module and the initial three-dimensional Gaussian parameter predictor constitute an initial three-dimensional scene reconstruction model.

[0094] The method comprises the following steps:

[0095] S11: obtain a data set, divide the data set into a training set and a test set, and respectively pre-process the training set and the test set to obtain pre-processed training set and test set;

[0096] S12: freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters;

[0097] S13: train the initial sparse view image encoder, the initial multi-modal guiding module and the initial three-dimensional Gaussian parameter predictor in an end-to-end manner by using the pre-processed training set, optimize the parameters of the initial three-dimensional scene reconstruction model through a gradient back propagation mechanism to minimize a loss function, and obtain the trained sparse view image encoder, the multi-modal guiding module and the three-dimensional Gaussian parameter predictor;

[0098] S14: combine the image segmentation model and the text encoder with frozen parameters and the trained sparse view image encoder, the multi-modal guiding module and the three-dimensional Gaussian parameter predictor to obtain the trained three-dimensional scene reconstruction model;

[0099] S15: test the trained three-dimensional scene reconstruction model by using the pre-processed test set, and if the test result meets the requirements, the trained three-dimensional scene reconstruction model is used as the final three-dimensional scene reconstruction model.

[0100] The loss function comprises a learnable perceptual image block similarity and a mean square error.

[0101] Specifically, during the training and testing process of the network, the present application first selects a large high-quality three-dimensional reconstruction dataset RealEstate10K disclosed, randomly divides the dataset into a training set and a test set according to a preset division method, and then generates a text description for the RealEstate10K dataset through a multi-modal large model Qwen2-VL. The present application selects a sparse view number of 2, and reconstructs a three-dimensional scene according to two sparse view input images (i.e., sparse view images). Two video frames with an interval frame number greater than 25 are randomly selected from the training scene (i.e., the training set) video frames as input images, and the images are all adjusted to a size of 256x256 pixels to enhance the diversity of the data. In the training stage, the model is trained end-to-end. Through the gradient back propagation mechanism, the model parameters are optimized to minimize the loss function, so that the model can realize high-quality three-dimensional scene reconstruction under sparse view. In the testing stage, the network weight parameters saved in the preset maximum training number in the training process are loaded into the model, the corresponding sparse view images and text descriptions of the images are input, the three-dimensional scene is reconstructed, and the rendering image of the new view is generated. In the model training, the present application model uses the rendering loss for end-to-end training, and the learned perceptual image patch similarity (LPIPS) and the mean square error (MSE) are used as the loss function to supervise the difference between the rendering image (i.e., the new view image) of the reconstructed scene under the new view and the real image, so as to ensure the quality of the reconstructed three-dimensional scene, and the overall form of the loss function is:

[0102] L=L MSE +λ·L LPIPS

[0103] wherein, L MSE is the MSE loss, L LPIPS is the LPIPS loss, λ is a weighted hyperparameter, representing the weight of the LPIPS loss in the overall loss function, which can be set to 0.05.

[0104] In order to evaluate the performance of the model, the present application calculates the peak signal-to-noise ratio (PSNR), structural similarity (SSIM) and learned perceptual image patch similarity (LPIPS) between the rendering image of the new view and the real image. The higher the values of PSNR and SSIM and the lower the value of LPIPS, the better the quality of the rendering image of the new view, the higher the quality of the three-dimensional scene reconstruction, and the better the performance of the model.

[0105] In addition, in order to verify the effectiveness of the present application, the proposed model is trained and tested on a server equipped with an NVIDIA RTXA6000 GPU and an Intel Xeon Gold 6330 CPU, implemented using the open-source Pytorch deep learning framework. The model is trained end-to-end on four NVIDIA RTXA6000 GPUs, with a total of 180,000 iterations until the loss converges, an initial learning rate of 2x10-4, and a preheating and cosine annealing strategy to adjust the learning rate. Two video frames with an interval of more than 25 frames are randomly selected from the training scene video frames as input images, and the images are adjusted to a size of 256x256 pixels. The text description is generated by the multi-modal large model Qwen2-VL. The image segmentation model and the text encoder with frozen parameters use the OMG-SEG model and the LongCLIP model trained on the image segmentation dataset and the cross-modal alignment dataset, respectively.

[0106] In the three-dimensional scene reconstruction model of the present application, the parameters of the image segmentation model and the text encoder are frozen, and the sparse view image encoder, the multi-modal guidance module, and the three-dimensional Gaussian parameter predictor are trained end-to-end. Through the gradient backpropagation mechanism, the model parameters are optimized to minimize the loss function, enabling the model to achieve high-quality three-dimensional scene reconstruction under sparse view. The structure of the three-dimensional scene reconstruction model is as shown in Figure 2 The present application fully considers the characteristics of multi-modal feature fusion and three-dimensional scene reconstruction, and provides a method for three-dimensional scene reconstruction under sparse view based on multi-modal feature guidance. Experimental results show that the present application performs superiorly in three-dimensional scene reconstruction using sparse view images under multi-modal feature guidance, and the reconstructed three-dimensional scene has strong extrapolation capability, and the images generated at new angles have excellent visual quality. In addition, the model can still effectively reconstruct the three-dimensional scene for different scenes, and has high robustness. The specific beneficial effects can be summarized as follows:

[0107] (1) High-quality three-dimensional scene reconstruction: the reconstructed three-dimensional scene has low distortion and maintains visual coherence, and can render clear images at new angles. The PSNR, SSIM, and LPIPS indicators are used for evaluation, with PSNR higher than 25 dB, SSIM higher than 0.85, and LPIPS lower than 0.15.

[0108] (2) Efficient generation of new view images: the time for rendering new view images based on the reconstructed three-dimensional scene is not more than 2 seconds.

[0109] In one implementation of the present embodiment, the data set is obtained, the data set is divided into a training set and a test set, and the training set and the test set are preprocessed respectively to obtain preprocessed training set and test set, specifically including:

[0110] S111: Obtain a data set, the data set comprising a plurality of videos and a plurality of camera parameters corresponding to each frame of the videos;

[0111] S112: Randomly divide the data set into a training set and a test set according to a preset division manner;

[0112] S113: Divide each video in the training set into a set number of sampling groups, process a first frame of each sampling group by a multi-modal large model to generate a text description corresponding to the first frame of the sampling group, and use the text description corresponding to the first frame of the sampling group for all video frames in the same sampling group to obtain a preprocessed training set;

[0113] S114: Process all video frames in each video in the test set by the multi-modal large model to generate text descriptions corresponding to all video frames in the test set to obtain a preprocessed test set.

[0114] Specifically, in the training and testing process of the network, the present application selects a large high-quality three-dimensional reconstruction data set RealEstate10K disclosed to train and test the proposed three-dimensional scene reconstruction model. The RealEstate10K data set contains about 80,000 video clips and about 10 million frames collected from a video website. After removing low-quality data, the RealEstate10K data set is divided into 67,477 training scenes (i.e., a training set) and 7,289 test scenes (i.e., a test set). In addition, the RealEstate10K data set provides estimated camera intrinsic and extrinsic parameters (i.e., camera parameters) for each video frame. The present application generates text descriptions for the RealEstate10K data set by a multi-modal large model Qwen2-VL. For the training scenes, each video is evenly divided into 5 sampling groups, and a text description is generated for the first frame of each sampling group, and the text description of the first frame of the sampling group is used for all video frames in the same sampling group; for the test scenes, a text description is generated for all video frames. It should be noted that the preprocessed training set and test set each include video frames, camera parameters corresponding to the video frames, and text descriptions, wherein the video frames are images under sparse view angle conditions.

[0115] S2: Obtain a sparse view angle image, and pre-process the sparse view angle image to obtain a sparse view angle text description corresponding to the sparse view angle image.

[0116] In one implementation manner of the present embodiment, the obtaining of the sparse view angle image and the pre-processing of the sparse view angle image to obtain the sparse view angle text description corresponding to the sparse view angle image specifically comprises:

[0117] S21: acquire a sparse view image and a camera parameter corresponding to the sparse view image;

[0118] S22: process the sparse view image by a multi-modal large model to generate a sparse view text description corresponding to the sparse view image.

[0119] Specifically, the present application processes the input image under the condition of sparse view by a multi-modal large model to obtain the text description of the input image, supplements the semantic information of the extracted features of the input image to obtain the refined feature representation with strong extrapolation ability, and realizes high-quality three-dimensional scene reconstruction. It should be noted that the sparse view image here is the input image to be reconstructed into a three-dimensional scene, and the video frames in the previous dataset are input images for training and testing the model.

[0120] S3: input the sparse view image into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, input the sparse view text description into the text encoder, and output text features.

[0121] In one implementation manner of the present embodiment, the inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, inputting the sparse view text description into the text encoder, and outputting text features specifically comprises:

[0122] S31: the sparse view image is denoted as I, and the sparse view text description corresponding to the sparse view image is denoted as T:

[0123]

[0124] wherein i represents a sparse view (i.e. the i-th sparse view), K represents the number of sparse views, I i represents the sparse view image corresponding to the sparse view i (i.e. under the sparse view i), h and w respectively represent the length and width of the sparse view image, R represents a real matrix, T i represents the sparse view text description corresponding to the sparse view i, and l represents the length of the sparse view text description.

[0125] S32: input the sparse view image I into the sparse view image encoder E i , and output the shallow image features F W of the sparse view image:

[0126] F W =E i (I);

[0127]

[0128] where c denotes the number of channels of the shallow image features of the sparse view i, F W i denotes the shallow image features of the sparse view image corresponding to the sparse view i;

[0129] S33: input the sparse view image I into the image segmentation model S i , and output the segmentation features F S of the sparse view image:

[0130] F S = S i (I);

[0131]

[0132] where F S i denotes the segmentation features of the sparse view image corresponding to the sparse view i.

[0133] S34: input the sparse view text description T into the text encoder E t , and output the text features F E of the sparse view text description:

[0134] F E = E t (T);

[0135]

[0136] where d denotes the dimension of the hidden state vector of each wordpiece, F E i denotes the text features of the sparse view text description corresponding to the sparse view i.

[0137] Specifically, the sparse view image I and the sparse view text description T are inputs of the three-dimensional scene reconstruction model, the sparse view image I is input into the sparse view image encoder E i , and the shallow image features F W of the sparse view image are extracted, the sparse view image I is input into the image segmentation model S i , and the segmentation features F S of the sparse view image are extracted, the sparse view text description T is input into the text encoder E t , and the text features F E of the sparse view text description are extracted, and the features of the image and text model are obtained.

[0138] S4: input the shallow image features, the segmentation features and the text features into the multi-modal guiding module, optimize and refine the shallow image features by using the segmentation features and the text features, and output multi-modal fusion features.

[0139] In one implementation manner of the embodiment, as shown in Figure 3 The multi-modal guiding module includes a cross-modal cross-attention module, a 1x1 convolution layer and a visual cross-attention module, the cross-modal cross-attention module includes a multi-head self-attention layer and a multi-head cross-attention layer, and the visual cross-attention module includes a window self-attention layer and a window cross-attention layer.

[0140] The inputting of the shallow image features, the segmentation features and the text features into the multi-modal guiding module, the optimization and refinement of the shallow image features by using the segmentation features and the text features, and the outputting of the multi-modal fusion features specifically include:

[0141] S41: first use the cross-modal cross-attention module f ca to fuse the segmentation features F S and the text features F E , the cross-modal cross-attention module includes a multi-head self-attention layer and a multi-head cross-attention layer, then use the 1x1 convolution layer f ch to compress the channel dimension, and obtain the cross-modal guiding features F C of the sparse perspective image:

[0142] F C =f ch (f ca (F S ,F E ));

[0143]

[0144] Wherein, F C i represents the cross-modal guiding features of the sparse perspective image corresponding to the sparse perspective i;

[0145] S42: use the visual cross-attention module f v to fuse the cross-modal guiding features F C and the shallow image features F W , the visual cross-attention module includes a window self-attention layer and a window cross-attention layer, that is, each feature map is divided into four sub-feature maps, and the multi-modal fusion features M of the sparse perspective image are obtained:

[0146] M=f v (F C ,FW

[0147]

[0148] wherein M i represents the multi-modal fusion feature of the sparse view image corresponding to the sparse view i.

[0149] Specifically, the present application introduces the supplementary information of the segmentation model and the text encoder (segmentation features and text features) from the features extracted from the sparse view image (shallow image features), to obtain refined feature representation with strong extrapolation ability (multi-modal fusion features), and through the analysis of the step-by-step fusion mechanism of the multi-modal features, it is ensured that the multi-modal features can supplement the semantic information in the three-dimensional scene reconstruction.

[0150] S5: inputting the multi-modal fusion feature into the three-dimensional Gaussian parameter predictor, outputting a three-dimensional Gaussian parameter, reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameter, and obtaining a new view image.

[0151] In one implementation manner of the embodiment, the three-dimensional Gaussian parameter predictor comprises a plane scanning module, a lightweight UNet network, a convolutional neural network-based up-sampler, a depth prediction module and a three-dimensional Gaussian prediction head.

[0152] The inputting of the multi-modal fusion feature into the three-dimensional Gaussian parameter predictor, the outputting of a three-dimensional Gaussian parameter, the reconstructing of a three-dimensional scene according to the three-dimensional Gaussian parameter, and the obtaining of a new view image specifically comprise:

[0153] S51: constructing a cost volume V corresponding to the multi-modal fusion feature M in the three-dimensional space through the plane scanning module, which stores the cross-view feature similarity of all potential depth candidate regions:

[0154]

[0155] wherein P(·) represents a feature projection operation, M j represents the multi-modal fusion feature of another sparse view j relative to the sparse view i, D i and D j respectively represent the projection matrix under the sparse view i and the sparse view j, d m represents the mth depth candidate region uniformly sampled on the inverse depth domain, represents the projection feature obtained by projecting the sparse view j to the sparse view i, represents the depth candidate region d m and the cost volume corresponding to the sparse view i, represents the depth candidate region d b and the cost volume corresponding to the sparse view i, and b represents the number of depth candidate regions, V​i denotes the cost volume corresponding to the sparse view i;

[0156] S52: the refined cost volume V is obtained by inputting the cost volume V and the depth candidate region G into the lightweight UNet network U and the convolutional neural network-based up-sampler f up C :

[0157] V C = f up (V + U(M, V));

[0158]

[0159] wherein V C i denotes the refined cost volume corresponding to the sparse view i, and U(·) denotes the lightweight UNet network;

[0160] S53: the refined cost volume V C and the depth candidate region G are input into the depth prediction module (including the lightweight UNet network U and the Softmax function f s ) to obtain the depth estimation map N of the sparse view image:

[0161] N = U(f s (V C )·G);

[0162] G = [d1, d2, …, d b ] ∈ R h×w ;

[0163]

[0164] wherein N i denotes the depth estimation map of the sparse view image corresponding to the sparse view i, and f s (·) denotes the Softmax function;

[0165] S54: the camera parameters p c corresponding to the sparse view image and the depth estimation map N of the sparse view image are input into the three-dimensional Gaussian prediction head P' to obtain the three-dimensional Gaussian parameters p g required for three-dimensional scene reconstruction:

[0166] p g = P'(N, p c );

[0167] S55: a new view image is obtained by reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters.

[0168] ​Specifically, by predicting the three-dimensional Gaussian parameters required for multi-modal fusion feature reconstruction, the three-dimensional scene reconstruction using sparse view images under the guidance of multi-modal features exhibits superior performance, the reconstructed three-dimensional scene has strong extrapolation capability, the generated images at new perspectives have excellent visual quality, and the model has high efficiency in generating new perspective images. In addition, for different scenes, the model can still effectively reconstruct the three-dimensional scene, and has high robustness.

[0169] In addition, based on the above-mentioned three-dimensional scene reconstruction method under sparse view based on multi-modal feature guidance, the application also correspondingly provides a three-dimensional scene reconstruction system under sparse view based on multi-modal feature guidance, wherein the preferred embodiment of the three-dimensional scene reconstruction system under sparse view based on multi-modal feature guidance comprises the following steps: Figure 4

[0170] The model training module 01 is used to freeze the parameters of the initial image segmentation model and the initial text encoder to obtain an image segmentation model and a text encoder with frozen parameters, and train an initial sparse view image encoder, an initial multi-modal guidance module and an initial three-dimensional Gaussian parameter predictor to obtain a trained sparse view image encoder, a multi-modal guidance module and a three-dimensional Gaussian parameter predictor.

[0171] The data acquisition module 02 is used to acquire sparse view images and pre-process the sparse view images to obtain sparse view text descriptions corresponding to the sparse view images.

[0172] The feature extraction module 03 is used to input the sparse view images into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, and input the sparse view text descriptions into the text encoder to output text features.

[0173] The feature fusion module 04 is used to input the shallow image features, the segmentation features and the text features into the multi-modal guidance module, optimize and refine the shallow image features by using the segmentation features and the text features, and output multi-modal fusion features.

[0174] The three-dimensional scene reconstruction module 05 is used to input the multi-modal fusion features into the three-dimensional Gaussian parameter predictor, output three-dimensional Gaussian parameters, reconstruct a three-dimensional scene according to the three-dimensional Gaussian parameters, and obtain a new perspective image.

[0175] In addition, based on the above-mentioned three-dimensional scene reconstruction method and system under sparse view based on multi-modal feature guidance, the application also correspondingly provides a terminal, wherein the preferred embodiment of the terminal comprises a processor 10, a memory 20 and a display 30. Figure 5 Figure 5 ​​Only some components of the terminal are shown, but it should be understood that all the shown components are not required, and more or less components can be provided.

[0176] The memory 20 can be an internal storage unit of the terminal in some embodiments, such as a hard disk or a memory of the terminal. The memory 20 can also be an external storage device of the terminal in other embodiments, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory 20 can include both an internal storage unit and an external storage device of the terminal. The memory 20 is used to store application software and various data installed on the terminal, such as program codes of the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In an embodiment, the memory 20 stores a sparse view three-dimensional scene reconstruction program based on multi-modal feature guidance 40, which can be executed by the processor 10 to implement the steps of the sparse view three-dimensional scene reconstruction method based on multi-modal feature guidance in the present application.

[0177] The processor 10 can be a central processing unit (CPU), a microprocessor, or other data processing chip in some embodiments, which is used to run program codes stored in the memory 20 or process data, such as executing the sparse view three-dimensional scene reconstruction program based on multi-modal feature guidance 40.

[0178] The display 30 can be an LED display, a liquid crystal display, a touch liquid crystal display, an OLED (Organic Light-Emitting Diode) touch, and the like in some embodiments. The display 30 is used to display information of the terminal and to display a visualized user interface. The components of the terminal communicate with each other through a system bus.

[0179] In an embodiment, the steps of the sparse view three-dimensional scene reconstruction method based on multi-modal feature guidance as described above are implemented when the processor 10 executes the sparse view three-dimensional scene reconstruction program based on multi-modal feature guidance 40 in the memory 20.

[0180] The present application also provides a computer-readable storage medium, wherein the computer-readable storage medium stores a sparse view three-dimensional scene reconstruction program based on multi-modal feature guidance, and the steps of the sparse view three-dimensional scene reconstruction method based on multi-modal feature guidance as described above are implemented when the processor executes the sparse view three-dimensional scene reconstruction program based on multi-modal feature guidance.

[0181] It should be noted that, in the present document, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0182] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer readable storage medium readable by a computer. When the program is executed, it can include the processes of the above-mentioned method embodiments. The computer readable storage medium can be a memory, a disk, an optical disk, etc.

[0183] It should be understood that the application is not limited to the above examples, and those skilled in the art can make improvements or changes according to the above description, and all these improvements and changes shall fall within the protection scope of the appended claims of the present application.

Claims

1. A sparse-view 3D scene reconstruction method based on multi-modal feature guided, characterized in that, The method comprises the following steps: Freezing parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters, training the initial sparse-view image encoder, the initial multi-modal guidance module and the initial three-dimensional Gaussian parameter predictor to obtain the trained sparse-view image encoder, the multi-modal guidance module and the three-dimensional Gaussian parameter predictor; Obtaining sparse-view images and preprocessing the sparse-view images to obtain sparse-view text descriptions corresponding to the sparse-view images; Inputting the sparse-view images into the sparse-view image encoder and the image segmentation model respectively to output shallow image features and segmentation features, and inputting the sparse-view text descriptions into the text encoder to output text features; Inputting the shallow image features, the segmentation features and the text features into the multi-modal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features to output multi-modal fusion features; Inputting the multi-modal fusion features into the three-dimensional Gaussian parameter predictor to output three-dimensional Gaussian parameters, and reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain new-view images.

2. The multi-modal feature guided sparse view 3D scene reconstruction method according to claim 1, wherein, The initial image segmentation model, the initial text encoder, the initial sparse-view image encoder, the initial multi-modal guidance module and the initial three-dimensional Gaussian parameter predictor constitute an initial three-dimensional scene reconstruction model; The method comprises the following steps: Obtaining a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain preprocessed training set and test set; Freezing parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters; Training the initial sparse-view image encoder, the initial multi-modal guidance module and the initial three-dimensional Gaussian parameter predictor end to end by using the preprocessed training set, optimizing parameters of the initial three-dimensional scene reconstruction model by a gradient back propagation mechanism to minimize a loss function, and obtaining the trained sparse-view image encoder, the multi-modal guidance module and the three-dimensional Gaussian parameter predictor; Combining the image segmentation model and the text encoder with frozen parameters and the trained sparse-view image encoder, the multi-modal guidance module and the three-dimensional Gaussian parameter predictor to obtain the trained three-dimensional scene reconstruction model; Testing the trained three-dimensional scene reconstruction model by using the preprocessed test set, and regarding the trained three-dimensional scene reconstruction model as the final three-dimensional scene reconstruction model if the test result meets the requirements. The loss function comprises a learnable perceptual image block similarity and a mean square error.

3. The method of claim 2, wherein, The data set is obtained, the data set is divided into a training set and a test set, and the training set and the test set are respectively preprocessed to obtain a preprocessed training set and a preprocessed test set, specifically including: Obtain a data set, the data set includes a plurality of videos and a plurality of camera parameters corresponding to each video frame in the plurality of videos; According to a preset division manner, the data set is randomly divided into a training set and a test set; Divide each video in the training set into a set number of sampling groups, process the first frame of each sampling group by a multi-modal large model to generate a text description corresponding to the first frame of the sampling group, and use the text description corresponding to the first frame of the sampling group for all video frames in the same sampling group to obtain a preprocessed training set; Process all video frames in each video in the test set by a multi-modal large model to generate a text description corresponding to all video frames in the test set, and obtain a preprocessed test set.

4. The multi-modal feature guided sparse view 3D scene reconstruction method of claim 1, wherein, The sparse view image is obtained, and the sparse view image is preprocessed to obtain a sparse view text description corresponding to the sparse view image, specifically including: Obtain a sparse view image and a camera parameter corresponding to the sparse view image; Process the sparse view image by a multi-modal large model to generate a sparse view text description corresponding to the sparse view image.

5. The method of claim 4, wherein, The sparse view image is input into the sparse view image encoder and the image segmentation model respectively, and shallow image features and segmentation features are output, and the sparse view text description is input into the text encoder, and text features are output, specifically including: The sparse view image is denoted as I, and the sparse view text description corresponding to the sparse view image is denoted as T: ； ； wherein, i denotes a sparse view, K denotes a number of sparse views, denotes a sparse view i a corresponding sparse view image, h and w denote a length and a width of the sparse view image, respectively, R denotes a real matrix, denotes a sparse view i a corresponding sparse view textual description, denotes a length of the sparse view textual description; inputting a sparse view image I to a sparse view image encoder in which the shallow image features are output : ； ； wherein c represents the number of channels of the shallow image features, denotes a sparse view angle i corresponding shallow image features; inputting sparse view images I to an image segmentation model outputting segmentation features : ； ； wherein, indicates a sparse view angle i corresponding segmentation features; inputting sparse view text description T into a text encoder outputting text features : ； ； where d denotes the dimension of the hidden state vector of each token, denotes a sparse view angle i corresponding text features.

6. The multi-modal feature guided sparse view 3D scene reconstruction method according to claim 5, wherein, The multi-modal guiding module includes a cross-modal cross attention module, a 1×1 convolution layer and a visual cross attention module; The shallow image features, the segmentation features and the text features are input into the multi-modal guiding module, the segmentation features and the text features are used to optimize and refine the shallow image features, and multi-modal fusion features are output, specifically including: First, a cross-modal cross-attention module is used to fuse the segmentation features and the text features to obtain cross-modal guidance features using a 1x1 convolution layer : ； ； wherein, representing sparse view angles i corresponding cross-modal guidance features; Using visual cross attention modules Fusing cross-modal guidance features and shallow image features to obtain a multi-modal fusion feature M: ； ； wherein, indicates a sparse view angle i corresponding multi-modal fusion features.

7. The multi-modal feature guided sparse view 3D scene reconstruction method according to claim 6, wherein, The three-dimensional Gaussian parameter predictor includes a plane scanning module, a lightweight UNet network, a convolutional neural network-based up-sampler, a depth prediction module and a three-dimensional Gaussian prediction head; The multi-modal fusion features are input into the three-dimensional Gaussian parameter predictor, and three-dimensional Gaussian parameters are output, and a three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new view image, specifically including: Through the plane scanning module, a cost volume V corresponding to the multi-modal fusion features M is constructed in a three-dimensional space: ； ； ； ； in, This represents the feature projection operation. Represents a sparse perspective i Another sparse perspective j Multimodal fusion features, and Representing sparse perspectives i and sparse perspective j The projection matrix below, Represents the uniform sampling in the inverse depth domain. m A deep candidate region, Representing a sparse perspective j Projected onto sparse view i The obtained projection features, Indicates deep candidate regions and sparse perspective i The corresponding cost volume, Indicates deep candidate region and sparse perspective i The corresponding cost volume, b Indicates the number of deep candidate regions. Representing a sparse perspective i Corresponding cost volume; By lightweight UNet network and a convolutional neural network-based up-sampler , the cost volume V is feature refined and channel aligned to obtain a refined cost volume : ； ； wherein, denotes a sparse view angle i corresponding to a refining cost volume, denotes a lightweight UNet network; Refining the cost volume and the depth candidate region G into a depth prediction module to obtain a depth estimation map N for the sparse view image: ； ； ； wherein, denotes a sparse view i a depth estimation map of the corresponding sparse view image, denotes a Softmax function; corresponding to the sparse view images and depth estimation map N of the sparse view images into a three-dimensional Gaussian predictor head, obtaining three-dimensional Gaussian parameters needed for three-dimensional scene reconstruction : ； A three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new view image.

8. A sparse-view 3D scene reconstruction system based on multi-modal feature guided, characterized in that, The sparse view three-dimensional scene reconstruction system based on multi-modal feature guidance includes: The model training module is configured to freeze parameters of an initial image segmentation model and an initial text encoder, obtain an image segmentation model and a text encoder with frozen parameters, train an initial sparse-view image encoder, an initial multi-modal guidance module, and an initial three-dimensional Gaussian parameter predictor, and obtain a trained sparse-view image encoder, a multi-modal guidance module, and a three-dimensional Gaussian parameter predictor. The data acquisition module is configured to acquire a sparse-view image, pre-process the sparse-view image, and obtain a sparse-view text description corresponding to the sparse-view image. The feature extraction module is configured to input the sparse-view image into the sparse-view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, and input the sparse-view text description into the text encoder, and output text features. The feature fusion module is configured to input the shallow image features, the segmentation features, and the text features into the multi-modal guidance module, optimize and refine the shallow image features by using the segmentation features and the text features, and output multi-modal fusion features. The three-dimensional scene reconstruction module is configured to input the multi-modal fusion features into the three-dimensional Gaussian parameter predictor, output three-dimensional Gaussian parameters, reconstruct a three-dimensional scene according to the three-dimensional Gaussian parameters, and obtain a new-view image.

9. A terminal, characterized by comprising: The terminal comprises a memory, a processor, and a multi-modal feature guided sparse-view three-dimensional scene reconstruction program stored on the memory and executable on the processor. When the multi-modal feature guided sparse-view three-dimensional scene reconstruction program is executed by the processor, the steps of the multi-modal feature guided sparse-view three-dimensional scene reconstruction method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-modal feature guided sparse-view three-dimensional scene reconstruction program. When the multi-modal feature guided sparse-view three-dimensional scene reconstruction program is executed by the processor, the steps of the multi-modal feature guided sparse-view three-dimensional scene reconstruction method according to any one of claims 1-7 are implemented.

Images