Three-dimensional scene reconstruction method under sparse view angle based on multi-modal feature guidance

Abstract

The invention discloses a three-dimensional scene reconstruction method under a sparse view angle based on multi-modal feature guidance. The method comprises the following steps: training an initial sparse view angle image encoder, an initial multi-modal guidance module and an initial three-dimensional Gaussian parameter predictor; respectively inputting the sparse view angle image into a trained sparse view angle image encoder and an image segmentation model of the frozen parameters to obtain shallow image features and segmentation features, and inputting the sparse view angle text description into a text encoder of the frozen parameters to obtain text features; in the trained multi-modal guide module, optimizing and refining the shallow image features by using the segmentation features and the text features to obtain multi-modal fusion features; and reconstructing a three-dimensional scene according to the multi-modal fusion features by using the trained three-dimensional Gaussian parameter predictor. According to the method, the features of image and text modals are fused, under the guidance of the multi-modal features, high-quality three-dimensional scene reconstruction is realized by using the sparse view angle image, and the robustness is relatively high.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional reconstruction technology, and in particular to a method, system, terminal and computer-readable storage medium for reconstructing a three-dimensional scene under sparse viewing angles based on multimodal feature guidance. Background Art

[0002] 3D scene reconstruction is an important research direction in the field of computer vision, and has wide application value in many fields such as autonomous driving, virtual reality, and industrial manufacturing. The algorithms used for 3D scene reconstruction can be divided into two categories according to the technical methods: 3D reconstruction algorithms based on traditional multi-view geometry and 3D reconstruction algorithms based on deep learning. In recent years, with the rapid development of deep learning technology, more and more researchers are committed to developing more efficient neural network models to improve the effect and performance of 3D scene reconstruction.

[0003] However, when current deep learning-based 3D reconstruction algorithms perform 3D scene reconstruction under more challenging sparse viewing conditions, they have weak extrapolation capabilities for incomplete or large-area occluded objects in the image, making it difficult to achieve high-quality 3D scene reconstruction.

[0004] Therefore, the prior art still needs to be improved and developed. Summary of the invention

[0005] The main purpose of the present invention is to provide a method, system, terminal and computer-readable storage medium for three-dimensional scene reconstruction under sparse perspective based on multimodal feature guidance, aiming to solve the problem that existing deep learning-based three-dimensional reconstruction algorithms are difficult to achieve high-quality three-dimensional scene reconstruction under more challenging sparse perspective conditions.

[0006] To achieve the above-mentioned object of the invention, the present invention provides a method for reconstructing a three-dimensional scene under a sparse viewing angle based on multimodal feature guidance, and the method for reconstructing a three-dimensional scene under a sparse viewing angle based on multimodal feature guidance comprises:

[0007] Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor to obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor;

[0008] Acquire a sparse perspective image, and preprocess the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image;

[0009] Inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, and inputting the sparse view text description into the text encoder, outputting text features;

[0010] Inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features using the segmentation features and the text features, and outputting multimodal fusion features;

[0011] The multimodal fusion features are input into the three-dimensional Gaussian parameter predictor, and three-dimensional Gaussian parameters are output. The three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new perspective image.

[0012] Optionally, the initial image segmentation model, the initial text encoder, the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor constitute an initial three-dimensional scene reconstruction model;

[0013] The method freezes the parameters of the initial image segmentation model and the initial text encoder, obtains the image segmentation model and the text encoder with frozen parameters, trains the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtains the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor, specifically including:

[0014] Obtaining a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain a preprocessed training set and a test set;

[0015] Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters;

[0016] The preprocessed training set is used to perform end-to-end training on the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and the parameters of the initial three-dimensional scene reconstruction model are optimized to minimize the loss function through a gradient back-propagation mechanism, thereby obtaining the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor;

[0017] Combine the frozen parameter image segmentation model and text encoder, as well as the trained sparse view image encoder, multimodal guidance module and 3D Gaussian parameter predictor to obtain a trained 3D scene reconstruction model;

[0018] The trained 3D scene reconstruction model is tested using the preprocessed test set, and if the test result meets the requirements, the trained 3D scene reconstruction model is used as the final 3D scene reconstruction model;

[0019] The loss function includes learnable perceptual image block similarity and mean square error.

[0020] Optionally, the acquiring of a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain a preprocessed training set and a test set, specifically includes:

[0021] Acquire a data set, the data set comprising a plurality of videos and a camera parameter corresponding to each video frame in the plurality of videos;

[0022] Randomly divide the data set into a training set and a test set according to a preset division method;

[0023] Each video in the training set is divided into a set number of sampling groups, the first frame of each sampling group is processed by the multimodal large model, a text description corresponding to the first frame of the sampling group is generated, and all video frames of the same sampling group use the text description corresponding to the first frame of the sampling group to obtain a preprocessed training set;

[0024] All video frames of each video in the test set are processed by a multimodal large model to generate text descriptions corresponding to all video frames in the test set, thereby obtaining a preprocessed test set.

[0025] Optionally, the acquiring a sparse perspective image and preprocessing the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image specifically includes:

[0026] Obtaining a sparse perspective image and camera parameters corresponding to the sparse perspective image;

[0027] The sparse perspective image is processed by a multimodal large model to generate a sparse perspective text description corresponding to the sparse perspective image.

[0028] Optionally, the step of inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, and inputting the sparse view text description into the text encoder, and outputting text features, specifically includes:

[0029] The sparse perspective image is denoted as I, and the sparse perspective text description corresponding to the sparse perspective image is denoted as T:

[0030]

[0031] Among them, i represents the sparse view, K represents the number of sparse view, I i represents the sparse view image corresponding to the sparse view i, h and w represent the length and width of the sparse view image respectively, R represents the real number matrix, T i represents the sparse perspective text description corresponding to the sparse perspective i, and l represents the length of the sparse perspective text description;

[0032] The sparse view image I is input into the sparse view image encoder E i In the output, shallow image features F W :

[0033] F W =E i (I);

[0034]

[0035] Among them, c represents the number of channels of shallow image features, F W i Represents the shallow image features corresponding to the sparse view angle i;

[0036] Input the sparse view image I into the image segmentation model S i In the output segmentation feature F S :

[0037] F S =S i (I);

[0038]

[0039] Among them, F S i Represents the segmentation features corresponding to the sparse view i;

[0040] Input the sparse view text description T to the text encoder E t In the output text feature F E :

[0041] F E =E t (T);

[0042]

[0043] Where d represents the dimension of the hidden state vector of each word, and F E i Represents the text features corresponding to the sparse view i.

[0044] Optionally, the multimodal guidance module includes a cross-modal cross-attention module, a 1×1 convolutional layer and a visual cross-attention module;

[0045] The step of inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting the multimodal fusion features specifically includes:

[0046] First use the cross-modal cross attention module f ca Fusion segmentation feature F S and text features F E , and then use a 1×1 convolutional layer f ch Compress the channel dimension to obtain the cross-modal guidance feature F C :

[0047] F C =f ch (f ca (F S ,F E ));

[0048]

[0049] Among them, F C i represents the cross-modal guidance features corresponding to the sparse view i;

[0050] Using the visual criss-cross attention module f v Fusion of cross-modal guidance features F C and shallow image features F W , and obtain the multimodal fusion feature M:

[0051] M=f v (F C ,F W );

[0052]

[0053] Among them, M i Represents the multimodal fusion features corresponding to the sparse view i.

[0054] Optionally, the three-dimensional Gaussian parameter predictor includes a plane scanning module, a lightweight UNet network, an upsampler based on a convolutional neural network, a depth prediction module and a three-dimensional Gaussian prediction head;

[0055] The step of inputting the multimodal fusion features into the three-dimensional Gaussian parameter predictor, outputting three-dimensional Gaussian parameters, and reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new perspective image specifically includes:

[0056] Through the plane scanning module, the cost volume V corresponding to the multimodal fusion feature M is constructed in the three-dimensional space:

[0057]

[0058] Among them, P(·) represents the feature projection operation, m j Denotes the multimodal fusion features of another sparse view j relative to sparse view i, D i and D j denote the projection matrices under sparse view i and sparse view j respectively, d m represents the mth depth candidate region uniformly sampled in the inverse depth domain, represents the projection feature obtained by projecting sparse view j to sparse view i, represents the depth candidate region d m and the cost volume corresponding to the sparse view i, represents the depth candidate region d b and the cost volume corresponding to the sparse view i, b represents the number of depth candidate regions, V i represents the cost volume corresponding to the sparse view i;

[0059] Through the lightweight UNet network U and the upsampler f based on convolutional neural network up , perform feature refinement and channel alignment on the cost volume V to obtain the refined cost volume V C :

[0060] V C =f up (V+U(M,V));

[0061]

[0062] Among them, V C i represents the refined cost volume corresponding to the sparse view i, and U(·) represents the lightweight UNet network;

[0063] Refine the cost volume V C The candidate depth region G is input into the depth prediction module to obtain the depth estimation map N of the sparse view image:

[0064] N=U(f s (V C )·G);

[0065] G=[d1,d2,…,d b ]∈R h×w ;

[0066]

[0067] Among them, N i represents the depth estimation map of the sparse view image corresponding to the sparse view i, fs (·) represents the Softmax function;

[0068] The camera parameters p corresponding to the sparse view image c The depth estimation map N of the sparse view image is input into the 3D Gaussian prediction head P' to obtain the 3D Gaussian parameters p required for 3D scene reconstruction. g :

[0069] p g =P'(N,p c );

[0070] The three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new viewing angle image.

[0071] To achieve the above-mentioned object of the invention, the present invention further provides a sparse perspective three-dimensional scene reconstruction system based on multimodal feature guidance, and the sparse perspective three-dimensional scene reconstruction system based on multimodal feature guidance comprises:

[0072] Model training module: used to freeze the parameters of the initial image segmentation model and the initial text encoder, obtain the image segmentation model and text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor;

[0073] Data acquisition module: used for acquiring a sparse perspective image, and preprocessing the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image;

[0074] Feature extraction module: used to input the sparse view image into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, input the sparse view text description into the text encoder, and output text features;

[0075] Feature fusion module: used for inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting multimodal fusion features;

[0076] 3D scene reconstruction module: used to input the multimodal fusion features into the 3D Gaussian parameter predictor, output 3D Gaussian parameters, reconstruct the 3D scene according to the 3D Gaussian parameters, and obtain a new perspective image.

[0077] To achieve the above-mentioned purpose of the invention, the present invention also provides a terminal, which includes: a memory, a processor, and a three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance stored in the memory and run on the processor, wherein the three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance, when executed by the processor, implements the steps of the three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance as described above.

[0078] To achieve the above-mentioned purpose of the invention, the present invention also provides a computer-readable storage medium, which stores a three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance. When the three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance is executed by a processor, the steps of the three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance as described above are implemented.

[0079] In the present invention, the parameters of the initial image segmentation model and the initial text encoder are frozen to obtain the image segmentation model and text encoder with frozen parameters, the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor are trained to obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor; a sparse view image is obtained, and the sparse view image is preprocessed to obtain a sparse view text description corresponding to the sparse view image; the sparse view image is respectively input into the sparse view image encoder and the image segmentation model, and shallow image features and segmentation features are output; the sparse view text description is input into the text encoder, and text features are output; the shallow image features, the segmentation features and the text features are input into the multimodal guidance module, the shallow image features are optimized and refined by using the segmentation features and the text features, and multimodal fusion features are output; the multimodal fusion features are input into the three-dimensional Gaussian parameter predictor, and three-dimensional Gaussian parameters are output, and the three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new view image. The present invention optimizes the 3D scene reconstruction model, introduces segmentation features and text feature supplementary information for shallow image features extracted from sparse view images, fuses the features of image modality and text modality, obtains multimodal features with strong extrapolation capability, and under the guidance of multimodal features, uses sparse view images to achieve high-quality 3D scene reconstruction, and the generated new view images have excellent visual quality and are highly efficient in generating new view images; the optimized 3D scene reconstruction model has high robustness, ensuring that high-quality 3D scene reconstruction can be achieved in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 It is a flowchart of a preferred embodiment of the method for reconstructing a three-dimensional scene under sparse viewing angles guided by multimodal features of the present invention;

[0081] Figure 2 It is a structural diagram of the three-dimensional scene reconstruction model of the present invention;

[0082] Figure 3 is a structural diagram of a multi-modal guidance module of the present invention;

[0083] Figure 4 It is a structural diagram of a preferred embodiment of a sparse viewing angle three-dimensional scene reconstruction system guided by multimodal features of the present invention;

[0084] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical solution and advantages of the present invention clearer and more specific, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0086] 3D scene reconstruction is an important research direction in the field of computer vision, and has wide application value in many fields such as autonomous driving, virtual reality, and industrial manufacturing. The algorithms used for 3D scene reconstruction can be divided into two categories according to the technical methods: 3D reconstruction algorithms based on traditional multi-view geometry and 3D reconstruction algorithms based on deep learning. In recent years, with the rapid development of deep learning technology, more and more researchers are committed to developing more efficient neural network models to improve the effect and performance of 3D scene reconstruction.

[0087] At present, the mainstream deep learning-based 3D reconstruction algorithms mainly use neural implicit representation methods such as neural radiation fields to describe 3D scenes by fusing neural networks with position and viewpoint conditions. However, this process is usually computationally intensive and requires a lot of training time and rendering resources. The 3D Gaussian Splatting technology models the scene through a set of Gaussian ellipsoids and rasterizes the Gaussian ellipsoids into images for efficient rendering. Due to its explicit representation and highly parallel workflow, this method significantly reduces training costs and improves real-time performance. In addition, the application of 3DGaussian Splatting technology has promoted the development of a series of downstream tasks such as 3D reconstruction, 3D editing, and 3D generation.

[0088] However, current 3D Gaussian Splatting-based methods still have some shortcomings when reconstructing 3D scenes under more challenging sparse view conditions. For example, although convolutional neural networks and visual transformers can effectively extract visual information from sparse view images and complete 3D reconstruction in regular scenes, these methods have weak extrapolation capabilities for incomplete or large-area occluded objects in the image, making it difficult to achieve high-quality 3D scene reconstruction.

[0089] In order to solve the above technical problems, the present invention provides a method for reconstructing a three-dimensional scene under a sparse perspective based on multimodal feature guidance, freezing the parameters of an initial image segmentation model and an initial text encoder to obtain an image segmentation model and a text encoder with frozen parameters, training an initial sparse perspective image encoder, an initial multimodal guidance module and an initial three-dimensional Gaussian parameter predictor to obtain a trained sparse perspective image encoder, a multimodal guidance module and a three-dimensional Gaussian parameter predictor; obtaining a sparse perspective image, and preprocessing the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image; and converting the sparse perspective image into a sparse perspective text description. The image is input into the sparse view image encoder and the image segmentation model respectively, and the shallow image features and segmentation features are output. The sparse view text description is input into the text encoder, and the text features are output; the shallow image features, the segmentation features and the text features are input into the multimodal guidance module, and the shallow image features are optimized and refined by using the segmentation features and the text features, and the multimodal fusion features are output; the multimodal fusion features are input into the three-dimensional Gaussian parameter predictor, and the three-dimensional Gaussian parameters are output. The three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new view image. The present invention optimizes the three-dimensional scene reconstruction model, introduces the supplementary information of the segmentation features and the text features for the shallow image features extracted from the sparse view image, fuses the features of the image modality and the text modality, and obtains a multimodal feature with strong extrapolation capability. Under the guidance of the multimodal features, the sparse view image is used to realize high-quality three-dimensional scene reconstruction, and the generated new view image has excellent visual quality, and the efficiency of generating the new view image is high; the optimized three-dimensional scene reconstruction model has high robustness, ensuring that high-quality three-dimensional scene reconstruction can be achieved in different environments.

[0090] The application content is further explained below through the description of embodiments in conjunction with the accompanying drawings.

[0091] A preferred embodiment of the method for reconstructing a three-dimensional scene under sparse viewing angles guided by multimodal features of the present invention is as follows: Figure 1 and Figure 2 As shown, specifically including:

[0092] S1: Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor.

[0093] In an implementation of this embodiment, the initial image segmentation model, the initial text encoder, the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor constitute an initial three-dimensional scene reconstruction model;

[0094] The method freezes the parameters of the initial image segmentation model and the initial text encoder, obtains the image segmentation model and the text encoder with frozen parameters, trains the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtains the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor, specifically including:

[0095] S11: Acquire a data set, divide the data set into a training set and a test set, and preprocess the training set and the test set respectively to obtain a preprocessed training set and a test set;

[0096] S12: Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain an image segmentation model and a text encoder with frozen parameters;

[0097] S13: using the preprocessed training set to perform end-to-end training on the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and optimizing the parameters of the initial three-dimensional scene reconstruction model to minimize the loss function through a gradient back-propagation mechanism, thereby obtaining the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor;

[0098] S14: combining the frozen parameter image segmentation model and text encoder, as well as the trained sparse view image encoder, multimodal guidance module and 3D Gaussian parameter predictor, to obtain a trained 3D scene reconstruction model;

[0099] S15: testing the trained 3D scene reconstruction model using the preprocessed test set, and if the test result meets the requirements, using the trained 3D scene reconstruction model as the final 3D scene reconstruction model;

[0100] The loss function includes learnable perceptual image block similarity and mean square error.

[0101] Specifically, during the training and testing process of the network, the present invention first selects a large-scale, high-quality 3D reconstruction dataset RealEstate10K, randomly divides the dataset into a training set and a test set according to its preset division method, and then generates a text description for the RealEstate10K dataset through a multimodal large model Qwen2-VL. The present invention selects the number of sparse perspectives as 2, and reconstructs the 3D scene based on two sparse perspective input images (i.e., sparse perspective images). Two video frames with an interval frame number greater than 25 are randomly selected from the video frames of the training scene (i.e., the training set) as input images, and the images are adjusted to a size of 256×256 pixels to enhance the diversity of the data. In the training phase, the model is trained end-to-end. Through the gradient backpropagation mechanism, the model parameters are optimized to minimize the loss function, so that the model can achieve high-quality 3D scene reconstruction under sparse perspective. In the testing phase, the network weight parameters saved by the preset maximum number of training times during the training process are loaded into the model, the corresponding sparse perspective images and the text descriptions corresponding to the images are input, the 3D scene is reconstructed, and a rendered image of a new perspective is generated. In model training, the model of the present invention uses rendering loss for end-to-end training, and uses Learned Perceptual Image Patch Similarity (LPIPS) and Mean Square Error (MSE) as loss functions to supervise the difference between the rendered image (i.e., new perspective image) and the real image of the reconstructed scene at a new perspective to ensure the quality of the reconstructed 3D scene. The overall form of the loss function is:

[0102] L=L MSE +λ·L LPIPS

[0103] Among them, L MSE is the MSE loss, L LPIPS is the LPIPS loss, and λ is a weighted hyperparameter, which indicates the weight of the LPIPS loss in the overall loss function and can be set to 0.05.

[0104] In order to evaluate the performance of the model, the present invention calculates the peak signal-to-noise ratio (PSNR), structural similarity (SSIM), and learnable perceptual image patch similarity (LPIPS) between the rendered image of the new perspective and the real image. Among them, the higher the values ​​of PSNR and SSIM, the lower the value of LPIPS, indicating that the quality of the rendered image of the new perspective is better, the quality of the 3D scene reconstruction is higher, and the performance of the model is better.

[0105] In addition, in order to verify the effectiveness of the present invention, the proposed model was trained and tested on a server equipped with NVIDIA RTXA6000 GPU and Intel Xeon Gold 6330CPU, and implemented using the open source Pytorch deep learning framework. The model was trained end-to-end on four NVIDIA RTXA6000 GPUs, with a total of 180,000 iterations until the loss converged. The initial learning rate was set to 2×10-4, and the learning rate was adjusted using warm-up and cosine annealing strategies. Two video frames with an interval of more than 25 frames were randomly selected from the training scene video frames as input images. The images were adjusted to a size of 256×256 pixels, and text descriptions were generated using the multimodal large model Qwen2-VL. The image segmentation model and text encoder with frozen parameters used the OMG-SEG model and LongCLIP model that have been trained on the image segmentation dataset and cross-modal alignment dataset, respectively.

[0106] In the 3D scene reconstruction model of the present invention, the parameters of the image segmentation model and the text encoder are frozen, and the sparse view image encoder, the multimodal guidance module and the 3D Gaussian parameter predictor are trained end-to-end. Through the gradient back-propagation mechanism, the model parameters are optimized to minimize the loss function, so that the model can achieve high-quality 3D scene reconstruction under sparse view. The structure of the 3D scene reconstruction model is as follows: Figure 2 As shown. The present invention fully considers the characteristics of multimodal feature fusion and three-dimensional scene reconstruction, and provides a three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance. Experimental results show that under the guidance of multimodal features, the present invention uses sparse perspective images to achieve three-dimensional scene reconstruction and exhibits superior performance. The reconstructed three-dimensional scene has strong extrapolation capabilities, and the image generated at the new perspective has excellent visual quality. In addition, for different scenes, the model can still effectively reconstruct the three-dimensional scene with high robustness. The specific beneficial effects can be summarized as follows:

[0107] (1) High-quality 3D scene reconstruction: The reconstructed 3D scene has low distortion, maintains visual coherence, and can render clear images from new perspectives. The PSNR, SSIM, and LPIPS indicators are used for evaluation. The PSNR is higher than 25dB, the SSIM is higher than 0.85, and the LPIPS is lower than 0.15.

[0108] (2) Highly efficient generation of new perspective images: The time required to render a new perspective image based on the reconstructed 3D scene does not exceed 2 seconds.

[0109] In an implementation of this embodiment, the step of obtaining a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain the preprocessed training set and test set specifically includes:

[0110] S111: Acquire a data set, where the data set includes a plurality of videos and a camera parameter corresponding to each video frame in the plurality of videos;

[0111] S112: randomly dividing the data set into a training set and a test set according to a preset division method;

[0112] S113: Divide each video in the training set into a set number of sampling groups, process the first frame of each sampling group using the multimodal large model, generate a text description corresponding to the first frame of the sampling group, and use the text description corresponding to the first frame of the sampling group for all video frames in the same sampling group to obtain a preprocessed training set;

[0113] S114: All video frames of each video in the test set are processed by the multimodal large model to generate text descriptions corresponding to all video frames in the test set, thereby obtaining a preprocessed test set.

[0114] Specifically, during the training and testing process of the network, the present invention selects the public large-scale high-quality 3D reconstruction dataset RealEstate10K to train and test the proposed 3D scene reconstruction model. The RealEstate10K dataset contains about 80,000 video clips and about 10 million frames collected from video websites. After removing low-quality data, the RealEstate10K dataset is divided into 67477 training scenes (i.e., training sets) and 7289 test scenes (i.e., test sets). In addition, the RealEstate10K dataset provides estimated camera intrinsic parameters and extrinsic parameters (i.e., camera parameters) for each video frame. The present invention generates a text description for the RealEstate10K dataset through the multimodal large model Qwen2-VL. For the training scene, each video is evenly divided into 5 sampling groups, and a text description is generated for the first frame of each sampling group. All video frames of the same sampling group use the text description of the first frame of the sampling group; for the test scene, a text description is generated for all video frames. It should be noted that the preprocessed training set and test set both include video frames and camera parameters and text descriptions corresponding to the video frames, wherein the video frames are images under sparse viewing conditions.

[0115] S2: Acquire a sparse perspective image, and preprocess the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image.

[0116] In an implementation of this embodiment, the acquiring of the sparse perspective image and preprocessing of the sparse perspective image to obtain the sparse perspective text description corresponding to the sparse perspective image specifically includes:

[0117] S21: Acquire a sparse perspective image and camera parameters corresponding to the sparse perspective image;

[0118] S22: Process the sparse perspective image through a multimodal large model to generate a sparse perspective text description corresponding to the sparse perspective image.

[0119] Specifically, the present invention processes the input image under sparse view conditions through a multimodal large model to obtain a text description of the input image, supplement the semantic information for the features extracted from the input image, so as to obtain a refined feature representation with strong extrapolation capability, and realize high-quality 3D scene reconstruction. It should be noted that the sparse view image here is the input image to be reconstructed into a 3D scene, and the video frames in the above data set are the input images used to train and test the model.

[0120] S3: Input the sparse view image into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, input the sparse view text description into the text encoder, and output text features.

[0121] In an implementation of this embodiment, the step of inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, and inputting the sparse view text description into the text encoder, and outputting text features, specifically includes:

[0122] S31: The sparse perspective image is recorded as I, and the sparse perspective text description corresponding to the sparse perspective image is recorded as T:

[0123]

[0124] Among them, i represents the sparse view (i.e., the i-th sparse view), K represents the number of sparse views, and I i represents the sparse view image corresponding to sparse view i (i.e., under sparse view i), h and w represent the length and width of the sparse view image respectively, R represents a real number matrix, T i represents the sparse perspective text description corresponding to the sparse perspective i, and l represents the length of the sparse perspective text description;

[0125] S32: Input the sparse view image I to the sparse view image encoder E i In the output, the shallow image features F of the sparse view image are W :

[0126] F W =E i (I);

[0127]

[0128] Among them, c represents the number of channels of shallow image features, F W i Represents the shallow image features of the sparse view image corresponding to the sparse view i;

[0129] S33: Input the sparse view image I into the image segmentation model S i In the output, the segmentation feature F of the sparse view image is S :

[0130] F S =S i (I);

[0131]

[0132] Among them, F S i Represents the segmentation features of the sparse view image corresponding to the sparse view i;

[0133] S34: Input the sparse view text description T to the text encoder E t In the output, the text features F of the sparse perspective text description are E :

[0134] F E =E t (T);

[0135]

[0136] Where d represents the dimension of the hidden state vector of each word, and F E i Represents the text features of the sparse view text description corresponding to sparse view i.

[0137] Specifically, the sparse view image I and the sparse view text description T are the input of the 3D scene reconstruction model. The sparse view image I is input into the sparse view image encoder E. i In the above example, the shallow image features F of the sparse view image are extracted. W , input the sparse view image I into the image segmentation model S i In the above example, we extract the segmentation features F of the sparse view image. S , the sparse view text description T is input into the text encoder E t In the above example, we extract the text features F of the sparse viewpoint text description. E , get the features of image and text models.

[0138] S4: Input the shallow image features, the segmentation features and the text features into the multimodal guidance module, use the segmentation features and the text features to optimize and refine the shallow image features, and output multimodal fusion features.

[0139] In one implementation of this embodiment, Figure 3 As shown, the multimodal guidance module includes a cross-modal cross-attention module, a 1×1 convolutional layer and a visual cross-attention module, the cross-modal cross-attention module includes a multi-head self-attention layer and a multi-head cross-attention layer, and the visual cross-attention module includes a window self-attention layer and a window cross-attention layer;

[0140] The step of inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting the multimodal fusion features specifically includes:

[0141] S41: First use the cross-modal cross attention module f ca Fusion segmentation feature F S and text features F E The cross-modal attention module consists of a multi-head self-attention layer and a multi-head cross-attention layer, and then a 1×1 convolutional layer f ch Compress the channel dimension to obtain the cross-modal guidance feature F of the sparse view image C :

[0142] F C =f ch (f ca (F S ,F E ));

[0143]

[0144] Among them, F C i Represents the cross-modal guidance features of the sparse view image corresponding to the sparse view i;

[0145] S42: Using visual cross-attention module f v Fusion of cross-modal guidance features F C and shallow image features F W , the visual cross-attention module contains a window self-attention layer and a window cross-attention layer, that is, each feature map is divided into four sub-feature maps, and the multimodal fusion feature M of the sparse view image is obtained:

[0146] M=f v (F C ,FW );

[0147]

[0148] Among them, M i Represents the multimodal fusion features of the sparse view image corresponding to the sparse view i.

[0149] Specifically, the present invention introduces the supplementary information of the segmentation model and the text encoder (segmentation features and text features) into the features extracted from the sparse view images (shallow image features), and obtains a refined feature representation (multimodal fusion features) with strong extrapolation capability. By analyzing the gradual fusion mechanism of multimodal features, it is ensured that the multimodal features can supplement the semantic information in the three-dimensional scene reconstruction.

[0150] S5: Input the multimodal fusion features into the three-dimensional Gaussian parameter predictor, output three-dimensional Gaussian parameters, reconstruct the three-dimensional scene according to the three-dimensional Gaussian parameters, and obtain a new perspective image.

[0151] In one implementation of this embodiment, the three-dimensional Gaussian parameter predictor includes a plane scanning module, a lightweight UNet network, an upsampler based on a convolutional neural network, a depth prediction module and a three-dimensional Gaussian prediction head;

[0152] The step of inputting the multimodal fusion features into the three-dimensional Gaussian parameter predictor, outputting three-dimensional Gaussian parameters, and reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new perspective image specifically includes:

[0153] S51: Through the plane scanning module, a cost volume V corresponding to the multimodal fusion feature M is constructed in the three-dimensional space, and the cross-view feature similarities of all potential depth candidate regions are stored:

[0154]

[0155] Among them, P(·) represents the feature projection operation, M j Denotes the multimodal fusion features of another sparse view j relative to sparse view i, D i and D j denote the projection matrices under sparse view i and sparse view j respectively, d m represents the mth depth candidate region uniformly sampled in the inverse depth domain, represents the projection feature obtained by projecting sparse view j to sparse view i, represents the depth candidate region d m and the cost volume corresponding to the sparse view i, represents the depth candidate region d b and the cost volume corresponding to the sparse view i, b represents the number of depth candidate regions, Vi represents the cost volume corresponding to the sparse view i;

[0156] S52: Through the lightweight UNet network U and the upsampler f based on the convolutional neural network up , perform feature refinement and channel alignment on the cost volume V to obtain the refined cost volume V C :

[0157] V C =f up (V+U(M,V));

[0158]

[0159] Among them, V C i represents the refined cost volume corresponding to the sparse view i, and U(·) represents the lightweight UNet network;

[0160] S53: Refine the cost volume V C and the depth candidate region G are input to the depth prediction module (including the lightweight UNet network U and the Softmax function f s ), the depth estimation map N of the sparse perspective image is obtained:

[0161] N=U(f s (V C )·G);

[0162] G=[d1,d2,…,d b ]∈R h×w ;

[0163]

[0164] Among them, N i represents the depth estimation map of the sparse view image corresponding to the sparse view i, f s (·) represents the Softmax function;

[0165] S54: The camera parameters p corresponding to the sparse view image are c The depth estimation map N of the sparse view image is input into the 3D Gaussian prediction head P' to obtain the 3D Gaussian parameters p required for 3D scene reconstruction. g :

[0166] p g =P'(N,p c );

[0167] S55: Reconstruct the three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new viewing angle image.

[0168] Specifically, the 3D Gaussian parameters required for reconstruction are predicted by multimodal fusion features. Under the guidance of multimodal features, sparse view images are used to achieve 3D scene reconstruction, which shows superior performance. The reconstructed 3D scene has strong extrapolation ability, and the images generated from new viewpoints have excellent visual quality. The model has high efficiency in generating new viewpoint images. In addition, for different scenes, the model can still effectively reconstruct 3D scenes and has high robustness.

[0169] In addition, based on the above-mentioned sparse perspective three-dimensional scene reconstruction method based on multimodal feature guidance, the present invention also provides a sparse perspective three-dimensional scene reconstruction system based on multimodal feature guidance, wherein a preferred embodiment of the sparse perspective three-dimensional scene reconstruction system based on multimodal feature guidance is as follows: Figure 4 As shown, specifically including:

[0170] Model training module 01: used to freeze the parameters of the initial image segmentation model and the initial text encoder, obtain the image segmentation model and text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor;

[0171] Data acquisition module 02: used to acquire a sparse perspective image, and pre-process the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image;

[0172] Feature extraction module 03: used to input the sparse view image into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, input the sparse view text description into the text encoder, and output text features;

[0173] Feature fusion module 04: used to input the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimize and refine the shallow image features using the segmentation features and the text features, and output multimodal fusion features;

[0174] The three-dimensional scene reconstruction module 05 is used to input the multimodal fusion features into the three-dimensional Gaussian parameter predictor, output three-dimensional Gaussian parameters, reconstruct the three-dimensional scene according to the three-dimensional Gaussian parameters, and obtain a new perspective image.

[0175] In addition, based on the above-mentioned method and system for reconstructing a three-dimensional scene under sparse viewing angles based on multimodal feature guidance, the present invention also provides a terminal accordingly, wherein a preferred embodiment of the terminal is as follows: Figure 5 As shown, it specifically includes a processor 10, a memory 20 and a display 30. Figure 5Only some components of the terminal are shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0176] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory of the terminal. In other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card) equipped on the terminal. Further, the memory 20 may also include both an internal storage unit of the terminal and an external storage device. The memory 20 is used to store application software and various types of data installed on the terminal, such as program codes of the storage terminal. The memory 20 may also be used to temporarily store data that has been output or is to be output. In one embodiment, a sparse perspective three-dimensional scene reconstruction program 40 based on multimodal feature guidance is stored on the memory 20, and the sparse perspective three-dimensional scene reconstruction program 40 based on multimodal feature guidance can be executed by the processor 10, thereby realizing the steps of the sparse perspective three-dimensional scene reconstruction method based on multimodal feature guidance in the present application.

[0177] In some embodiments, the processor 10 can be a central processing unit (CPU), a microprocessor or other data processing chip, used to run the program code or process data stored in the memory 20, such as executing a sparse perspective three-dimensional scene reconstruction program 40 based on multi-modal feature guidance.

[0178] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, an OLED (Organic Light-Emitting Diode) touch device, etc. The display 30 is used to display information on the terminal and to display a visual user interface. The components of the terminal communicate with each other through a system bus.

[0179] In one embodiment, when the processor 10 executes the sparse view angle three-dimensional scene reconstruction program 40 based on multimodal feature guidance in the memory 20, the steps of the sparse view angle three-dimensional scene reconstruction method based on multimodal feature guidance as described above are implemented.

[0180] The present invention also provides a computer-readable storage medium accordingly, wherein the computer-readable storage medium stores a three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance, and when the three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance is executed by a processor, the steps of the three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance as described above are implemented.

[0181] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal. In the absence of further restrictions, an element defined by the sentence "includes a ..." does not exclude the existence of other identical elements in the process, method, article or terminal including the element.

[0182] Of course, those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware (such as a processor, a controller, etc.) through a computer program, and the program can be stored in a computer-readable storage medium that can be read by a computer, and the program can include the processes of the above-mentioned method embodiments when executed. The computer-readable storage medium can be a memory, a disk, an optical disk, etc.

[0183] It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, improvements or changes can be made based on the above description. All these improvements and changes should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A sparse-view three-dimensional scene reconstruction method based on multimodal feature guidance, characterized in that: The sparse perspective three-dimensional scene reconstruction method based on multimodal feature guidance includes: Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor to obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor; Acquire a sparse perspective image, and preprocess the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image; Inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, and inputting the sparse view text description into the text encoder, outputting text features; Inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features using the segmentation features and the text features, and outputting multimodal fusion features; The multimodal fusion features are input into the three-dimensional Gaussian parameter predictor, and three-dimensional Gaussian parameters are output. The three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new perspective image.

2. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 1, characterized in that: The initial image segmentation model, the initial text encoder, the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor constitute an initial three-dimensional scene reconstruction model; The method freezes the parameters of the initial image segmentation model and the initial text encoder, obtains the image segmentation model and the text encoder with frozen parameters, trains the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtains the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor, specifically including: Obtaining a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain a preprocessed training set and a test set; Freeze the parameters of the initial image segmentation model and the initial text encoder to obtain the image segmentation model and the text encoder with frozen parameters; The preprocessed training set is used to perform end-to-end training on the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and the parameters of the initial three-dimensional scene reconstruction model are optimized to minimize the loss function through a gradient back-propagation mechanism, thereby obtaining the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor; Combine the frozen parameter image segmentation model and text encoder, as well as the trained sparse view image encoder, multimodal guidance module and 3D Gaussian parameter predictor to obtain a trained 3D scene reconstruction model; The trained 3D scene reconstruction model is tested using the preprocessed test set, and if the test result meets the requirements, the trained 3D scene reconstruction model is used as the final 3D scene reconstruction model; The loss function includes learnable perceptual image block similarity and mean square error.

3. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 2, characterized in that: The step of obtaining a data set, dividing the data set into a training set and a test set, and preprocessing the training set and the test set respectively to obtain a preprocessed training set and a test set specifically includes: Acquire a data set, the data set comprising a plurality of videos and a camera parameter corresponding to each video frame in the plurality of videos; Randomly divide the data set into a training set and a test set according to a preset division method; Each video in the training set is divided into a set number of sampling groups, the first frame of each sampling group is processed by the multimodal large model, a text description corresponding to the first frame of the sampling group is generated, and all video frames of the same sampling group use the text description corresponding to the first frame of the sampling group to obtain a preprocessed training set; All video frames of each video in the test set are processed by a multimodal large model to generate text descriptions corresponding to all video frames in the test set, thereby obtaining a preprocessed test set.

4. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 1, characterized in that: The acquiring of the sparse perspective image and preprocessing the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image specifically includes: Obtaining a sparse perspective image and camera parameters corresponding to the sparse perspective image; The sparse perspective image is processed by a multimodal large model to generate a sparse perspective text description corresponding to the sparse perspective image.

5. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 4, characterized in that: The step of inputting the sparse view image into the sparse view image encoder and the image segmentation model respectively, outputting shallow image features and segmentation features, and inputting the sparse view text description into the text encoder, and outputting text features, specifically includes: The sparse perspective image is denoted as I, and the sparse perspective text description corresponding to the sparse perspective image is denoted as T: Among them, i represents the sparse view, K represents the number of sparse view, I i represents the sparse view image corresponding to the sparse view i, h and w represent the length and width of the sparse view image respectively, R represents the real number matrix, T i represents the sparse perspective text description corresponding to the sparse perspective i, and l represents the length of the sparse perspective text description; The sparse view image I is input into the sparse view image encoder E i In the output, shallow image features F W : F W =E i (I); Among them, c represents the number of channels of shallow image features, F W i Represents the shallow image features corresponding to the sparse view angle i; Input the sparse view image I into the image segmentation model S i In the output segmentation feature F S : F S =S i (I); Among them, F S i Represents the segmentation features corresponding to the sparse view i; Input the sparse view text description T to the text encoder E t In the output text feature F E : F E =E t (T); Where d represents the dimension of the hidden state vector of each word, and F E i Represents the text features corresponding to the sparse view i.

6. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 5, characterized in that: The multimodal guidance module includes a cross-modal cross-attention module, a 1×1 convolutional layer, and a visual cross-attention module; The step of inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting the multimodal fusion features specifically includes: First use the cross-modal cross attention module f ca Fusion segmentation feature F S and text features F E , and then use a 1×1 convolutional layer f ch Compress the channel dimension to obtain the cross-modal guidance feature F c : F C =f ch (f ca (F S ,F E )); Among them, F C i represents the cross-modal guidance features corresponding to the sparse view i; Using the visual criss-cross attention module f v Fusion of cross-modal guidance features F C and shallow image features F W , and obtain the multimodal fusion feature M: M=f v (F C ,F W ); Among them, M i Represents the multimodal fusion features corresponding to the sparse view i.

7. The method for 3D scene reconstruction under sparse viewing angles based on multimodal feature guidance according to claim 6, characterized in that: The three-dimensional Gaussian parameter predictor includes a plane scanning module, a lightweight UNet network, an upsampler based on a convolutional neural network, a depth prediction module and a three-dimensional Gaussian prediction head; The step of inputting the multimodal fusion features into the three-dimensional Gaussian parameter predictor, outputting three-dimensional Gaussian parameters, and reconstructing a three-dimensional scene according to the three-dimensional Gaussian parameters to obtain a new perspective image specifically includes: Through the plane scanning module, the cost volume V corresponding to the multimodal fusion feature M is constructed in the three-dimensional space: Among them, P(·) represents the feature projection operation, M j Denotes the multimodal fusion features of another sparse view j relative to sparse view i, D i and D j denote the projection matrices under sparse view i and sparse view j respectively, d m represents the mth depth candidate region uniformly sampled in the inverse depth domain, represents the projection feature obtained by projecting sparse view j to sparse view i, represents the depth candidate region d m and the cost volume corresponding to the sparse view i, represents the depth candidate region d b and the cost volume corresponding to the sparse view i, b represents the number of depth candidate regions, V i represents the cost volume corresponding to the sparse view i; Through the lightweight UNet network U and the upsampler f based on convolutional neural network up , perform feature refinement and channel alignment on the cost volume V to obtain the refined cost volume V C : V C =f up (V+U(M,V)); Among them, V C i represents the refined cost volume corresponding to the sparse view i, and U(·) represents the lightweight UNet network; Refine the cost volume V C The candidate depth region G is input into the depth prediction module to obtain the depth estimation map N of the sparse view image: N=U(f s (V C )·G); G=[d1,d2,…,d b ]∈R h×w ; Among them, N i represents the depth estimation map of the sparse view image corresponding to the sparse view i, f s (·) represents the Softmax function; The camera parameters p corresponding to the sparse view image c The depth estimation map N of the sparse view image is input into the 3D Gaussian prediction head P' to obtain the 3D Gaussian parameters p required for 3D scene reconstruction. g : p g =P'(N,p c ); The three-dimensional scene is reconstructed according to the three-dimensional Gaussian parameters to obtain a new viewing angle image.

8. A sparse-view three-dimensional scene reconstruction system based on multimodal feature guidance, characterized in that: The sparse perspective three-dimensional scene reconstruction system based on multimodal feature guidance includes: Model training module: used to freeze the parameters of the initial image segmentation model and the initial text encoder, obtain the image segmentation model and text encoder with frozen parameters, train the initial sparse view image encoder, the initial multimodal guidance module and the initial three-dimensional Gaussian parameter predictor, and obtain the trained sparse view image encoder, the multimodal guidance module and the three-dimensional Gaussian parameter predictor; Data acquisition module: used for acquiring a sparse perspective image, and preprocessing the sparse perspective image to obtain a sparse perspective text description corresponding to the sparse perspective image; Feature extraction module: used to input the sparse view image into the sparse view image encoder and the image segmentation model respectively, output shallow image features and segmentation features, input the sparse view text description into the text encoder, and output text features; Feature fusion module: used for inputting the shallow image features, the segmentation features and the text features into the multimodal guidance module, optimizing and refining the shallow image features by using the segmentation features and the text features, and outputting multimodal fusion features; 3D scene reconstruction module: used to input the multimodal fusion features into the 3D Gaussian parameter predictor, output 3D Gaussian parameters, reconstruct the 3D scene according to the 3D Gaussian parameters, and obtain a new perspective image.

9. A terminal, characterized in that: The terminal includes: a memory, a processor, and a three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance stored in the memory and executable on the processor. When the three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance is executed by the processor, the steps of the three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance are implemented as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance. When the three-dimensional scene reconstruction program under sparse perspective based on multimodal feature guidance is executed by a processor, the steps of the three-dimensional scene reconstruction method under sparse perspective based on multimodal feature guidance as described in any one of claims 1 to 7 are implemented.

Images