Multi-view image three-dimensional reconstruction method based on video data hybrid supervision fine tuning

Abstract

The invention provides a multi-view image three-dimensional reconstruction method based on video data hybrid supervision fine tuning, and belongs to the field of computer vision. The three-dimensional reconstruction method comprises the following steps: constructing a training sample data set based on unlabeled video data, and establishing a pre-trained multi-view three-dimensional reconstruction network for forward reasoning to obtain a three-dimensional point cloud and three-dimensional Gaussian parameters; constructing a mixed supervision training framework of new view rendering supervision based on video consistency, geometric supervision based on external sparse three-dimensional point cloud and anchor regularization supervision based on an original strong supervision data set, and calculating joint loss to optimize a multi-view three-dimensional reconstruction network; and outputting consistent three-dimensional point cloud and renderable three-dimensional Gaussian representation to complete three-dimensional reconstruction. According to the method, the performance of the three-dimensional reconstruction model can be enhanced by effectively utilizing a large-scale unmarked video, the precision of camera pose estimation is improved, and remarkable performance improvement and strong generalization ability are achieved on a plurality of standard evaluation references.

Description

Technical Field

[0001] This invention belongs to the field of computer vision, specifically relating to a multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning. Background Technology

[0002] 3D reconstruction refers to the creation of mathematical models of 3D objects suitable for computer representation and processing. 3D reconstruction technology can be used in fields such as scene reconstruction, various augmented reality (AR) or virtual reality (VR) applications, house layout reconstruction and display, object (3D) display, autonomous driving, drones, and environmental perception.

[0003] Reconstructing 3D scenes from poseless sparse multi-view images or randomly captured videos has always been a significant challenge in computer vision. Multi-view stereo (MVS) reconstruction in natural scenes requires estimating camera parameters (intrinsic and extrinsic parameters). Obtaining these parameters is often tedious and complex, but they are crucial for triangulation of corresponding pixels in 3D space and are the core of all high-performance MVS algorithms. Therefore, many methods have been developed that use end-to-end data-driven networks to directly estimate pixel alignment maps in a unified coordinate system, replacing the traditional Structure for Motion Restoration (SfM) process.

[0004] For example, Reference 1 (DUSt3R: Geometric 3D Vision Made Easy) discloses a novel paradigm for dense and unconstrained stereo 3D reconstruction of arbitrary image sets, operating without prior information about camera calibration or viewpoint pose. It transforms the 3D reconstruction problem of paired images into a pointmap regression problem, relaxing the strict constraints of typical projection camera models. It demonstrates that this formalized expression smoothly unifies monocular and binocular reconstruction. For cases with more than two images, a global alignment strategy is further proposed, which represents all paired pointmaps within a common reference frame. Similarly, Reference 2 (Grounding ImageMatching in 3D with MASt3R) discloses a Transformers-based 3D reconstruction framework based on pointmap regression, exhibiting robustness when matching views with extreme viewpoint variations. It enhances the DUSt3R network with a new head, outputting dense local features, and trains with additional matching loss. Furthermore, it introduces a fast reciprocal matching scheme, accelerating the matching process by several orders of magnitude while maintaining theoretical guarantees. Further work such as MV-DUSt3R, Fast3R, and VGGT has extended to multi-view input, achieving dense 3D reconstruction with a single forward propagation.

[0005] While these models perform admirably, they nearly exhaust the available high-quality 3D supervised datasets and heavily rely on costly sensor annotations. This scarcity of diverse, scalable supervised data has become a key bottleneck, limiting further improvements in 3D reconstruction capabilities. In stark contrast, unlabeled web videos offer a vast amount of diverse data that naturally contains multi-view information and is extremely inexpensive to acquire. For example, the RealEstate10K dataset from YouTube videos contains over 20,000 unique scenes, far exceeding the approximately 1,000 scenes found in labeled datasets like ScanNet.

[0006] However, due to the lack of ground truth annotations, this video data has not been fully utilized for 3D reconstruction. Existing attempts, such as using video depth estimation models to generate pseudo-depth labels for supervision, often degrade reconstruction performance due to inconsistencies between predicted depth and pose scales, as well as noise in the depth estimation itself. Therefore, how to effectively utilize unlabeled video data to enhance the performance of 3D reconstruction models has become an urgent technical problem to be solved. Summary of the Invention

[0007] To address the shortcomings of existing technologies, this invention provides a multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning. It constructs a method combining two-dimensional multi-view... Figure 1 A hybrid supervised training framework with uniformity, sparse 3D geometric constraints, and a small amount of strongly supervised data enables the reconstruction of high-quality 3D scenes from disordered images without camera parameters or dense annotations. It also supports the synthesis of new perspectives and is suitable for 3D understanding and generation tasks of static scenes, especially for 3D reconstruction applications of real-world video data.

[0008] To achieve the above-mentioned objectives, this invention provides a multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning, comprising the following steps: Step 1: Sample a training sample dataset containing multiple frames of images from unlabeled video data; Step 2: Input the training sample dataset into the pre-trained multi-view dataset. Figure 3 3D reconstruction network, the multi-view Figure 3 The 3D reconstruction network uses forward inference to output a 3D point cloud aligned with the training sample pixels, as well as 3D Gaussian parameters for new perspective synthesis. Step 3: Based on multi-view Figure 3 The 3D point cloud and 3D Gaussian parameters output by the 3D reconstruction network are used in parallel for new perspective rendering supervision based on video consistency, geometric supervision based on external sparse 3D point clouds, and anchor point regularization supervision based on the original strongly supervised dataset. These are then used to compute the joint loss to optimize multi-view rendering. Figure 3The 3D reconstruction network outputs a consistent 3D point cloud and renderable 3D Gaussian parameters to complete the 3D reconstruction.

[0009] Preferably, step 1 specifically includes: The unlabeled video data in the training sample dataset is uniformly sampled at fixed time intervals to obtain the initial frame sequence; A random perturbation is applied to each sampling time point in the initial frame sequence to obtain a sampling frame sequence with random perturbation as training samples.

[0010] By employing a random perturbation sampling strategy for video frame sampling, a random time offset is introduced for each sampling time point on the basis of fixed interval sampling, so as to increase the diversity of training perspectives while ensuring scene overlap between frames.

[0011] Preferably, in step 2, the pre-trained multi-view... Figure 3 The dimensional reconstruction network, trained on the original dataset, includes: an encoder, a decoder, a regression head, and a Gaussian parameter prediction head; The encoder is used to take training samples as input and output disordered multi-view visual representations. The decoder is used to calculate the cross attention between views based on the disordered arrangement of multi-view visual representations through a cross-reference graph attention mechanism, in order to realize cross-view information exchange and aggregation, and output fused scene features; The regression head is used to regress and predict the three-dimensional spatial coordinates of each pixel in each view, taking the fused scene features as input, and generate a pixel-aligned three-dimensional point cloud. The Gaussian parameter prediction head is used in parallel with the regression head, taking fused scene features as input, to predict 3D Gaussian parameters for each pixel of each view, for use in downstream new perspective synthesis tasks.

[0012] More preferably, the pre-trained multi-view... Figure 3 The 3D reconstruction network is MV-DUSt3R, and the multi-view... Figure 3 During forward inference in the dimensional reconstruction network, the Gaussian parameter prediction head is frozen, and only the regression head parameters are optimized.

[0013] Preferably, the Gaussian parameters include at least: center position, opacity, rotation, scale, and appearance spherical harmonic coefficient.

[0014] Preferably, in step 3, the new perspective rendering supervision based on video consistency includes: using the temporal relationship between image frames in the training samples, taking some frames as context input and other frames as new perspective supervision target frames, synthesizing Gaussian parameters into a new perspective image through differentiable rendering, and calculating the rendering loss between the new perspective image frame and the real target image frame to constrain video consistency. Among them, rendering loss , For the image frames in the training sample dataset, and These are the context frame and the new perspective supervised target frame, respectively. For new perspective image frames, In order to perceive loss, It is the L2 norm. To perceive loss weights, These are L2 norm weights.

[0015] Preferably, in step 3, the geometric supervision based on the external sparse 3D point cloud includes: Sparse 3D point clouds are reconstructed from video segments corresponding to image frames in the training sample dataset using the structure-on-motion method. Sparse 3D point clouds are projected onto the image plane to generate sparse depth supervision, and confidence-aware geometric alignment loss is calculated between the 3D point clouds and the image plane in 3D space to provide geometric scale and positional constraints. Among them, geometric alignment loss , , For view indexing, For the total number of views, For pixel index, For view Medium-strength supervised pixel set, For pixels The prediction confidence level For pixels The normalized geometric regression error, For the predicted three-dimensional coordinates, The three-dimensional coordinates of the sparse truth value. To predict the normalized scaling factor of 3D coordinates, The normalized scaling factor for the sparse true three-dimensional coordinates. is the regularization coefficient.

[0016] Preferably, in step 3, the anchor point regularization supervision based on the original strongly supervised dataset includes: during the training process, using strongly supervised samples with 3D truth labels in the mixed 3D truth dataset as anchor point data for regularization.

[0017] More preferably, the number of strongly supervised samples accounts for approximately 3% of the total video data.

[0018] Preferably, by finding the optimal multi-view Figure 3 3D Reconstruction Network Parameters To minimize the joint loss function in order to compute the joint loss: , For the training sample dataset, For a three-dimensional truth dataset, For the input training samples, For strongly supervised samples, To monitor loss using anchor point regularization, To provide a new perspective on the loss corresponding to rendering supervision and geometric supervision, Provide supervision signals for rendering supervision and geometric supervision from a new perspective.

[0019] The present invention also provides a multi-view image 3D reconstruction device based on video data hybrid supervised fine-tuning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning.

[0020] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning.

[0021] Compared with the prior art, the beneficial effects of the present invention include at least the following: (1) Make full use of unlabeled video data: This method can effectively utilize large-scale unlabeled network videos for fine-tuning of 3D reconstruction models, breaking through the bottleneck of scarce labeled data. Compared with RealEstate10K's 20,000+ scenes, traditional labeled datasets such as ScanNet only have about 1,000 scenes.

[0022] (2) Significantly improved reconstruction performance: Consistent performance improvement was achieved on multiple standard benchmarks. Compared with the baseline MV-DUSt3R, this method reduced the Chamfer distance by approximately 6%, 20%, 29%, and 42% on ScanNet, 7Scenes, TUM-RGBD, and MP3D, respectively.

[0023] (3) Strong generalization ability: It performs well on zero-shot datasets, demonstrating that incorporating diverse video data can enhance the generalization ability to unseen environments.

[0024] (4) High computational efficiency: This method allows shallower pre-trained models to outperform state-of-the-art methods. Compared to VGGT, which requires 64 A100 GPUs to train for 9 days, this method only requires 4 RTX 4090 GPUs to train for 2 days to achieve better performance on multiple benchmarks.

[0025] (5) Improved pose estimation accuracy: Fine-tuning of video data not only improves the reconstruction quality, but also improves the accuracy of model pose estimation, resulting in more geometrically reliable output. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0027] Figure 1 This is a flowchart illustrating the multi-view image 3D reconstruction method based on video data hybrid supervision and fine-tuning provided by the present invention.

[0028] Figure 2 This invention provides a schematic diagram illustrating the principle of a multi-view image 3D reconstruction method based on video data hybrid supervision and fine-tuning.

[0029] Figure 3 The image shows the visualization results of reconstructed point clouds on multiple datasets provided for the example. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and given in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to illustrate selected embodiments of the invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0031] like Figure 1 As shown in the embodiment, a multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning includes the following steps: S1. Sample a training sample dataset containing multiple frames of images from unlabeled video data.

[0032] like Figure 2 As shown, the present invention aims to extract from a set of disordered RGB images A consistent 3D representation of the reconstructed scene, in which the camera's intrinsic parameters are used. and external parameter position All are unavailable. Multiview Figure 3 The 3D reconstruction network performs forward inference to predict pixel-aligned 3D point maps. Each pixel encodes the 3D coordinates of its observed visible scene point, and all 3D coordinates are represented in a specified reference image. In the coordinate system.

[0033] Furthermore, this invention requires extracting effective multi-view training samples from video sequences without accessing ground truth depth data or sufficiently dense point clouds. Therefore, in this embodiment, at fixed time intervals... Perform uniform sampling to obtain the initial frame sequence. This is to ensure a certain degree of overlap between views. , for The corresponding sampling frame, For sampling index, The sampling number is used; then, a random perturbation is applied to each sampling time point in the initial frame sequence. The resulting sequence of sampled frames with random perturbations is used as training samples. This perturbation sampling strategy assumes that the inherent temporal continuity of the video ensures sufficient scene overlap between frames. By introducing controllable randomness on top of regular sampling, this method promotes greater viewpoint diversity while preserving the temporal structure.

[0034] S2. Input the training sample dataset into the pre-trained multi-view... Figure 3 3D reconstruction network, the multi-view Figure 3 The 3D reconstruction network uses forward inference to output a 3D point cloud aligned with the training sample pixels, as well as 3D Gaussian parameters for new perspective synthesis.

[0035] In this embodiment, multi-view training is performed on multiple datasets with ground truth annotations. Figure 3 The multi-view reconstruction network MV-DUSt3R is used to obtain pre-trained multi-view reconstructions. Figure 3 Reconstruct the network. Details are as follows: Videos from RealEstate10K and DL3DV were used as the primary training source; these datasets contain a large number of diverse scenes from common real-life environments. All video clips were first preprocessed, and samples were randomly selected from the preprocessed set during each iteration of training. A total of over 30,000 video clips were used for training.

[0036] The training video dataset is preprocessed; specifically, RealEstate10K and DL3DV contain minimal dynamic content, making them ideal for fine-tuning pre-trained multi-view datasets. Figure 3 A 3D point cloud reconstruction network is constructed. Specifically, COLMAP is used to preprocess all video clips. For videos with publicly available camera poses, the sparse 3D point cloud and associated camera parameters for each frame are recovered using COLMAP with known poses.

[0037] Based on multi-view Figure 3 The MV-DUSt3R multi-view reconstruction network is used as the base model. MV-DUSt3R is a multi-view reconstruction network used to predict dense 3D point clouds from multiple input images. Training uses input images with a resolution of 224×224. Each training sample contains 8 input views, and 2 or 8 new views are rendered for supervision. Unlabeled video data from RealEstate10K and DL3DV are mixed during training. In addition, approximately 3% (about 700) of pre-trained strongly supervised training samples are included, making the ratio of video data to pre-trained supervised data approximately 30:1. To preserve the pre-trained 3D Gaussian representation, the Gaussian head is frozen, and multi-view optimization is performed only. Figure 3 Rebuild the network. Multi-view Figure 3 The multi-view reconstruction network was trained for 5 epochs, with 15,000 training trajectories sampled from each dataset in each epoch, resulting in a pre-trained multi-view network. Figure 3 Rebuild the network.

[0038] The pre-trained multi-view Figure 3 The 3D reconstruction network includes: encoder, decoder, regression head, and Gaussian parameter prediction head; The encoder is used to take the training sample dataset as input and output unordered multi-view visual representations (Tokens). The decoder is used to calculate the cross attention between views based on the disordered arrangement of multi-view visual representations through a cross-reference graph attention mechanism, in order to realize cross-view information exchange and aggregation, and output fused scene features; The regression head is used to regress and predict the three-dimensional spatial coordinates of each pixel in each view, taking the fused scene features as input, and generate a pixel-aligned three-dimensional point cloud. The Gaussian parameter prediction head, used in parallel with the regression head, takes fused scene features as input to predict 3D Gaussian parameters for each pixel in each view, for use in downstream new viewpoint synthesis tasks. , For Gaussian parameter prediction function, For the first One view, For the first Camera intrinsics for each view, Number of views For mapping operations, For pixel indexing; each 3D Gaussian parameter is centered Opacity Rotation ,scale spherical harmonic coefficient definition.

[0039] S3, based on multi-view Figure 3 The 3D point cloud and 3D Gaussian parameters output by the 3D reconstruction network are used in parallel for new perspective rendering supervision based on video consistency, geometric supervision based on external sparse 3D point clouds, and anchor point regularization supervision based on the original strongly supervised dataset. These are then used to compute the joint loss to optimize multi-view rendering. Figure 3 The 3D reconstruction network outputs a consistent 3D point cloud and renderable 3D Gaussian parameters to complete the 3D reconstruction.

[0040] This step proposes a hybrid supervised fine-tuning framework, aiming to expose the model to a wider range of scene diversity using unlabeled video data and to achieve supervision through an efficient training strategy. Figure 1 As shown, this framework combines three complementary supervision strategies to constrain the model and prevent drift, thereby achieving stable training and improving reconstruction quality: 1) Novel viewpoint rendering supervision based on video consistency: utilizing multi-view between sampled video frames Figure 1 Consistency; 2) Geometric supervision based on external sparse 3D point clouds: using sparse 3D geometric constraints from external reconstructed point clouds; 3) Anchor point regularization supervision from the original strongly supervised dataset: a small amount of strongly supervised data from the original pre-trained dataset. Details are as follows: A Novel Perspective on Video Consistency-Based Rendering Supervision: A Hybrid Supervised Fine-Tuning Framework Fully Utilizes Rich Novel Perspective Information in Video Data for Training. Given sampled video segments... (Include The time-ordered frames are divided into context frames (as input) and new perspective supervision target frames (as supervision labels for new perspective synthesis). To ensure effective supervision, sampling interpolation frames are selected. This serves as the target perspective to avoid artifacts that might be introduced in Gaussian sputtering reconstruction due to extrapolation perspective.

[0041] Rendering new perspective image frames from predicted point clouds using differentiable Gaussian sputtering. The rendering loss function during training is... , For the image frames in the training sample dataset, and These are the context frame and the new perspective supervised target frame, respectively. For new perspective image frames, In order to perceive loss, It is the L2 norm. To perceive loss weights, The weights are L2 norm. This loss function aims to encourage the model to predict globally consistent 3D reconstructions and generate realistic new perspectives.

[0042] Geometric Supervision Based on External Sparse 3D Point Clouds: While novel perspective supervision introduces a large amount of input scene data, its constraints alone are often insufficient and may limit the accuracy of reconstructed 3D geometry. Therefore, this invention introduces sparse 3D point clouds reconstructed by the Structure of Motion Reconstruction (COLMAP) method as additional sparse geometric constraints. Specifically, the sparse 3D point cloud is projected onto each image to generate a corresponding sparse depth map as depth supervision. Following the methods of DUSt3R and MV-DUSt3R, confidence-aware geometric alignment loss is calculated with the 3D point cloud in 3D space to provide geometric scale and positional constraints. Defined as: , , For view indexing, For the total number of views, For pixel index, For view Medium-strength supervised pixel set, For pixels The prediction confidence level For pixels The normalized geometric regression error, For the predicted three-dimensional coordinates, The three-dimensional coordinates of the sparse truth value. To predict the normalized scaling factor of 3D coordinates, The normalized scaling factor for the sparse true three-dimensional coordinates. is the regularization coefficient.

[0043] Anchor-Regulated Supervision Based on Original Strongly Supervised Datasets: Although the above supervision strategy effectively utilizes information from video data, in practice, it has been observed that this training scheme may lead to suboptimal convergence or model drift. Therefore, this invention incorporates strongly supervised samples from a small number of three-dimensional ground truth datasets. (Approximately 3% of the video training samples) are used as anchor points for regularization. This regularization helps constrain the model around the original optimum, allowing it to be further optimized without compromising reconstruction quality or introducing fine-tuning instability.

[0044] Based on the three supervision signals mentioned above, the optimal multi-view method is found. Figure 3 3D Reconstruction Network Parameters To minimize the joint loss function in order to compute the joint loss: , For the training sample dataset, For a three-dimensional truth dataset, For the input training samples, For strongly supervised samples, To monitor loss using anchor point regularization, To provide a new perspective on the loss corresponding to rendering supervision and geometric supervision, Provide supervision signals for rendering supervision and geometric supervision from a new perspective.

[0045] Based on optimal multi-view Figure 3 The 3D reconstruction network outputs a consistent 3D point cloud and a renderable Gaussian representation, thus completing the 3D reconstruction.

[0046] Based on the same inventive concept, the embodiment also provides a multi-view image 3D reconstruction device based on video data hybrid supervision fine-tuning, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the multi-view image 3D reconstruction method based on video data hybrid supervision fine-tuning.

[0047] In addition, the embodiment also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the multi-view image 3D reconstruction method based on video data hybrid supervision fine-tuning.

[0048] To verify the feasibility and effectiveness of the multi-view image 3D reconstruction method provided by this invention, validation was performed on multiple datasets. For example... Figure 3 As shown, the method of the present invention has wide application value, including but not limited to: Indoor Scene Reconstruction: This method reconstructs 3D indoor scenes from sparse perspective input, suitable for applications such as AR / VR and indoor navigation. Experiments show that this method achieves state-of-the-art performance on ScanNet, 7Scenes, TUM RGB-D (TUM), and Matterport3D (MP3D) indoor datasets.

[0049] Outdoor scene reconstruction: Although the training videos are mainly indoor scenes, this method also shows good generalization ability on the outdoor UASOL dataset because it includes some outdoor scenes.

[0050] Pose estimation: The method of this invention not only improves the quality of 3D reconstruction but also enhances the accuracy of camera pose estimation. Using the predicted point map, camera pose can be estimated through RANSAC and PnP algorithms.

[0051] Zero-shot generalization: This method demonstrates strong zero-shot generalization ability on datasets not seen during training, proving the value of utilizing diverse video data.

[0052] Finally, it should be noted that the above-described embodiments are merely specific implementations of the present invention, used to illustrate the technical solutions of the present invention, and not to limit it. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can still modify or easily conceive of changes to the technical solutions described in the foregoing embodiments within the technical scope disclosed in the present invention, or make equivalent substitutions for some of the technical features; and these modifications, changes, or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for multi-view image 3D reconstruction based on video data hybrid supervised fine-tuning, characterized in that, Includes the following steps: Step 1: Sample a training sample dataset containing multiple frames of images from unlabeled video data; Step 2: Input the training sample dataset into the pre-trained multi-view 3D reconstruction network. The multi-view 3D reconstruction network outputs a 3D point cloud aligned with the training sample pixels and 3D Gaussian parameters for new view synthesis using forward inference. Step 3: Based on the 3D point cloud and 3D Gaussian parameters output by the multi-view 3D reconstruction network, perform parallel supervision of new perspective rendering based on video consistency, geometric supervision based on external sparse 3D point cloud, and anchor point regularization supervision based on the original strongly supervised dataset. These are used to calculate the joint loss to optimize the multi-view 3D reconstruction network, outputting a consistent 3D point cloud and renderable 3D Gaussian parameters, thus completing the 3D reconstruction.

2. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 1, characterized in that, Step 1 specifically includes: The unlabeled video data in the training sample dataset is uniformly sampled at fixed time intervals to obtain the initial frame sequence; A random perturbation is applied to each sampling time point in the initial frame sequence to obtain a sampling frame sequence with random perturbation as training samples.

3. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 2, characterized in that, In step 2, the pre-trained multi-view 3D reconstruction network, trained on the original dataset, includes: an encoder, a decoder, a regression head, and a Gaussian parameter prediction head; The encoder is used to take training samples as input and output disordered multi-view visual representations. The decoder is used to calculate the cross attention between views based on the disordered arrangement of multi-view visual representations through a cross-reference graph attention mechanism, in order to realize cross-view information exchange and aggregation, and output fused scene features; The regression head is used to regress and predict the three-dimensional spatial coordinates of each pixel in each view, taking the fused scene features as input, and generate a pixel-aligned three-dimensional point cloud. The Gaussian parameter prediction head is used in parallel with the regression head, taking fused scene features as input, to predict 3D Gaussian parameters for each pixel of each view, for use in downstream new perspective synthesis tasks.

4. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 3, characterized in that, The three-dimensional Gaussian parameters include at least: center position, opacity, rotation, scale, and appearance spherical harmonic coefficients.

5. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 1, characterized in that, In step 3, the new perspective rendering supervision based on video consistency includes: using the temporal relationship between image frames in the training samples, taking some frames as context input and other frames as new perspective supervision target frames, synthesizing Gaussian parameters into a new perspective image through differentiable rendering, and calculating the rendering loss between the new perspective image frame and the real target image frame to constrain video consistency. Among them, rendering loss , For the image frames in the training sample dataset, and These are the context frame and the new perspective supervised target frame, respectively. For new perspective image frames, In order to perceive loss, It is the L2 norm. To perceive loss weights, These are L2 norm weights.

6. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 3, characterized in that, Step 3, the geometric supervision based on external sparse 3D point clouds, includes: Sparse 3D point clouds are reconstructed from video segments corresponding to image frames in the training sample dataset using the structure-of-motion method. The sparse 3D point cloud is projected onto the image plane to generate sparse depth supervision, and confidence-aware geometric alignment loss is calculated between the 3D point cloud and the image plane in 3D space to provide geometric scale and positional constraints. Among them, geometric alignment loss , , For view indexing, For the total number of views, For pixel index, For view Medium-strength supervised pixel set, For pixels The prediction confidence level For pixels The normalized geometric regression error, For the predicted three-dimensional coordinates, The three-dimensional coordinates of the sparse truth value. To predict the normalized scaling factor of 3D coordinates, The normalized scaling factor for the sparse true three-dimensional coordinates. This is the regularization coefficient.

7. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 2, characterized in that, In step 3, the anchor point regularization supervision based on the original strongly supervised dataset includes: during the training process, strongly supervised samples with three-dimensional truth labels in the mixed three-dimensional truth dataset are used as anchor point data for regularization.

8. The multi-view image 3D reconstruction method based on video data hybrid supervised fine-tuning according to claim 7, characterized in that, By finding the optimal network parameters for multi-view 3D reconstruction To minimize the joint loss function in order to compute the joint loss: , For the training sample dataset, For a three-dimensional truth dataset, For the input training samples, For strongly supervised samples, To monitor loss using anchor point regularization, To provide a new perspective on the loss corresponding to rendering supervision and geometric supervision, Provide supervision signals for rendering supervision and geometric supervision from a new perspective.

9. A multi-view image 3D reconstruction device based on video data hybrid supervised fine-tuning, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the multi-view image 3D reconstruction method based on video data hybrid supervision fine-tuning as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the multi-view image 3D reconstruction method based on video data hybrid supervision fine-tuning as described in any one of claims 1-8.

Images