Sparse visual angle three-dimensional reconstruction method and device, equipment, storage medium and program product

Abstract

The invention relates to a sparse view angle three-dimensional reconstruction method and device, computer equipment, a computer readable storage medium and a computer program product. Comprising the following steps: extracting two-dimensional features from a sparse view angle image; pixel and voxel matching is carried out based on the two-dimensional features, and optimized features are obtained; processing the optimization features corresponding to the sliding windows of different scales through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; extracting global features of the scene to be reconstructed from the optimized features; carrying out fusion processing on the layered local features and the global features to obtain fusion features; and performing three-dimensional reconstruction according to the fusion features to obtain three-dimensional model data of the scene to be reconstructed. By adopting the method, geometric structure information of two-dimensional features can be corrected, so that the problem that a deep learning algorithm is insensitive to geometric constraints is solved. Local features and global features of a to-be-reconstructed scene and the relation between the two features are captured, so that the detail representation of a reconstruction structure is enhanced, and the reconstruction quality of three-dimensional model data is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a sparse perspective 3D reconstruction method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Technology

[0002] Traditional 3D reconstruction methods require a sufficiently high density of 2D viewpoints for reconstruction, making them ineffective for sparse viewpoint reconstruction tasks. Deep learning algorithms can utilize pre-trained large visual models to compensate for the information loss caused by sparse viewpoints, but they cannot effectively capture these geometric constraints in scenarios with strong constraints.

[0003] Currently, deep learning-based 3D reconstruction methods often only focus on the global features of the scene while ignoring local features. This makes the reconstructed scene, especially for some large-scale scenes with large instance differences, very blurry in terms of detail. Summary of the Invention

[0004] Therefore, it is necessary to provide a sparse perspective 3D reconstruction method, apparatus, computer equipment, computer-readable storage medium, and computer program product to address the above-mentioned technical problems. This method can correct the geometric structure information of 2D features, enhance the detail representation of the reconstructed structure, and improve the reconstruction quality of 3D model data.

[0005] Firstly, this application provides a sparse viewpoint 3D reconstruction method, including:

[0006] Extracting two-dimensional features from sparse viewpoint images;

[0007] Pixel-voxel matching is performed based on two-dimensional features to obtain optimized features;

[0008] By processing the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism, the hierarchical local features of the scene to be reconstructed are obtained.

[0009] Extract global features of the scene to be reconstructed from the optimized features;

[0010] The hierarchical local and global features are fused to obtain the fused features;

[0011] Based on the fusion features, 3D reconstruction is performed to obtain 3D model data of the scene to be reconstructed.

[0012] In one embodiment, the method is performed by a trained 3D reconstruction model, which includes an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module.

[0013] In one embodiment, the pixel-to-voxel matching module employs a cross-attention mechanism. During the training phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the sampled two-dimensional features and voxelized structures based on the cross-attention mechanism to obtain optimized features. During the inference phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the two-dimensional features of the image to be reconstructed based on a self-attention mechanism to obtain optimized features.

[0014] In one embodiment, the method further includes:

[0015] Sample two-dimensional sample images and the corresponding voxelized structures from a three-dimensional model;

[0016] The three-dimensional reconstruction model is trained using two-dimensional sample images and their corresponding voxelized structures to obtain a trained three-dimensional reconstruction model.

[0017] In one embodiment, before training the 3D reconstruction model using the 2D sample image and its corresponding voxelized structure, the method further includes:

[0018] An image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module are constructed to obtain a 3D reconstruction model.

[0019] In one embodiment, an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module are constructed, including:

[0020] An image encoder module is constructed using a pre-trained visual encoder;

[0021] A pixel-to-voxel matching module is constructed using a cross-attention mechanism;

[0022] A hierarchical attention module is constructed by combining attention mechanisms of different dimensions;

[0023] A self-attention module is constructed using a multi-head self-attention mechanism;

[0024] A feature fusion module is constructed using a pre-trained deep residual network model;

[0025] A 3D reconstruction module is constructed using a pre-trained 3D decoder or a pre-trained generative model.

[0026] Secondly, this application also provides a sparse viewpoint 3D reconstruction device, comprising:

[0027] An image encoder module is used to extract two-dimensional features from sparse viewpoint images;

[0028] The pixel-to-voxel matching module is used to perform pixel-to-voxel matching based on two-dimensional features to obtain optimized features.

[0029] The hierarchical attention module is used to process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain the hierarchical local features of the scene to be reconstructed.

[0030] The self-attention module is used to extract global features of the scene to be reconstructed from the optimized features;

[0031] The feature fusion module is used to fuse hierarchical local features and global features to obtain fused features;

[0032] The 3D reconstruction module is used to perform 3D reconstruction based on fusion features to obtain 3D model data of the scene to be reconstructed.

[0033] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in the first aspect above.

[0034] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in the first aspect above.

[0035] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method described in the first aspect above.

[0036] The aforementioned sparse-view 3D reconstruction method, apparatus, computer equipment, computer-readable storage medium, and computer program product extract two-dimensional features from sparse-view images; perform pixel-to-voxel matching based on the two-dimensional features to obtain optimized features; process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; extract global features of the scene to be reconstructed from the optimized features; fuse the hierarchical local features and global features to obtain fused features; and perform 3D reconstruction based on the fused features to obtain 3D model data of the scene to be reconstructed. Through this method, during the 3D reconstruction process, the relationship between two-dimensional features and voxelized structures is learned based on the pixel-to-voxel matching mechanism, which can correct the geometric structural information of the two-dimensional features, thereby solving the problem of deep learning algorithms being insensitive to geometric constraints. By extracting and fusing hierarchical local features and global features, the local features, global features, and relationships between the two types of features of the scene to be reconstructed can be captured, thereby enhancing the detailed representation of the reconstructed structure, providing rich and accurate feature information for 3D reconstruction, and improving the reconstruction quality of the 3D model data. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart illustrating a sparse viewpoint 3D reconstruction method in one embodiment;

[0039] Figure 2 This is a schematic diagram of the processing flow of a 3D reconstruction model in one embodiment;

[0040] Figure 3 This is a flowchart illustrating the sparse viewpoint 3D reconstruction method in another embodiment;

[0041] Figure 4 This is a structural block diagram of a sparse viewpoint 3D reconstruction device in one embodiment;

[0042] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0044] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0045] In one exemplary embodiment, such as Figure 1 As shown, a sparse viewpoint 3D reconstruction method is provided. This embodiment illustrates the method applied to a server. It is understood that this method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0046] Step 102: Extract two-dimensional features from sparse viewpoint images.

[0047] Sparse viewpoint images can be understood as a small set of images with large intervals between shooting angles. Optionally, deep learning algorithms can be used to extract two-dimensional features from sparse viewpoint images to address the failure of traditional reconstruction algorithms under sparse viewpoints. For example, sparse viewpoint images are input into an image encoder module to extract the corresponding two-dimensional features. The image encoder module can be constructed from a pre-trained visual encoder, such as the DINO model or the CLIP (Contrastive Language-Image Pre-training) model.

[0048] Step 104: Perform pixel-to-voxel matching based on two-dimensional features to obtain optimized features.

[0049] Specifically, for the current 2D features, the relationship between the pre-learned 2D features and the voxelized structure is utilized to optimize the 2D features, resulting in optimized features. Optionally, a pixel-to-voxel matching module is used to input the 2D features, learn the relationship between the voxelized structure of the 3D model and its corresponding sparse 2D viewpoint, thereby optimizing the 2D features corresponding to the sparse viewpoint image and obtaining optimized features. For example, the pixel-to-voxel matching module is composed of a cross-attention mechanism.

[0050] Step 106: Process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain the hierarchical local features of the scene to be reconstructed.

[0051] Here, hierarchical local features refer to local feature information at different levels. Optionally, a hierarchical attention module is used to process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain local feature information at different levels in the scene to be reconstructed, i.e., hierarchical local features. For example, the hierarchical attention module consists of sliding windows of different sizes and their corresponding self-attention mechanisms.

[0052] Step 108: Extract global features of the scene to be reconstructed from the optimized features.

[0053] Optionally, a self-attention module can be used to extract global features of the scene to be reconstructed from the optimized features for subsequent 3D scene reconstruction. For example, the self-attention module may consist of a self-attention mechanism without linear masks or a multi-head self-attention mechanism.

[0054] Step 110: The hierarchical local features and global features are fused to obtain the fused features.

[0055] Specifically, the hierarchical local features extracted in step 106 and the global features extracted in step 108 are fused to combine the detailed information of the scene to be reconstructed with global information, thereby enhancing the detailed representation of the reconstructed scene. Optionally, the hierarchical local features and global features are input into the feature fusion module and finally fused to obtain the fused features. The feature fusion module can be a pre-trained deep residual network model, such as one constructed from network structures like VGG-16, ResNet50, or cross-attention mechanisms.

[0056] Step 112: Perform 3D reconstruction based on the fusion features to obtain the 3D model data of the scene to be reconstructed.

[0057] Optionally, a 3D reconstruction module is used to recover the 3D scene from the features obtained by fusing hierarchical local and global features, obtaining the 3D model data of the scene to be reconstructed, and restoring the fused features into a visual 3D model. The 3D reconstruction module can be composed of a pre-trained 3D decoder, which ultimately outputs different types of 3D representations through multiple heads (the output part of the model). This 3D reconstruction module is built from a pre-trained 3D decoder or a pre-trained generative model, such as a VAE (Variational Auto Encoder), a 3D diffusion model, etc.

[0058] For example, applying the method provided in this application to the reconstruction of large scenes under sparse perspectives in digital twin technology can reconstruct the details in large scenes. For instance, in the field of cultural heritage protection, for some large ancient building sites, it is difficult to obtain enough image data due to perspective limitations. The method provided in this application can reconstruct the three-dimensional model of the ancient building from sparse perspective images, providing more accurate three-dimensional data for the digital protection and research of cultural heritage.

[0059] For example, the method provided in this application embodiment can be used to recover 3D models of small objects from sparse viewpoint images, which can then be used as components of a game scene. For instance, if a game containing various complex buildings and natural landscapes needs to be developed, the method provided in this application embodiment can generate high-quality 3D scene components from a small number of reference images, improving game development efficiency.

[0060] In the aforementioned sparse-view 3D reconstruction method, two-dimensional features are extracted from the sparse-view image; pixel-to-voxel matching is performed based on the two-dimensional features to obtain optimized features; the optimized features corresponding to sliding windows of different sizes are processed through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; global features of the scene to be reconstructed are extracted from the optimized features; the hierarchical local features and global features are fused to obtain fused features; and 3D reconstruction is performed based on the fused features to obtain the 3D model data of the scene to be reconstructed. Through this method, in the 3D reconstruction process, the relationship between two-dimensional features and voxelized structures is learned based on the pixel-to-voxel matching mechanism, which can correct the geometric structural information of the two-dimensional features, thereby solving the problem of deep learning algorithms being insensitive to geometric constraints. By extracting and fusing hierarchical local features and global features, the local features, global features, and relationships between the two types of features of the scene to be reconstructed can be captured, thereby enhancing the detailed representation of the reconstructed structure, providing rich and accurate feature information for 3D reconstruction, and improving the reconstruction quality of the 3D model data.

[0061] In one exemplary embodiment, the method is performed by a trained 3D reconstruction model, which includes an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module.

[0062] The image encoder module takes sparse 2D viewpoint images as input and outputs 2D features corresponding to the sparse viewpoint images, used to perform step 102. The pixel and voxel matching module takes 2D features as input and outputs optimized features, used to perform step 104. The hierarchical attention module takes optimized features as input and outputs hierarchical local features of the scene to be reconstructed, used to perform step 106. The self-attention module takes optimized features as input and outputs global features of the scene to be reconstructed, used to perform step 108. The feature fusion module takes hierarchical local features and global features as input and outputs fused features, used to perform step 110. The 3D reconstruction module takes fused features as input and outputs 3D model data, used to perform step 112. The hierarchical attention module and the self-attention module constitute a dual-branch attention mechanism.

[0063] The 3D reconstruction model provided in this application includes an image encoder module, a pixel-to-voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module. It corrects the geometric information in 2D image features using voxel structure information and enhances the details of the reconstructed scene through a dual-attention branch. Combining the advantages of deep learning algorithms and traditional 3D reconstruction algorithms, a pixel-to-voxel matching module is proposed. By learning the relationship between 2D features and voxelized structures, it corrects the geometric structure information of 2D features and uses it for subsequent 3D reconstruction. A dual-attention fusion mechanism is proposed. The hierarchical attention module extracts detailed features of the 3D scene, the self-attention mechanism extracts global features of the 3D scene, and the feature fusion module connects the global and detailed features, ultimately enhancing the geometric structure and detail representation of the reconstructed 3D scene. A 3D reconstruction mechanism under a sparse perspective is proposed, using voxel structure information guidance and a dual-branch attention structure to enhance the deep learning algorithm's capture of scene geometry. This mechanism can be applied to 3D scene reconstruction, generation of editable 3D scenes, digital twins, and other fields.

[0064] In an exemplary embodiment, the pixel-to-voxel matching module employs a cross-attention mechanism. During the training phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the sampled two-dimensional features and voxelized structures based on the cross-attention mechanism to obtain optimized features. During the inference phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the two-dimensional features of the image to be reconstructed based on a self-attention mechanism to obtain optimized features.

[0065] The pixel-to-voxel matching module learns the relationship between the voxelized structure of the 3D model and its corresponding sparse 2D viewpoint during training, and optimizes the 2D features corresponding to the sparse viewpoint image during inference to obtain optimized features. In its implementation, the pixel-to-voxel matching module uses a cross-attention mechanism to establish the connection between 2D features and the voxelized structure. During training, the module uses the cross-attention mechanism, treating the voxelized structure as query data and the 2D features as key and value data, learning the differences between the 2D features and the voxelized structure to achieve geometric information correction. During inference, the module uses a self-attention mechanism, simultaneously treating the 2D features as query data, key data, and value data. During training, the input to the pixel-to-voxel matching module is the 2D features and the voxelized structure; the output is the optimized features. During inference, the input to the pixel-to-voxel matching module is the 2D features; the output is the optimized features.

[0066] Understandably, in the field of 3D reconstruction, traditional deep learning algorithms are insufficient in capturing scenes with strong geometric constraints. In this embodiment, the pixel-to-voxel matching module learns the relationship between 2D features and voxelized structures, thereby correcting the geometric structure information of the 2D features. Utilizing a cross-attention mechanism, the voxelized structure is used as query data and the 2D features as key and value data during the training phase, thus guiding the deep learning algorithm to learn the geometric constraints of the 3D scene. Through this innovative matching method, even under sparse perspectives, the geometric constraints in the reconstructed scene can be effectively optimized, compensating for the shortcomings of traditional deep learning algorithms in capturing geometric constraints.

[0067] In an exemplary embodiment, the method further includes: sampling two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images from the three-dimensional model; and training the three-dimensional reconstruction model using the two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images to obtain a trained three-dimensional reconstruction model.

[0068] Among them, reference Figure 2 For a given 3D model, based on multiple sparse 2D viewpoints, images of the 3D model corresponding to these 2D viewpoints are sampled from the 3D model. Through simplification operations such as meshing, a simplified voxelized structure of the 3D model is obtained.

[0069] Reference Figure 2 During training, sampled 2D images are input into the image encoder module to obtain 2D features extracted by the image encoder module. These 2D features and voxelized structures are then input into the pixel-to-voxel matching module, which outputs optimized features. The optimized features are then input into the hierarchical attention module and the self-attention module. The hierarchical attention module processes the optimized features corresponding to sliding windows of different sizes using a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed. The self-attention module extracts global features of the scene to be reconstructed from the optimized features. The feature fusion module fuses the hierarchical local features and global features to obtain fused features. The 3D reconstruction module performs 3D reconstruction based on the fused features to obtain 3D model data. Based on the differences between the 3D model and the 3D model data, the model parameters of the 3D reconstruction model are adjusted until the training parameters meet preset conditions, such as reaching the maximum number of iterations or the loss function value being less than a preset value, ultimately resulting in a trained 3D reconstruction model.

[0070] In an exemplary embodiment, before training the 3D reconstruction model using 2D sample images and their corresponding voxelized structures, the method further includes: constructing an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module to obtain the 3D reconstruction model.

[0071] In one alternative implementation, for each module, a model type is selected, a model architecture is defined, a loss function and an optimizer are selected based on a preset script tool, and an evaluation metric is defined. Based on the selected model type, model architecture, loss function, optimizer and evaluation metric, the model is initialized and compiled, and the constructed multiple modules are combined to obtain a 3D reconstruction model.

[0072] In an exemplary embodiment, constructing an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module includes: constructing an image encoder module using a pre-trained visual encoder; constructing a pixel and voxel matching module using a cross-attention mechanism; constructing a hierarchical attention module using a combination of attention mechanisms of different dimensions; constructing a self-attention module using a multi-head self-attention mechanism; constructing a feature fusion module using a pre-trained deep residual network model; and constructing a 3D reconstruction module using a pre-trained 3D decoder or a pre-trained generative model.

[0073] The system comprises the following modules: an image encoder module (e.g., DINO or CLIP model) is constructed using a pre-trained visual encoder; a pixel-to-voxel matching module is constructed using a learnable cross-attention mechanism; a hierarchical attention module is constructed using a combination of attention mechanisms of different dimensions; a self-attention module is constructed using a multi-head self-attention mechanism; a feature fusion module is constructed using a pre-trained deep residual network model, such as ResNet50; and a 3D reconstruction module is constructed using a pre-trained 3D decoder or a pre-trained generative model, such as a VAE model or a 3D diffusion model.

[0074] In one exemplary embodiment, refer to Figure 3 The sparse viewpoint 3D reconstruction method includes the following steps:

[0075] 1. Data preprocessing and module building stage:

[0076] S1-1: Sample two-dimensional perspectives and voxelized structures from the 3D model. Specifically, based on multiple sparse two-dimensional perspectives, samples are taken from the 3D model to obtain images of the 3D model corresponding to these two-dimensional perspectives; through simplification operations such as meshing, the simplified voxelized structure of the 3D model is obtained.

[0077] S1-2: Construct the image encoder module, pixel and voxel matching module, hierarchical attention module, self-attention module, feature fusion module, and 3D reconstruction module.

[0078] For example, an image encoder module is constructed using a pre-trained DINO model; a pixel and voxel matching module is constructed using a learnable cross-attention mechanism; a hierarchical attention module is constructed using a combination of attention mechanisms of different dimensions; a self-attention module is constructed using a multi-head self-attention mechanism; a feature fusion module is constructed using a pre-trained ResNet50 model; and a 3D reconstruction module is constructed using a pre-trained 3D diffusion model.

[0079] 2. Training Phase:

[0080] S2-1: The image encoder module processes the two-dimensional viewpoint and obtains two-dimensional features.

[0081] In this process, a 2D image from a sparse perspective is input into an image encoder module. The image encoder then encodes the 2D features corresponding to the 2D perspective image, which are used for subsequent 3D reconstruction.

[0082] S2-2: The pixel and voxel matching module learns the differences between two-dimensional features and voxelized structures, corrects the geometric information in the two-dimensional features, and obtains optimized features.

[0083] Specifically, through the cross-attention mechanism, the differences between two-dimensional features and voxelized structures are learned. By learning these differences, the geometric information in the two-dimensional features obtained by the image encoder is corrected to make them closer to the geometric constraints in the real three-dimensional scene, thus obtaining optimized features.

[0084] S2-3: The hierarchical attention module extracts hierarchical local features from the optimized features; the self-attention module extracts global features from the optimized features.

[0085] The hierarchical attention module extracts local features of different vision scales from the optimized features through attention mechanisms of different dimensions. The self-attention module directly extracts global features from the optimized features.

[0086] S2-4: The feature fusion module fuses hierarchical local features and global features.

[0087] The feature fusion module uses the same set of pre-trained ResNet50 to further encode the hierarchical local features and global features respectively, and adds the two sets of encoded features in the corresponding dimensions to achieve the fusion of local features and global features.

[0088] S2-5: The 3D reconstruction module recovers the 3D reconstruction model from the result of fusing hierarchical local features and global features.

[0089] The 3D reconstruction module uses a pre-trained 3D diffusion model and a Markov chain to gradually restore the invisible fused features into a visible 3D model.

[0090] 3. Reasoning stage:

[0091] S3-1: The pixel and voxel matching module performs geometric information correction on the input two-dimensional features to obtain optimized features.

[0092] During the inference phase, there is only a sparse two-dimensional viewpoint without a corresponding three-dimensional model and its voxelized structure. Therefore, during the inference phase, the pixel and voxel matching module uses the parameters learned during the training phase to perform self-attention processing on the two-dimensional features corresponding to the two-dimensional viewpoint, rather than cross-attention processing.

[0093] Using the sparse perspective 3D reconstruction method described above, the following can be achieved:

[0094] 1. Geometric Correction Based on Pixel-Voxel Matching Module. In the field of 3D reconstruction, traditional deep learning algorithms are insufficient in capturing scenes with strong geometric constraints. The pixel-voxel matching module, by learning the relationship between 2D features and voxelized structures, can correct the geometric structural information of 2D features. Utilizing a cross-attention mechanism, voxelized structures are used as query data and 2D features as key and value data during the training phase, thereby guiding the deep learning algorithm to learn the geometric constraints of the 3D scene. Through this innovative matching method, even under sparse perspectives, the geometric constraints in the reconstructed scene can be effectively optimized, making up for the shortcomings of traditional deep learning algorithms in capturing geometric constraints.

[0095] 2. Extracting Features from Different Scale Viewpoints Based on a Dual-Attention Weighted Mechanism. Traditional deep learning-based 3D reconstruction methods often focus only on global features of the scene while ignoring local features, resulting in blurred details in the reconstructed scene. This application's embodiments design a hierarchical attention module and a self-attention module. The hierarchical attention module extracts detailed features of the 3D scene through sliding windows of different scales and their corresponding self-attention mechanisms; the self-attention module extracts global features. Furthermore, a feature fusion module connects global and detailed features, ultimately enhancing the geometric structure and detail representation of the reconstructed 3D scene. This dual-branch attention structure can simultaneously consider both global and local features, significantly improving the quality of 3D reconstruction in terms of detail.

[0096] 3. A sparse-view 3D reconstruction mechanism based on a pre-trained large-scale model. This application's embodiments alleviate the failure problem of traditional 3D algorithms caused by sparse viewpoints by using a pre-trained large-scale visual model. In the case of sparse viewpoints, traditional 3D reconstruction methods require a sufficiently high density of 2D viewpoints used for reconstruction. This application combines the advantages of deep learning algorithms and traditional 3D reconstruction algorithms, utilizing a pre-trained model to compensate for the information loss caused by sparse viewpoints, and strengthening the deep learning algorithm's ability to capture scene geometry through voxel structure information guidance and a bi-branch attention structure. This mechanism provides a new solution for 3D reconstruction in the special scenario of sparse viewpoints, expanding the application scope of 3D reconstruction technology.

[0097] For example, taking the reconstruction of a large scene from a sparse perspective in digital twin technology as an example, the requirement for 3D reconstruction is to construct a digital twin model corresponding to a large industrial park. Due to the large area of ​​the industrial park, image data is acquired from a limited number of perspectives (sparse perspectives) for 3D reconstruction. Traditional 3D reconstruction methods struggle to accurately reconstruct the entire 3D structure of the park under such sparse perspective conditions, especially details such as various buildings, roads, and green spaces. The method provided in this application can utilize its pixel-to-voxel matching module and dual-attention weighting mechanism, guided by a pre-trained large visual model and voxel structure information, to recover the 3D model of the industrial park from these sparse perspective images. It can accurately reconstruct the shape, size, and relative position of different buildings within the park, as well as details such as road directions and the distribution of green areas, providing strong support for the application of digital twin technology in large-scale scenes.

[0098] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0099] Based on the same inventive concept, this application also provides a sparse-view 3D reconstruction apparatus for implementing the sparse-view 3D reconstruction method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the sparse-view 3D reconstruction apparatus provided below can be found in the limitations of the sparse-view 3D reconstruction method described above, and will not be repeated here.

[0100] In one exemplary embodiment, such as Figure 4 As shown, a sparse viewpoint 3D reconstruction device is provided, comprising:

[0101] Image encoder module 402 is used to extract two-dimensional features from sparse viewpoint images;

[0102] The pixel-to-voxel matching module 404 is used to perform pixel-to-voxel matching based on two-dimensional features to obtain optimized features;

[0103] The hierarchical attention module 406 is used to process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain the hierarchical local features of the scene to be reconstructed.

[0104] Self-attention module 408 is used to extract global features of the scene to be reconstructed from the optimized features;

[0105] The feature fusion module 410 is used to fuse hierarchical local features and global features to obtain fused features;

[0106] The 3D reconstruction module 412 is used to perform 3D reconstruction based on fusion features to obtain 3D model data of the scene to be reconstructed.

[0107] In the aforementioned sparse-view 3D reconstruction device, two-dimensional features are extracted from the sparse-view image; pixel-to-voxel matching is performed based on the two-dimensional features to obtain optimized features; the optimized features corresponding to sliding windows of different sizes are processed through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; global features of the scene to be reconstructed are extracted from the optimized features; the hierarchical local features and global features are fused to obtain fused features; and 3D reconstruction is performed based on the fused features to obtain the 3D model data of the scene to be reconstructed. Through this method, during the 3D reconstruction process, the connection between two-dimensional features and voxelized structures is learned based on the pixel-to-voxel matching mechanism, which can correct the geometric structural information of the two-dimensional features, thereby solving the problem of deep learning algorithms being insensitive to geometric constraints. By extracting and fusing hierarchical local features and global features, the local features, global features, and the connection between the two types of features of the scene to be reconstructed can be captured, thereby enhancing the detailed representation of the reconstructed structure, providing rich and accurate feature information for 3D reconstruction, and improving the reconstruction quality of the 3D model data.

[0108] In an exemplary embodiment, the pixel-to-voxel matching module 404 employs a cross-attention mechanism; wherein, during the training phase, the pixel-to-voxel matching module 404 performs pixel-to-voxel matching on the sampled two-dimensional features and voxelized structures based on the cross-attention mechanism to obtain optimized features; during the inference phase, the pixel-to-voxel matching module 404 performs pixel-to-voxel matching on the two-dimensional features of the image to be reconstructed based on a self-attention mechanism to obtain optimized features.

[0109] In an exemplary embodiment, the sparse viewpoint 3D reconstruction device further includes a training module; the training module is used to sample two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images from the 3D model; and to train the 3D reconstruction model using the two-dimensional sample images and the voxelized structures corresponding to the two-dimensional sample images to obtain a trained 3D reconstruction model; the 3D reconstruction model includes an image encoder module 402, a pixel and voxel matching module 404, a hierarchical attention module 406, a self-attention module 408, a feature fusion module 410, and a 3D reconstruction module 412.

[0110] In an exemplary embodiment, the sparse viewpoint 3D reconstruction device further includes a model building module for building an image encoder module 402, a pixel and voxel matching module 404, a hierarchical attention module 406, a self-attention module 408, a feature fusion module 410, and a 3D reconstruction module 412 to obtain a 3D reconstruction model.

[0111] In an exemplary embodiment, the model building module is further configured to: build an image encoder module 402 using a pre-trained visual encoder; build a pixel and voxel matching module 404 using a cross-attention mechanism; build a hierarchical attention module 406 using a combination of attention mechanisms of different dimensions; build a self-attention module 408 using a multi-head self-attention mechanism; build a feature fusion module 410 using a pre-trained deep residual network model; and build a 3D reconstruction module 412 using a pre-trained 3D decoder or a pre-trained generative model.

[0112] Each module in the aforementioned sparse-view 3D reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0113] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores model data, training data, and inference data for the 3D reconstruction model. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When executed by the processor, the computer program implements a sparse-view 3D reconstruction method.

[0114] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0115] In one exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: extracting two-dimensional features from a sparse viewpoint image; performing pixel-to-voxel matching based on the two-dimensional features to obtain optimized features; processing the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; extracting global features of the scene to be reconstructed from the optimized features; fusing the hierarchical local features and global features to obtain fused features; and performing three-dimensional reconstruction based on the fused features to obtain three-dimensional model data of the scene to be reconstructed.

[0116] In one embodiment, when the processor executes the computer program, it further performs the following steps: sampling two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images from the three-dimensional model; and training the three-dimensional reconstruction model using the two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images to obtain a trained three-dimensional reconstruction model.

[0117] In one embodiment, when the processor executes the computer program, it further implements the following steps: constructing an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module to obtain a 3D reconstruction model.

[0118] In one embodiment, when the processor executes the computer program, it further implements the following steps: constructing an image encoder module using a pre-trained visual encoder; constructing a pixel and voxel matching module using a cross-attention mechanism; constructing a hierarchical attention module using a combination of attention mechanisms of different dimensions; constructing a self-attention module using a multi-head self-attention mechanism; constructing a feature fusion module using a pre-trained deep residual network model; and constructing a 3D reconstruction module using a pre-trained 3D decoder or a pre-trained generative model.

[0119] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon. When executed by a processor, the computer program performs the following steps: extracting two-dimensional features from a sparse viewpoint image; performing pixel-to-voxel matching based on the two-dimensional features to obtain optimized features; processing the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; extracting global features of the scene to be reconstructed from the optimized features; fusing the hierarchical local features and global features to obtain fused features; and performing three-dimensional reconstruction based on the fused features to obtain three-dimensional model data of the scene to be reconstructed.

[0120] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sampling two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images from the three-dimensional model; and training the three-dimensional reconstruction model using the two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images to obtain a trained three-dimensional reconstruction model.

[0121] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: constructing an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module to obtain a 3D reconstruction model.

[0122] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: constructing an image encoder module using a pre-trained visual encoder; constructing a pixel and voxel matching module using a cross-attention mechanism; constructing a hierarchical attention module using a combination of attention mechanisms of different dimensions; constructing a self-attention module using a multi-head self-attention mechanism; constructing a feature fusion module using a pre-trained deep residual network model; and constructing a 3D reconstruction module using a pre-trained 3D decoder or a pre-trained generative model.

[0123] In one embodiment, when the computer program is executed by a processor, it further performs the following steps: In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: extracting two-dimensional features from a sparse viewpoint image; performing pixel-to-voxel matching based on the two-dimensional features to obtain optimized features; processing the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain hierarchical local features of the scene to be reconstructed; extracting global features of the scene to be reconstructed from the optimized features; fusing the hierarchical local features and global features to obtain fused features; and performing three-dimensional reconstruction based on the fused features to obtain three-dimensional model data of the scene to be reconstructed.

[0124] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: sampling two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images from the three-dimensional model; and training the three-dimensional reconstruction model using the two-dimensional sample images and voxelized structures corresponding to the two-dimensional sample images to obtain a trained three-dimensional reconstruction model.

[0125] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: constructing an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module to obtain a 3D reconstruction model.

[0126] In one embodiment, when the computer program is executed by the processor, it further implements the following steps: constructing an image encoder module using a pre-trained visual encoder; constructing a pixel and voxel matching module using a cross-attention mechanism; constructing a hierarchical attention module using a combination of attention mechanisms of different dimensions; constructing a self-attention module using a multi-head self-attention mechanism; constructing a feature fusion module using a pre-trained deep residual network model; and constructing a 3D reconstruction module using a pre-trained 3D decoder or a pre-trained generative model.

[0127] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0128] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0129] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0130] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A sparse viewpoint 3D reconstruction method, characterized in that, The method includes: Extracting two-dimensional features from sparse viewpoint images; Pixel-voxel matching is performed based on the two-dimensional features to obtain optimized features; By processing the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism, the hierarchical local features of the scene to be reconstructed are obtained. Extract global features of the scene to be reconstructed from the optimized features; The hierarchical local features and the global features are fused to obtain fused features; Based on the fusion features, 3D reconstruction is performed to obtain the 3D model data of the scene to be reconstructed.

2. The method according to claim 1, characterized in that, The method is executed by a trained 3D reconstruction model, which includes an image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module.

3. The method according to claim 2, characterized in that, The pixel-to-voxel matching module employs a cross-attention mechanism. During the training phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the sampled two-dimensional features and voxelized structures based on the cross-attention mechanism to obtain optimized features. During the inference phase, the pixel-to-voxel matching module performs pixel-to-voxel matching on the two-dimensional features of the image to be reconstructed based on a self-attention mechanism to obtain optimized features.

4. The method according to claim 3, characterized in that, The method further includes: Sample two-dimensional sample images and the corresponding voxelized structures from a three-dimensional model; The three-dimensional reconstruction model is trained using the two-dimensional sample image and the corresponding voxelized structure to obtain a trained three-dimensional reconstruction model.

5. The method according to claim 4, characterized in that, Before training the 3D reconstruction model using the 2D sample image and the corresponding voxelized structure, the method further includes: An image encoder module, a pixel and voxel matching module, a hierarchical attention module, a self-attention module, a feature fusion module, and a 3D reconstruction module are constructed to obtain a 3D reconstruction model.

6. The method according to claim 5, characterized in that, The constructed image encoder module, pixel and voxel matching module, hierarchical attention module, self-attention module, feature fusion module, and 3D reconstruction module include: An image encoder module is constructed using a pre-trained visual encoder; A pixel-to-voxel matching module is constructed using a cross-attention mechanism; A hierarchical attention module is constructed by combining attention mechanisms of different dimensions; A self-attention module is constructed using a multi-head self-attention mechanism; A feature fusion module is constructed using a pre-trained deep residual network model; A 3D reconstruction module is constructed using a pre-trained 3D decoder or a pre-trained generative model.

7. A sparse-viewpoint 3D reconstruction device, characterized in that, The device includes: An image encoder module is used to extract two-dimensional features from sparse viewpoint images; The pixel and voxel matching module is used to perform pixel and voxel matching based on the two-dimensional features to obtain optimized features; The hierarchical attention module is used to process the optimized features corresponding to sliding windows of different sizes through a self-attention mechanism to obtain the hierarchical local features of the scene to be reconstructed. A self-attention module is used to extract global features of the scene to be reconstructed from the optimized features; The feature fusion module is used to fuse the hierarchical local features and the global features to obtain fused features; The 3D reconstruction module is used to perform 3D reconstruction based on the fusion features to obtain 3D model data of the scene to be reconstructed.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images