A three-dimensional reconstruction method and apparatus, electronic device, storage medium and computer program product

Abstract

This disclosure relates to the field of computer technology, and in particular to a 3D reconstruction method and apparatus, electronic device, storage medium, and computer program product. The method includes: determining a visual token sequence and a text token sequence corresponding to the target scene based on raw video data captured in the target scene; concatenating the visual token sequence and the text token sequence and inputting them into a Mamba-transformer hybrid model to obtain an enhanced multimodal token sequence corresponding to the target scene; determining 3D reconstruction parameters corresponding to the target scene based on the enhanced visual token sequence included in the enhanced multimodal token sequence; and performing 3D reconstruction based on the 3D reconstruction parameters to obtain a 3D model corresponding to the target scene. The embodiments of this disclosure can effectively improve the modeling efficiency of 3D reconstruction.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, and particularly relates to a three-dimensional reconstruction method and device, electronic equipment, storage medium and computer program product. BACKGROUND

[0002] Three-dimensional reconstruction refers to a process of reconstructing or synthesizing a three-dimensional model from two-dimensional images or videos, which combines computer vision, deep learning and geometric modeling technology. Existing three-dimensional reconstruction methods face problems of large consumption of computing resources and high memory occupation when processing large-scale data. SUMMARY

[0003] Therefore, the present disclosure provides a three-dimensional reconstruction method, which comprises: determining a visual token sequence and a text token sequence corresponding to a target scene according to original video data shot under the target scene; inputting the visual token sequence and the text token sequence after splicing into a mamba-transformer hybrid model to obtain an enhanced multi-modal token sequence corresponding to the target scene; determining three-dimensional reconstruction parameters corresponding to the target scene according to an enhanced visual token sequence included in the enhanced multi-modal token sequence; and performing three-dimensional reconstruction according to the three-dimensional reconstruction parameters to obtain a three-dimensional model corresponding to the target scene.

[0004] In a possible implementation manner, the visual token sequence corresponding to the target scene is determined according to the original video data shot under the target scene, which comprises: performing spatio-temporal compression on the original video data to obtain a video latent representation; performing motion structure recovery on the original video data to obtain a camera pose representation; and performing double-branch feature extraction on the video latent representation and the camera pose representation to obtain the visual token sequence.

[0005] In a possible implementation manner, the original video data is subjected to spatio-temporal compression to obtain a video latent representation, which comprises: performing local feature extraction on the original video data to obtain compressed video data; and performing inter-frame information interaction on the compressed video data based on a causal attention mechanism to obtain the video latent representation.

[0006] In a possible implementation manner, the original video data is subjected to motion structure recovery to obtain a camera pose representation, which comprises: determining a camera pose of each original video frame included in the original video data; and performing Plucker coordinate conversion on the camera pose of each original video frame to obtain the camera pose representation.

[0007] In one possible implementation, the step of performing bi-branch feature extraction on the video latent representation and the camera pose representation to obtain the visual token sequence includes: performing video feature extraction on the video latent representation to obtain video features of the target feature dimension; performing camera feature extraction on the camera pose representation to obtain camera features of the target feature dimension; and performing flattening and stitching dimensionality reduction on the video features and the camera features to obtain the visual token sequence of the target feature dimension.

[0008] In one possible implementation, determining the text token sequence corresponding to the target scene based on the original video data captured in the target scene includes: determining the scene description text corresponding to the target scene based on the original video data; and encoding the scene description text to obtain the text token sequence of the target feature dimension.

[0009] In one possible implementation, determining the 3D reconstruction parameters corresponding to the target scene based on the enhanced visual token sequence included in the enhanced multimodal token sequence includes: inputting the enhanced visual token sequence into a preset decoder and obtaining the 3D reconstruction parameters through regression prediction.

[0010] In one possible implementation, the 3D reconstruction parameters include: 3D point cloud coordinates, Gaussian parameters, and transparency.

[0011] In one possible implementation, the mamba-transformer hybrid model includes multiple network layers, each of which includes multiple mamba layers and one transformer layer.

[0012] According to one aspect of this disclosure, a three-dimensional reconstruction apparatus is provided, comprising: a first determining module, configured to determine a visual token sequence and a text token sequence corresponding to the target scene based on original video data captured in the target scene; a second determining module, configured to concatenate the visual token sequence and the text token sequence and input them into a Mamba-transformer hybrid model to obtain an enhanced multimodal token sequence corresponding to the target scene; a third determining module, configured to determine three-dimensional reconstruction parameters corresponding to the target scene based on the enhanced visual token sequence included in the enhanced multimodal token sequence; and a fourth determining module, configured to perform three-dimensional reconstruction based on the three-dimensional reconstruction parameters to obtain a three-dimensional model corresponding to the target scene.

[0013] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.

[0014] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.

[0015] According to another aspect of this disclosure, a computer program product is provided, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above-described method.

[0016] The 3D reconstruction method of this disclosure determines the visual token sequence and text token sequence corresponding to the target scene based on the original video data captured in the target scene. Subsequently, the text token sequence can be used for semantic guidance to supplement the information lost in the video token sequence. The visual token sequence and text token sequence are concatenated and input into the Mamba-transformer hybrid model. Compared with the simple transformer model, the Mamba-transformer hybrid model can effectively reduce the consumption of large computing resources and high memory usage, thereby efficiently outputting the enhanced multimodal token sequence. This enables the generation of high-quality 3D reconstruction parameters based on the high-quality enhanced visual token sequence included in the enhanced multimodal token sequence obtained by the semantic guidance mechanism. Then, 3D reconstruction is performed based on the high-quality 3D reconstruction parameters to obtain a high-quality 3D model that meets the user's needs and effectively improves the modeling efficiency of 3D reconstruction.

[0017] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0018] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0019] Figure 1 A flowchart of a three-dimensional reconstruction method according to an embodiment of the present disclosure is shown.

[0020] Figure 2 A block diagram of a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. Detailed Implementation

[0022] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0023] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.

[0024] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.

[0025] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.

[0026] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0027] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0028] In the field of 3D reconstruction and rendering, existing research has proposed various feed-forward methods for reconstructing 3D scenes or objects from images and videos. Early work mainly employed convolutional neural networks (CNNs) to estimate neural points or multi-plane images, and achieved rendering through point splatting or alpha compositing. Recent studies have yielded extensive results, such as volumetric rendering methods based on NeRF (Neural Radiation Field), which utilize models such as epipolar lines or plane-swept cost-volumes, similar to multi-view stereo (MVS) methods. These methods perform well in handling object and scene-level reconstruction, but have limitations in learning and generalizing multi-view correspondences.

[0029] Recently, the emergence of 3D Gaussian Splatting technology has provided competitive rendering effects and significantly efficient scene rendering. Based on this, the Large Reconstruction Model (LRM) method, by introducing a transformer architecture, achieves high-quality 3D reconstruction. While the LRM method based on the transformer architecture can generate high-quality 3D models, it suffers from the following problems: 1. High computational resource consumption: The transformer architecture requires significant computational resources when processing high-resolution video input, leading to slow training and inference processes. 2. High memory usage: The self-attention mechanism of the transformer architecture causes the LRM method to consume a large amount of video memory when processing large-scale video data, limiting the number and resolution of input videos. 3. Low modeling efficiency: Due to the aforementioned limitations in computational and memory resources, existing 3D reconstruction methods struggle to efficiently process large amounts of video data, affecting the efficiency and quality of 3D reconstruction. 4. Lack of semantic guidance: Existing LRM methods mainly rely on the latent representation of the video itself, lacking semantic guidance for the 3D reconstruction process, making it difficult to meet users' high requirements for 3D reconstruction quality.

[0030] To address the related technical problems, this disclosure provides a three-dimensional reconstruction method that can improve the quality of three-dimensional reconstruction while reducing the computational resource consumption and memory usage during the three-dimensional reconstruction process. The three-dimensional reconstruction method provided by this disclosure is described in detail below.

[0031] Figure 1 A flowchart illustrating a three-dimensional reconstruction method according to an embodiment of this disclosure is shown. Figure 1 As shown, the method includes:

[0032] In step S11, the visual token sequence and text token sequence corresponding to the target scene are determined based on the original video data captured in the target scene.

[0033] In step S12, the visual token sequence and the text token sequence are concatenated and input into the mamba-transformer hybrid model to obtain the enhanced multimodal token sequence corresponding to the target scene.

[0034] In step S13, the three-dimensional reconstruction parameters corresponding to the target scene are determined based on the enhanced visual token sequence included in the enhanced multimodal token sequence.

[0035] In step S14, three-dimensional reconstruction is performed based on the three-dimensional reconstruction parameters to obtain the three-dimensional model corresponding to the target scene.

[0036] The 3D reconstruction method of this disclosure determines the visual token sequence and text token sequence corresponding to the target scene based on the original video data captured in the target scene. Subsequently, the text token sequence can be used for semantic guidance to supplement the information lost in the video token sequence. The visual token sequence and text token sequence are concatenated and input into the Mamba-transformer hybrid model. Compared with the simple transformer model, the Mamba-transformer hybrid model can effectively reduce the consumption of large computing resources and high memory usage, thereby efficiently outputting the enhanced multimodal token sequence. This enables the generation of high-quality 3D reconstruction parameters based on the high-quality enhanced visual token sequence included in the enhanced multimodal token sequence obtained by the semantic guidance mechanism. Then, 3D reconstruction is performed based on the high-quality 3D reconstruction parameters to obtain a high-quality 3D model that meets the user's needs.

[0037] The target scene is the scene that needs to be reconstructed in three dimensions. The specific form can be flexibly changed according to actual needs. This disclosure does not impose any specific limitations on it.

[0038] In one possible implementation, the visual token sequence corresponding to the target scene is determined based on the original video data captured in the target scene, including: performing spatiotemporal compression on the original video data to obtain a video latent representation; performing motion structure recovery on the original video data to obtain a camera pose representation; and performing bi-branch feature extraction on the video latent representation and the camera pose representation to obtain a visual token sequence.

[0039] Spatiotemporal compression is performed on the raw video data to extract its spatiotemporal features, resulting in a video latent representation. This reduces the dimensionality and redundancy of the raw video data, decreases memory usage, and improves the efficiency of subsequent processing. Motion structure recovery is then performed on the raw video data to recover the camera pose representation by analyzing the motion information of objects within the video data, thereby enhancing the understanding of the geometric structure in the target scene. Furthermore, a dual-branch feature extraction process is applied to the video latent representation and the camera pose representation, fusing the visual and geometric information from the raw video data to obtain a visual token sequence that can be processed by the Mamba-transformer hybrid model.

[0040] In one example, the raw video data captured in the target scene is (B=1, t=49, C=3, W=480, H=720). Here, B is the batch size, B=1 indicates one raw video data point; t is the number of video frames, t=49 indicates 49 raw video frames; C is the number of channels, C=3 indicates 3 channels, for example, RGB; W is the width of the video frame, and H is the height of the video frame, W=480 and H=720 indicate that the resolution of each raw video frame is 480×720.

[0041] After spatiotemporal compression of the original video data (B=1, t=49, C=3, W=480, H=720), the video latent representation (B=1, t=13, C=16, W=90, H=60) is obtained.

[0042] In one possible implementation, spatiotemporal compression of the original video data is performed to obtain the video latent representation, including: extracting local features from the original video data to obtain compressed video data; and performing inter-frame information exchange on the compressed video data based on a causal attention mechanism to obtain the video latent representation.

[0043] Local feature extraction is performed on the original video data to capture local spatial features within video frames, resulting in compressed video data. Then, based on a causal attention mechanism, inter-frame information exchange is performed on the compressed video data to focus on the temporal dependencies between video frames. This effectively extracts the temporal features from the compressed video data, yielding a compact and information-rich latent video representation, providing strong data support for subsequent 3D reconstruction tasks.

[0044] In one example, the original video data is spatiotemporally compressed based on the encoder in a 3D variational autoencoder (3D VAE) to obtain a latent representation of the video, thereby effectively reducing the video memory usage.

[0045] In one example, the encoder in the 3D VAE uses a CogVideoX-style causal tokenizer to spatiotemporally compress the raw video data to obtain the video latent representation.

[0046] Taking the original video data (B=1, t=49, C=3, W=480, H=720) as an example, firstly, local features are extracted from the original video data (B=1, t=49, C=3, W=480, H=720), thereby downsampling the spatiotemporal dimensions (t, W, H) of the original video data (B=1, t=49, C=3, W=480, H=720) and preserving the local spatiotemporal features.

[0047] For example, a 3D convolutional kernel (5×8×8) is used to extract local features from the original video data (B=1, t=49, C=3, W=480, H=720), where the 3D convolutional kernel corresponds to (t×W×H). After spatiotemporal compression following local feature extraction, compressed video data (B=1, t=13, C=16, W=90, H=60) is obtained. The compressed video data (B=1, t=13, C=16, W=90, H=60) is a feature map with C=16 channels, and the spatiotemporal resolution is compressed from t×W×H=49×480×720 in the original video data to t×W×H=13×90×60.

[0048] Furthermore, based on the causal attention mechanism, a spatiotemporal attention layer is used to perform inter-frame information interaction on the compressed video data (B=1, t=13, C=16, W=90, H=60) to model the global inter-frame dependency, enhance temporal consistency, and obtain the final video latent representation.

[0049] For example, the compressed video data (B=1, t=13, C=16, W=90, H=60) is flattened in the time dimension to obtain a token sequence (B=1, N=t×W×H=13×90×60=70200, C=16). The token sequence (B=1, N=70200, C=16) is then subjected to position encoding and attention calculation, ensuring that the current frame only focuses on historical frames to avoid information leakage. Then, the token sequence after position encoding and attention calculation is restored to the video latent representation (B=1, t=13, C=16, W=90, H=60).

[0050] Although the video latent representation (B=1, t=13, C=16, W=90, H=60) and the compressed video data (B=1, t=13, C=16, W=90, H=60) have the same form, the global spatiotemporal context has been incorporated into the video latent representation (B=1, t=13, C=16, W=90, H=60).

[0051] In one possible implementation, motion structure recovery is performed on the original video data to obtain a camera pose representation, including: determining the camera pose of each original video frame included in the original video data; and performing Plück coordinate transformation on the camera pose of each original video frame to obtain a camera pose representation.

[0052] By determining the camera pose of each original video frame included in the original video data, the motion change patterns in the original video data can be accurately captured, providing reliable basic data for subsequent 3D reconstruction.

[0053] In one example, the colmap tool is used to calculate the camera position and angle of each original video frame, thereby obtaining the camera pose of each original video frame. The specific process of the colmap tool in calculating the camera pose can be found in related technologies, and this disclosure does not specifically limit it.

[0054] The camera pose of each original video frame is converted into Plück coordinates to obtain a differentiable and compact mathematical representation: the camera pose representation in Plück coordinates. This simplifies ray manipulation and enhances the geometric perception capabilities of subsequent models.

[0055] For details on the specific process of converting the camera pose of each original video frame into Plück coordinates, please refer to relevant technologies; this disclosure does not impose any specific limitations on it.

[0056] After obtaining the video latent representation and camera pose representation based on the original video data, a dual-branch feature extraction is performed on the video latent representation and camera pose representation to obtain a visual token sequence that can be processed by the Mamba-Transformer hybrid model.

[0057] In one possible implementation, a dual-branch feature extraction is performed on the video latent representation and the camera pose representation to obtain a visual token sequence, including: extracting video features from the video latent representation to obtain video features in the target feature dimension; extracting camera features from the camera pose representation to obtain camera features in the target feature dimension; and flattening and stitching the video features and camera features to reduce dimensionality and obtain a visual token sequence in the target feature dimension.

[0058] By extracting features independently from the video latent representation and the camera pose representation, we can fully explore their unique feature information. Then, we can flatten and stitch the extracted video features and camera features to reduce dimensionality, thereby effectively achieving the fusion of multimodal features and significantly reducing computational complexity.

[0059] In one example, the dual branches include a video feature extraction branch and a camera feature extraction branch, wherein the video feature extraction branch and the camera feature extraction branch can be pre-designed convolutional neural networks (CNNs). The CNN network corresponding to the video feature extraction branch and the CNN network corresponding to the camera feature extraction branch can have the same network structure or different network structures, and this disclosure does not specifically limit them.

[0060] Taking the aforementioned video latent representation (B=1, t=13, C=16, W=90, H=60) as an example, the CNN network corresponding to the video feature extraction branch is used to extract video features from the video latent representation (B=1, t=13, C=16, W=90, H=60), resulting in video features (B=1, t=13, C=768, W=90, H=60), where the target feature dimension C=768.

[0061] Using the CNN network corresponding to the camera feature extraction branch, camera features are extracted from the camera pose representation to obtain camera features of the same size as the video features (B=1, t=13, C=768, W=90, H=60).

[0062] The video features (B=1, t=13, C=768, W=90, H=60) and camera features (B=1, t=13, C=768, W=90, H=60) are flattened into one-dimensional token sequences (B=1, N=t×W×H=13×90×60=70200, C=768), where N=70200 is the sequence length.

[0063] Two one-dimensional token sequences (B=1, N=70200, C=768) are concatenated along the feature dimension C to achieve multimodal feature fusion, resulting in a one-dimensional concatenated token sequence (B=1, N=70200, C=1536).

[0064] The one-dimensional concatenated token sequence (B=1, N=70200, C=1536) is passed through a linear layer to reduce the feature dimension C, resulting in a one-dimensional visual token sequence (B=1, N=70200, C=768). Here, N=70200 is the sequence length of the visual token sequence, meaning that the visual token sequence contains N=70200 visual tokens; C=768 is the target feature dimension of the visual token sequence, meaning that each visual token in the visual token sequence has C=768 feature dimensions.

[0065] The target feature dimension C can be set to the above-mentioned 768, or it can be set to other values ​​according to the actual application scenario requirements. This disclosure does not make specific limitations on this.

[0066] In one possible implementation, the text token sequence corresponding to the target scene is determined based on the original video data captured in the target scene, including: determining the scene description text corresponding to the target scene based on the original video data; and encoding the scene description text to obtain the text token sequence of the target feature dimension.

[0067] Based on the original video data, the scene description text corresponding to the target scene is determined, and then the scene description text is encoded to effectively map the relevant information of the target scene to the text token sequence of the target feature dimension, thereby achieving more efficient scene understanding and subsequent auxiliary visual token sequence for modeling.

[0068] In one example, the raw video data is input into the qwen-VL model, which automatically generates scene description text corresponding to the target scene. Then, the scene description text is encoded using a T5 encoder to obtain a text token sequence (B=1, N=70200, C=1536), where N=70200 is the sequence length of the text token sequence, that is, the text token sequence includes N=70200 text tokens; C=768 is the target feature dimension of the text token sequence, that is, each text token in the text token sequence has C=768 feature dimensions.

[0069] After determining the visual token sequence and text token sequence corresponding to the target scene, the visual token sequence and text token sequence are concatenated and input into the Mamba-transformer hybrid model to achieve attention fusion between the multimodal components and obtain the enhanced multimodal token sequence corresponding to the target scene.

[0070] In 3D reconstruction, the transformer model is responsible for handling long-range dependencies and global features in video data, ensuring that the generated 3D model has high quality and accuracy. However, when processing large-scale scene data, the transformer model faces problems such as high computational resource consumption, high GPU memory usage, and slow training speed.

[0071] Mamba is a lightweight neural network architecture used for efficiently processing the spatiotemporal characteristics of video data. Through its unique network design, the Mamba model can significantly reduce memory usage and computational resource consumption.

[0072] The mamba-transformer hybrid model in this embodiment is a hybrid neural network model that combines the mamba model and the transformer model.

[0073] In one possible implementation, the mamba-transformer hybrid model includes multiple network layers, each of which comprises multiple mamba layers and one transformer layer.

[0074] The number of network layers included in the mamba-transformer hybrid model, and the number of Mamba layers included in each network layer, can be flexibly set according to the actual application needs, and this disclosure does not impose specific limitations on this.

[0075] In one example, the training process for the mamba-transformer hybrid model includes:

[0076] The first step is to determine the visual token sequence and text token sequence corresponding to the training scenario based on the original video data captured in the training scenario.

[0077] The training scenario is a scenario used to train the mamba-transformer hybrid model. The specific form can be flexibly set according to actual needs, and this disclosure does not impose specific limitations on it.

[0078] The process of determining the visual token sequence and text token sequence corresponding to the training scene is similar to the process of determining the visual token sequence and text token sequence corresponding to the target scene, and will not be elaborated here.

[0079] The second step involves concatenating the visual token sequence and the text token sequence and inputting them into the mamba-transformer hybrid model to obtain the enhanced multimodal token sequence corresponding to the training scenario.

[0080] The third step is to determine the 3D reconstruction parameters corresponding to the training scene based on the enhanced visual token sequence included in the enhanced multimodal token sequence.

[0081] The fourth step is to render a new perspective video based on the 3D reconstruction parameters.

[0082] The fifth step involves comparing the rendered new perspective video with the original video data captured in the training scene, and calculating the training loss using perceptual loss and L1 loss functions.

[0083] The fifth step is to optimize the mamba-transformer hybrid model based on the training loss.

[0084] The sixth step is to iteratively execute the above training process until the training loss meets the preset conditions, thus obtaining the trained mamba-transformer hybrid model.

[0085] The Mamba-transformer hybrid model can utilize the Mamba layer to process long sequences to reduce computational costs, and leverage the global modeling capabilities of the transformer layer to ensure high quality and high accuracy of 3D reconstruction.

[0086] Compared to existing transformer models that only include computationally expensive transformer layers, the Mamba-transformer hybrid model of this disclosure reduces computational costs and increases training speed by replacing some transformer layers with Mamba layers.

[0087] Furthermore, a well-trained Mamba-transformer hybrid model can improve inference speed when used for 3D reconstruction. Additionally, due to reduced computational costs, a well-trained Mamba-transformer hybrid model can handle longer and higher-resolution raw video data when used for 3D reconstruction, effectively improving the 3D reconstruction results.

[0088] The visual token sequence and text token sequence corresponding to the target scene determined above are concatenated and then input into the trained mamba-transformer hybrid model.

[0089] Taking the above visual token sequence (B=1, N=70200, C=768) and text token sequence (B=1, N=70200, C=1536) as an example, after concatenating the visual token sequence (B=1, N=70200, C=768) and the text token sequence (B=1, N=70200, C=1536), the fused token sequence (B=1, N=140400, C=768) is obtained.

[0090] The fusion token sequence (B=1, N=140400, C=768) is input into the trained Mamba-transformer hybrid model. The Mamba-transformer hybrid model performs cross-modal attention fusion, thereby enabling text-guided modeling to supplement the information lost in the video latent representation and improve the quality of subsequent 3D reconstruction.

[0091] The Mamba-transformer hybrid model performs cross-modal attention fusion on the fused token sequence, outputting an enhanced multimodal token sequence (B=1, N=140400, C=768). Although the enhanced multimodal token sequence (B=1, N=140400, C=768) and the fused token sequence (B=1, N=140400, C=768) have the same form, the enhanced multimodal token sequence (B=1, N=140400, C=768) has already incorporated information from the text token sequence into the visual token sequence.

[0092] The enhanced multimodal token sequence (B=1, N=140400, C=768) also includes two parts: the enhanced visual token sequence (B=1, N=70200, C=768) and the enhanced text token sequence (B=1, N=70200, C=768). During the 3D reconstruction process, only the enhanced visual token sequence (B=1, N=70200, C=768) containing image information is used.

[0093] In one possible implementation, the 3D reconstruction parameters corresponding to the target scene are determined based on the enhanced visual token sequence included in the enhanced multimodal token sequence, including: converting the format of the enhanced visual token sequence to obtain the enhanced video latent representation; inputting the enhanced video latent representation into a preset decoder, and regressing to predict the 3D reconstruction parameters.

[0094] The enhanced visual token sequence (B=1, N=70200, C=768) is converted into a tensor format (B, t, C, W, H) corresponding to the video latent representation, i.e., the enhanced video latent representation (B, t, C, W, H). Then, the enhanced video latent representation (B, t, C, W, H) is input into the decoder in the 3D VAE, and regression prediction is used to obtain the 3D reconstruction parameters corresponding to the target scene.

[0095] In one possible implementation, the 3D reconstruction parameters include: 3D point cloud coordinates, Gaussian parameters, and transparency.

[0096] Based on 3D point cloud coordinates, Gaussian parameters, and transparency, it is possible to complete the 3D reconstruction of the target scene and obtain the corresponding 3D model of the target scene.

[0097] The process of 3D reconstruction based on 3D reconstruction parameters can refer to relevant technologies, and this disclosure does not make specific limitations on it.

[0098] The 3D reconstruction method of this disclosure determines the visual token sequence and text token sequence corresponding to the target scene based on the original video data captured in the target scene. Subsequently, the text token sequence can be used for semantic guidance to supplement the information lost in the video token sequence. The visual token sequence and text token sequence are concatenated and input into the Mamba-transformer hybrid model. Compared with the simple transformer model, the Mamba-transformer hybrid model can effectively reduce the consumption of large computing resources and high memory usage, thereby efficiently outputting the enhanced multimodal token sequence. This enables the generation of high-quality 3D reconstruction parameters based on the high-quality enhanced visual token sequence included in the enhanced multimodal token sequence obtained by the semantic guidance mechanism. Then, 3D reconstruction is performed based on the high-quality 3D reconstruction parameters to obtain a high-quality 3D model that meets the user's needs and effectively improves the modeling efficiency of 3D reconstruction.

[0099] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0100] In addition, this disclosure also provides a three-dimensional reconstruction apparatus, electronic equipment, computer-readable storage medium, and program, all of which can be used to implement any of the three-dimensional reconstruction methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding section of the method and will not be repeated here.

[0101] Figure 2 A block diagram of a three-dimensional reconstruction apparatus according to an embodiment of the present disclosure is shown. Figure 2 As shown, device 20 includes:

[0102] The first determining module 21 is used to determine the visual token sequence and text token sequence corresponding to the target scene based on the original video data captured in the target scene.

[0103] The second determining module 22 is used to concatenate the visual token sequence and the text token sequence and input them into the mamba-transformer hybrid model to obtain the enhanced multimodal token sequence corresponding to the target scene;

[0104] The third determining module 23 is used to determine the three-dimensional reconstruction parameters corresponding to the target scene based on the enhanced visual token sequence included in the enhanced multimodal token sequence.

[0105] The fourth determining module 24 is used to perform three-dimensional reconstruction based on the three-dimensional reconstruction parameters to obtain the three-dimensional model corresponding to the target scene.

[0106] In one possible implementation, the first determining module 21 is specifically used for:

[0107] Spatiotemporal compression is performed on the original video data to obtain the video latent representation;

[0108] Motion structure recovery is performed on the original video data to obtain the camera pose representation;

[0109] A dual-branch feature extraction process is performed on the video latent representation and the camera pose representation to obtain a visual token sequence.

[0110] In one possible implementation, the first determining module 21 is specifically used for:

[0111] Local feature extraction is performed on the original video data to obtain compressed video data;

[0112] Based on the causal attention mechanism, inter-frame information exchange is performed on the compressed video data to obtain the video latent representation.

[0113] In one possible implementation, the first determining module 21 is specifically used for:

[0114] Determine the camera pose of each raw video frame included in the raw video data;

[0115] Perform Plück coordinate transformation on the camera pose of each original video frame to obtain the camera pose representation.

[0116] In one possible implementation, the first determining module 21 is specifically used for:

[0117] Video features are extracted from the video latent representation to obtain video features of the target feature dimension;

[0118] Camera features are extracted from the camera pose representation to obtain camera features in the target feature dimension;

[0119] The video features and camera features are flattened and stitched together to reduce dimensionality, resulting in a visual token sequence with the target feature dimension.

[0120] In one possible implementation, the first determining module 21 is specifically used for:

[0121] Based on the original video data, determine the scene description text corresponding to the target scene;

[0122] The scene description text is encoded to obtain a text token sequence of the target feature dimension.

[0123] In one possible implementation, the third determining module 23 is specifically used for:

[0124] The format of the enhanced visual token sequence is converted to obtain the enhanced video latent representation;

[0125] The enhanced video latent representation is input into a preset decoder, and regression prediction is used to obtain the 3D reconstruction parameters.

[0126] In one possible implementation, the 3D reconstruction parameters include: 3D point cloud coordinates, Gaussian parameters, and transparency.

[0127] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0128] This disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above method.

[0129] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.

[0130] This disclosure also provides a computer program product, including a computer program or a non-volatile computer-readable storage medium carrying the computer program, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0131] Figure 3 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. (Refer to...) Figure 3 Device 1900 can be provided as a server or terminal device. (See reference...) Figure 3 The apparatus 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0132] Device 1900 may also include a power supply component 1926 configured to perform power management of device 1900, a wired or wireless network interface 1950 configured to connect device 1900 to a network, and an input / output interface 1958 (I / O interface). Device 1900 can operate on an operating system, such as Windows Server, stored in memory 1932. TM macOS X TM UnixTM Linux TM FreeBSD TM Or similar.

[0133] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of the device 1900 to perform the above-described method.

[0134] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0135] The computer program (or computer-readable program instructions) described herein can be downloaded from a computer-readable storage medium to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage medium in the respective computing / processing device.

[0136] The computer program (or computer program instructions) used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing state information from the computer-readable program instructions to implement various aspects of this disclosure.

[0137] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0138] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0139] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0141] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A three-dimensional reconstruction method, characterized by, The method comprises the following steps: According to the original video data shot under the target scene, the visual token sequence corresponding to the target scene and the text token sequence corresponding to the target scene are determined; The visual token sequence and the text token sequence are spliced and input into a mamba-transformer hybrid model to obtain an enhanced multi-modal token sequence corresponding to the target scene, wherein the mamba-transformer hybrid model comprises a plurality of network layers, each network layer comprises a plurality of mamba layers and a transformer layer; According to the enhanced visual token sequence included in the enhanced multi-modal token sequence, the three-dimensional reconstruction parameters corresponding to the target scene are determined; According to the three-dimensional reconstruction parameters, three-dimensional reconstruction is performed to obtain a three-dimensional model corresponding to the target scene; According to the original video data shot under the target scene, the text token sequence corresponding to the target scene is determined, comprising: According to the original video data, the scene description text corresponding to the target scene is determined; The scene description text is encoded to obtain the text token sequence of the target feature dimension.

2. The method of claim 1, wherein, According to the original video data shot under the target scene, the visual token sequence corresponding to the target scene is determined, comprising: The original video data is spatio-temporally compressed to obtain a video latent representation; The original video data is subjected to motion structure recovery to obtain a camera pose representation; The video latent representation and the camera pose representation are subjected to double-branch feature extraction to obtain the visual token sequence.

3. The method of claim 2, wherein, The original video data is spatio-temporally compressed to obtain a video latent representation, comprising: Local feature extraction is performed on the original video data to obtain compressed video data; Based on the causal attention mechanism, inter-frame information interaction is performed on the compressed video data to obtain the video latent representation.

4. The method of claim 2, wherein, The original video data is subjected to motion structure recovery to obtain a camera pose representation, comprising: The camera pose of each original video frame included in the original video data is determined; The camera pose of each original video frame is subjected to Plucker coordinate conversion to obtain the camera pose representation.

5. The method of claim 2, wherein, The video latent representation and the camera pose representation are subjected to double-branch feature extraction to obtain the visual token sequence, comprising: Video feature extraction is performed on the video latent representation to obtain video features of a target feature dimension; Camera feature extraction is performed on the camera pose representation to obtain camera features of a target feature dimension; The video features and the camera features are subjected to flat splicing and dimension reduction to obtain the visual token sequence of a target feature dimension.

6. The method of claim 1, wherein, The enhanced visual token sequence is subjected to format conversion to obtain an enhanced video latent representation; The enhanced video latent representation is input into a preset decoder to regress and predict the three-dimensional reconstruction parameters. The three-dimensional reconstruction parameters include three-dimensional point cloud coordinates, Gaussian parameters, and transparency.

7. The method of claim 6, wherein, The method comprises the following steps:

8. A three-dimensional reconstruction apparatus, characterized by comprising: ​ The first determining module is configured to determine a visual token sequence and a text token sequence corresponding to the target scene according to original video data captured in the target scene; The second determining module is configured to input the visual token sequence and the text token sequence after splicing into a mamba-transformer hybrid model to obtain an enhanced multi-modal token sequence corresponding to the target scene, wherein the mamba-transformer hybrid model comprises a plurality of network layers, and each network layer comprises a plurality of mamba layers and a transformer layer; The third determining module is configured to determine three-dimensional reconstruction parameters corresponding to the target scene according to an enhanced visual token sequence included in the enhanced multi-modal token sequence; The fourth determining module is configured to perform three-dimensional reconstruction according to the three-dimensional reconstruction parameters to obtain a three-dimensional model corresponding to the target scene; The first determining module is configured to determine scene description text corresponding to the target scene according to the original video data; encode the scene description text to obtain the text token sequence of the target feature dimension.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory, wherein the computer program comprises instructions that, when executed by the processor, cause the electronic device to perform the method of any one of claims 1-8. The processor executes the computer program to implement the steps of the method of any one of claims 1 to 7.

10. A non-transitory computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

11. A computer program product comprising a computer program or a non-transitory computer-readable storage medium bearing a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 7.

Images