Ancient building three-dimensional reconstruction method based on multi-modal data fusion

Abstract

The invention discloses an ancient building three-dimensional reconstruction method based on multi-modal data fusion, and relates to the technical field of data processing, and the method comprises the steps: obtaining multi-modal data of an ancient building and historical culture, and constructing a data warehouse for storing the obtained multi-modal data of the ancient building and historical culture; performing semantic alignment processing on the multi-modal data of the historic building and the historical culture to obtain multi-modal fusion feature information; inputting the multi-modal fusion feature information into a completely trained artificial intelligence neural network model for depth calculation and estimation judgment, and generating depth map data; and obtaining point cloud data based on the depth map data, and performing three-dimensional model reconstruction according to the obtained point cloud data and a three-dimensional reconstruction technology to complete virtual reality implementation of the ancient building. Based on an artificial intelligence multi-modal data fusion technology, the invention creates a multi-modal data fusion three-dimensional reconstruction virtual reality method for an ancient building and historical culture thereof based on artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method for three-dimensional reconstruction of ancient buildings based on multimodal data fusion. Background Art

[0002] With the continuous development of information technology and the advent of the big data era, multimedia data in different modalities (such as text, images, audio, video, etc.) are explosively presented on the Internet. These different modal data are usually combined to describe the same topic or event. They are heterogeneous and multi-source in terms of expression, and the semantic information expressed is interrelated. According to human brain cognitive science research, the fusion of multiple sensory information is more helpful for the human brain to perceive and understand the objective world. Therefore, semantic analysis and association modeling of multi-(cross-)modal data have become hot research topics in the fields of machine learning and pattern recognition. In recent years, with the rapid development of information collection equipment and emerging social media networks, people are sharing a large amount of relevant multimodal information while increasingly expecting to accurately and efficiently obtain content of interest from massive multi-source data. Therefore, multimodal information feature extraction and fusion technology has always occupied an important position in the field of target analysis and detection.

[0003] Currently, common technologies are mostly based on a single modality, such as images, text, and video. In today's era of prevalent multimodal data, this feature extraction technology reduces the flexibility and diversity of people's information acquisition methods and limits their multisensory understanding of information. Therefore, a new cross-modality feature representation model is needed that can associate and represent data from different modalities with the same semantics, providing a foundation for downstream multimodal tasks. For example, for cross-modal retrieval tasks, semantically related results from any modality can be returned based on a query example. For example, given a laser scanned image of a particular scenic spot or object as a query, cross-modal feature representation can return relevant text descriptions, audio materials, and other modal content to complement their visual perception. Specifically, given an image with the visual content "airplane," a text description of "airplane" is returned; this is the text subtask of image retrieval. Conversely, given a text description of "bus," a related visual image is returned; this is the image subtask of text retrieval. This feature representation method can return semantically related or complementary multimedia data, thereby more comprehensively and accurately interpreting a topic and significantly improving the efficiency of target retrieval and classification. Therefore, the research and innovation of cross-modal representation has far-reaching practical significance. This also has great practical significance in the multimodal data fusion of ancient buildings and their historical culture, and in the reconstruction and protection of ancient buildings and historical culture based on the semantic correlation and complementarity of images and texts.

[0004] Previous data sources for ancient architectural history and culture reveal that much existing data on ancient buildings are single, fragmented, and unintegrated datasets. This presents a drawback: using single data sets (such as images, audio, or text) for 3D reconstruction of ancient buildings is likely to result in incomplete 3D reconstructions, often missing important information related to the buildings. This can lead to poor results. Furthermore, 3D reconstructions of ancient buildings based on single data are imperfect, carry a risk of defects, and have significant limitations.

[0005] Therefore, in order to better display the historical features of ancient buildings, providing a new technical solution to improve the above-mentioned problems is an issue that technicians in this field urgently need to solve. Summary of the Invention

[0006] In view of this, the present invention provides a three-dimensional reconstruction method of ancient buildings based on multimodal data fusion to solve the above technical problems.

[0007] To achieve the above object, the present invention provides the following technical solutions:

[0008] A method for 3D reconstruction of ancient buildings based on multimodal data fusion, comprising: acquiring multimodal data of ancient buildings and historical culture, and building a data warehouse to store the acquired multimodal data of ancient buildings and historical culture;

[0009] Performing semantic alignment processing on the multimodal data of the ancient buildings and historical culture to obtain multimodal fusion feature information;

[0010] Input the multimodal fusion feature information into a well-trained artificial intelligence neural network model for depth calculation and estimation judgment to generate depth map data;

[0011] Point cloud data is obtained based on the depth map data, and a three-dimensional model is reconstructed according to the obtained point cloud data and three-dimensional reconstruction technology to complete the virtual reality realization of the ancient building.

[0012] Furthermore, the multimodal data of ancient buildings and historical culture includes three groups of modal data: laser scanning image data, voice data and text data.

[0013] Furthermore, semantic alignment processing is performed on the multimodal data of the ancient architecture and historical culture to obtain multimodal fusion feature information including:

[0014] Get any two sets of modal data and use the feature extractor corresponding to each modality to obtain the corresponding serialized modal features;

[0015] Add a special annotation to each corresponding serialized modal feature, and add a global annotation to the two sets of modal data obtained;

[0016] Adding a modality type embedding vector and a position embedding vector to each modality feature to obtain an input feature vector, wherein the input feature vector includes the serialized modality features, the annotation features, the modality type embedding vector, and the position embedding vector of each modality data;

[0017] The input feature vector is input into the transformer network for feature alignment operation. Based on the annotation information, it is determined whether the two sets of modal semantics of the input match. The two sets of modal features are further matched by learning the language model. After completing the matching between the multimodal data of ancient buildings and historical culture, multimodal fusion feature information is obtained.

[0018] Furthermore, the artificial intelligence neural network model is constructed based on the multi-view depth prediction network MVSNet, and the artificial intelligence neural network model includes an image feature extraction module, a cost volume generation module, an initial depth estimation module and a depth map optimization module. The image feature extraction module is used to output N 32-channel feature maps through a 2D convolutional neural network; the cost volume generation module is used to construct a 3D cost volume based on the extracted feature map and camera parameters; the initial depth estimation module is used to perform pixel-by-pixel depth estimation based on the expectation in the depth direction on the basis of constructing the 3D cost volume; the depth map optimization module is used to use a reference image to optimize the predicted depth map based on the residual network.

[0019] Furthermore, the three-dimensional model reconstruction based on the obtained point cloud data and the three-dimensional reconstruction technology includes:

[0020] Preprocess point cloud data;

[0021] The point cloud data is segmented and clustered into multiple point clouds. Each point cloud corresponds to an independent object. A point is selected and a sphere centered on the point is calculated using a kd tree. All points within the sphere belong to the object. The radius of the sphere is set to a threshold, and the remaining points of the object are traversed to segment the point cloud into individual objects.

[0022] The point cloud is triangulated to obtain the spatial topological structure of the point cloud, and mesh rendering is performed to complete 3D modeling to obtain a three-dimensional reconstructed model.

[0023] In summary, the beneficial effects of the present invention are:

[0024] 1. The present invention is different from the traditional 3D reconstruction method. It is divided into data fusion and model construction, avoiding the defects, risks and limitations of the 3D reconstruction method of ancient buildings using single data.

[0025] 2. The model combines multi-source data, laser scanning image data, voice data and text data to generate a three-dimensional model of the ancient building that is more conducive to the display of the historical features of the ancient building, making the display of the three-dimensional model of the ancient building more accurate and high-quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings constituting a part of this application are used to provide a further understanding of the present invention. The illustrative embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0027] Figure 1 The figure is a schematic diagram of the specific steps of the method for 3D reconstruction of ancient buildings based on multimodal data fusion in the present invention.

[0028] Figure 2 Schematic diagram of the process of the ancient building 3D reconstruction method based on multimodal data fusion in this embodiment.

[0029] Figure 3 This is an example diagram of the implementation process of the ancient building three-dimensional reconstruction method based on multimodal data fusion in this embodiment.

[0030] Figure 4 Schematic diagram of the structure of the ancient building 3D reconstruction system based on multimodal data fusion in this embodiment.

[0031] Figure 5 Schematic diagram of the unified overall framework of multimodal feature alignment in this embodiment.

[0032] Figure 6 Schematic diagram of the text-image alignment framework in this embodiment.

[0033] Figure 7 Schematic diagram of video feature extraction in this embodiment.

[0034] Figure 8 This is a schematic diagram of the MVSNet network architecture in this embodiment.

[0035] Figure 9 Schematic diagram of the comparison plot of the probability plot estimation process. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments and the accompanying drawings. Here, the exemplary embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0037] In order to overcome the drawback of poor virtual reality of ancient buildings due to single data three-dimensional reconstruction, the present invention introduces multimodal data fusion technology based on artificial intelligence, and constructs a multimodal data fusion three-dimensional reconstruction method of ancient buildings and their historical culture based on artificial intelligence.

[0038] like Figures 1 to 8 As shown, a method for 3D reconstruction of ancient buildings based on multimodal data fusion of the present invention comprises the following steps:

[0039] Step S1: Acquire multimodal data of ancient buildings and historical culture, and build a data warehouse to store the acquired multimodal data of ancient buildings and historical culture.

[0040] The multimodal data of ancient buildings and historical culture includes three groups of modal data: laser scanning image data, voice data and text data.

[0041] like Figure 2 As shown, the multimodal data related to ancient buildings and their historical culture, including pictures, voice and text, are photographed, collected, sorted, processed and stored. A corresponding data warehouse is built to store these multimodal data for subsequent data analysis, processing and application.

[0042] Step S2: semantic alignment processing is performed on the multimodal data of the ancient architecture and historical culture to obtain multimodal fusion feature information.

[0043] Acquire multimodal data of ancient buildings and their historical and cultural data (including pictures, voice, text, etc.), and perform semantic alignment of multimodal data.

[0044] Specifically, semantic alignment processing is performed on the multimodal data of the ancient architecture and historical culture to obtain multimodal fusion feature information including:

[0045] Step S20: Obtain any two sets of modal data, and use the feature extractor corresponding to each modality to obtain corresponding serialized modal features;

[0046] Step S21: adding a special annotation to each corresponding serialized modal feature, and adding a global annotation to the two sets of acquired modal data;

[0047] Step S22: adding a modality type embedding vector and a position embedding vector to each modality feature to obtain an input feature vector, wherein the input feature vector includes the serialized modality features, the annotation features, the modality type embedding vector, and the position embedding vector of each modality data;

[0048] Step S23: Input the input feature vector into the transformer network for feature alignment operation, determine whether the two sets of input modal semantics match based on the annotation information, and further match the two sets of modal features by learning the language model. After completing the matching between the multimodal data of ancient buildings and historical culture, multimodal fusion feature information is obtained.

[0049] In this embodiment, the multimodal feature alignment method belongs to a single-stream model, and its purpose is to perform fusion and alignment operations on multimodal information from the beginning. In order to meet the requirements of uniformly aligning multimodal features in a framework, the present invention abstracts the network framework and modularizes different modal information to improve network scalability. The schematic diagram of the model framework design is as follows: Figure 5 As shown in the figure, the numbers represent the position embedding vector and the modality type embedding vector.

[0050] First, given two different modal data, the corresponding feature extractors are used to extract the corresponding serialized modal features. For example, BERT is used to obtain text tokens for text data, and Resnet or Swin is used to extract image features for images. After extracting the modality-specific features, a special [CLS] token is added to each modality to facilitate integration with subsequent downstream tasks. Furthermore, a global [CLS] token is added to the two input modalities. This token determines whether the semantics of the two modalities match. If the two modalities have the same semantics, the output is true, thereby aligning the features. Due to the differences between modalities, for example, visual features often contain semantic information in a deeper network, while text features are relatively shallow. To enable the network to effectively distinguish between modalities, a modal-type embedding is added to each modal feature. Furthermore, because the Transformer treats all input data uniformly, it cannot capture positional information, so a positional embedding is added to the input data. Finally, during the training process, the multimodal unified framework aligns multiple modal information by aligning two modal information at a time. The following is a specific method for aligning different modal features:

[0051] For text-image modality alignment, previous methods mostly use detectors (FasterRCNN) to detect objects in the image and then extract the features of the objects as input. However, text usually describes the entire image, not just a single object in the image. Therefore, this paper intends to use the pixel-level features of the image to align with the text. This can preserve the contextual information of the image as much as possible, thereby aligning the global features of the image with the text. Figure 6 shown.

[0052] The text-image feature alignment problem is transformed into a feature matching problem and language model. For the feature matching problem, the global [CLS] token is used for matching. First, two modal data are randomly sampled. For text data, the BERT network is used to extract features, denoted as t = {t1, t2, ..., t n}, where n represents the length of the sentence. For laser scanning image data, CNN neural network is used to extract global features, denoted as f = {f1, f2, ..., f h×w}, where h and w represent the height and width of the feature, respectively. Since global image features contain significant redundant information, inputting all of them into the subsequent transformer network would incur a significant computational burden. Therefore, to improve network computational efficiency and mitigate overfitting to a certain extent, random sampling of global image features is performed.

[0053] In addition, inspired by Oscar, we introduce image labels as anchors to assist in learning the semantic matching between images and text. Similarly, we use BERT to extract the features of image object labels and text, denoted as l = {l1, l2, ..., l C Then, after adding the corresponding modality embedding vector and position embedding vector to the features obtained above, all the vectors are combined together and sent to the transformer network for feature alignment. Finally, the output feature corresponding to the [CLS] token is passed through the fully connected layer to obtain the matching score. And use the two-class loss to drive the training of the entire network. The loss function is:

[0054]

[0055] Where y = 1 indicates a match between the two modalities, and y = 0 indicates a mismatch. σ(·) represents a sigmoid function that maps prediction confidence to probability. Since this matching task does not involve downstream tasks, the [CLS] token for each modality is omitted for simplicity.

[0056] In addition to the feature matching task, the two modal features are further matched by learning a language model. When the semantics of the two modalities match, 15% of the words in a sentence are masked, following the BERT training method. For each masked word, there is an 80% probability of masking it directly, a 10% probability of replacing it with another word, and a 10% probability of leaving it unchanged. The masked locations are then classified, hoping to correctly restore the masked words. If the semantics of the two modalities do not match, this process is omitted.

[0057] Both video and audio can be considered as data with temporal information, so they are described together. For video data, since there is not only spatial information in a single frame but also temporal information between frames, the present invention uses the Transformer encoder to extract spatial and temporal information in order to obtain complete video features, such as Figure 7 shown

[0058] Specifically, let the video sequence be F = {F1, F2, ..., F M}, where M represents the number of video frames. For each frame of image F i , first divide the frame into non-overlapping small blocks Where i represents the i-th frame and N represents the number of blocks. These image blocks are then added with positional encoding information and fed into the Transformer encoder to extract spatial features. The process can be expressed as:

[0059] S i =fs p_encoder (P i +p i )

[0060] Where pi represents the position embedding vector and Si represents the spatial features of the i-th frame. It is worth mentioning that the Transformer encoder parameters of each frame are shared.

[0061] After obtaining the spatial features, the Transformer encoder is used to capture the temporal information. Similar to capturing spatial features, the spatial features are first added to the frame position encoding information, and then sent to the encoder to extract the temporal features. The implementation process is as follows:

[0062] T i =f tm_encoder (S i +fp i )

[0063] Where Ti represents the spatial temporal features of the i-th frame, and fpi represents the frame position embedding vector, which indicates the frame number to which the current feature belongs. Similarly, audio features can be extracted using a similar framework. The difference is that since audio features do not have spatial information, only the temporal information needs to be extracted.

[0064] After obtaining the video / audio features, the subsequent method is the same as the text-image alignment method. Data from the two modalities are randomly extracted, and the text features, video / audio annotation features, and video / audio features are added with the corresponding position embedding vectors and modality embedding vectors. Then, a feature matching task and a language model task are constructed to align the features of each modality.

[0065] Step S3: Input the multimodal fusion feature information into a well-trained artificial intelligence neural network model for depth calculation and estimation judgment to generate depth map data.

[0066] The artificial intelligence neural network model is constructed based on the multi-view depth prediction network MVSNet. The artificial intelligence neural network model includes an image feature extraction module, a cost volume generation module, an initial depth estimation module and a depth map optimization module. The image feature extraction module is used to output N 32-channel feature maps through a 2D convolutional neural network; the cost volume generation module is used to construct a 3D cost volume based on the extracted feature map and camera parameters; the initial depth estimation module is used to perform pixel-by-pixel depth estimation based on the expectation in the depth direction on the basis of constructing the 3D cost volume; the depth map optimization module is used to use a reference image to optimize the predicted depth map based on the residual network.

[0067] Perform deep AI calculations and estimations, update AI neural network algorithm parameters, and continuously optimize the algorithm model. Output a deep computing model that integrates multimodal data on ancient buildings and their historical and cultural heritage, and generate corresponding point cloud data based on this model.

[0068] like Figure 8 As shown, in this embodiment, the training and optimization process of the artificial intelligence model includes:

[0069] ① Image feature extraction:

[0070] The first step of MVSNet is to take N input images Perform feature extraction on it to obtain N deep feature maps These feature maps are used for subsequent dense matching. The feature extraction network uses an 8-layer 2D convolutional neural network, with a convolution stride of 2 in the third and sixth layers, generating feature maps at three scales. As with conventional matching networks, the parameters of the feature extraction network share weights. The output of the 2D convolutional neural network is N 32-channel feature maps.

[0071] ② Cost body generation:

[0072] Construct a 3D cost volume based on the extracted feature map and camera parameters. This step mainly uses

[0073] a. Homography matrix, used to describe the mapping relationship between two planes;

[0074] A homography matrix typically describes the transformation relationship between two images of points on a common plane. Consider a pair of well-matched feature points P1 and P2 in images I1 and I2. These feature points lie on a plane. Let the equation satisfying the plane be:

[0075] nT P+d=0

[0076] Right now:

[0077]

[0078] We can get:

[0079]

[0080] So we get a direct description of the transformation between image coordinates P1 and P2, and record the middle part as H (homography matrix), so we have:

[0081] p2=Hp1

[0082] The homography matrix is ​​a 3*3 matrix.

[0083] b. Differentiable homography

[0084] N feature maps are projected onto several parallel planes under the reference image to form N feature volumes. The planar homography transform determines the coordinate transformation from the feature map to the cost volume at the depth value d. i (d)·x, where ~ represents the projection equation, H i (d) indicates F at depth d i (i=1, 2, ..., N) and the homography transformation matrix of F1. n1 represents the main axis direction of the reference camera. The homography matrix can be expressed as a 3*3 matrix:

[0085]

[0086] The process of homography transformation projection is similar to the classic plane sweep algorithm, the only difference is that the sampling points come from the feature map Instead of images

[0087] c. Cost Metric

[0088] Next, multiple feature bodies Aggregated into a cost volume C. To accommodate any number of view inputs, MVSNet uses a variance-based cost metric M, which is used to measure the similarity between N views. W, H, D, and F represent the width, depth, number of samples, and number of channels of the input image, respectively. Therefore, the size of the feature volume is: The cost indicator defines the mapping relationship M:

[0089]

[0090] d. Cost Volume Regularization

[0091] The initial cost volume calculated from the image feature map may contain noise (due to non-Lambertian surfaces or line of sight occlusion), and smoothing is required to predict the depth map. The regularization step involves refining the cost volume to obtain a probability volume. Inspired by work such as SurfaceNet, a multi-scale 3D CNN network is used for cost volume regularization. The four-scale network is similar to a 3D version of UNet, using an encoder-decoder architecture to aggregate neighborhood information over a large receptive field with relatively low storage and computational costs. To reduce the network's computational cost, after the first 3D convolutional layer, the 32-channel cost volume is reduced to 8 channels, and the number of convolutional layers at each scale is reduced from 3 to 2. The output of the final convolutional layer is a 1-channel volume. Finally, a softmax operation is used depthwise to normalize the probabilities.

[0092] ③ Depth map acquisition

[0093] a. Initial depth estimation

[0094] The simplest way to recover the depth map from a probability volume is to use a "winner takes all" strategy (such as argmax). However, the argmax operation cannot produce sub-pixel estimates, and non-differentiable ones cannot be back-propagated through the network. Therefore, we compute the expectation in the depth direction, which is the weighted sum of all hypothesized depth values:

[0095]

[0096] where P(d) is the estimated probability at depth d. This operation, called a softargmin, is differentiable and produces the result of an argmax. During cost volume construction, depth hypotheses are uniformly sampled within the range [dmin, dmax], resulting in continuous predicted depth values. The output depth map has the same dimensions as the 2D feature map: 1 / 4 the size of the original image in each dimension.

[0097] b. Estimation of probability maps

[0098] The probability distribution along the depth direction can also reflect the quality of depth estimation. Although the multi-scale 3DCNN has a certain regularization ability and can adjust the probability distribution to a unimodal distribution, it should be noted that due to the occurrence of mismatching, the probability distribution of depth is scattered and cannot be concentrated on a peak. Figure 9As shown, the probability map of the reference view (a) is shown in Figure (d), where (a) is the reference image, (b) is the predicted depth map, (c) is the probability distribution, and (d) is the probability map

[0099] C. Depth Map Optimization

[0100] Recovering a depth map from a probability volume is a straightforward and reasonable process, but some issues remain. The larger receptive field during regularization can cause overly smooth boundaries in the reconstructed depth map. These issues also exist in semantic segmentation and image matting. A reference image contains boundary information, so it can be used as a guide to optimize the predicted depth map. Inspired by the image matting algorithm, a deep residual learning network is added to the end of MVSNet.

[0101] The predicted depth map and the resized reference image are concatenated to form a 4-channel input, which is fed into three 32-channel 2D convolutional layers and one 1-channel convolutional layer to learn the depth residual. To enable learning of negative residuals, the final network layer does not use batch normalization layers or ReLU units. Similarly, to avoid bias at specific depth values, the initial depth amplitude is scaled to the range [0, 1] and then converted back after the depth map optimization is complete.

[0102] ④ Loss function considerations

[0103] The loss of both the initial depth value and the optimized depth value should be taken into account. The L1 loss between the real depth map and the estimated depth is used as the training loss. Considering that not every pixel in the real depth map has a value, only valid pixels are considered, that is, pixels with a ground truth label.

[0104] The loss function is defined as follows:

[0105]

[0106] Step S4: obtaining point cloud data based on the depth map data, reconstructing a three-dimensional model based on the obtained point cloud data and three-dimensional reconstruction technology, and completing the virtual reality realization of the ancient building.

[0107] Reconstructing a three-dimensional model based on the acquired point cloud data and the three-dimensional reconstruction technology to generate point cloud data includes:

[0108] Step S40: pre-processing the point cloud data;

[0109] Step S41: Segment the point cloud data to cluster the entire point cloud into multiple point clouds. Each point cloud corresponds to an independent object. A point is selected and a sphere centered on the point is calculated using a kd-tree. All points within the sphere belong to the object. The radius of the sphere is set to a threshold. The remaining points of the object are traversed to segment the point cloud into individual objects.

[0110] Step S42: triangulating the point cloud to obtain the spatial topological structure of the point cloud, and performing mesh rendering to complete 3D modeling to obtain a three-dimensional reconstructed model.

[0111] In order to facilitate subsequent mesh rendering, the point cloud needs to be triangulated in advance, and the algorithm used is usually the convex hull or concave hull algorithm.

[0112] The above steps yield the spatial topology of a point cloud. However, to achieve realistic objects, mesh rendering is required. Mesh rendering primarily involves texture mapping, which involves applying an image from a digital camera to the mesh. This ultimately completes 3D modeling (also known as 3D reconstruction).

[0113] In this embodiment, commonly used methods for pre-processing point cloud data include filtering and denoising, data reduction, data interpolation, etc.

[0114] The present invention completes the 3D reconstruction of ancient buildings based on the multi-view image-based depth (computation) estimation network (MVSNet). In the depth estimation network (MVSNet), feature extraction is first performed on the 2D image to obtain a feature map. Then, a differentiable homography transformation is used to construct a 3D cost volume based on the camera frustum of the reference view. The cost volume is regularized using 3D convolution and regressed to obtain an initial depth map; the initial depth map is optimized by the reference image to obtain the final depth map. As shown below Figure 8 Shown is: MVSNet network architecture diagram.

[0115] The end-to-end multi-view depth prediction network MVSNet, similar to other depth map-based MVS methods, takes as input a reference image and multiple source images and predicts a depth map for the reference image, rather than the entire 3D scene. The key to the network lies in its differentiable homography operation. During the construction of a 3D cost volume from a 2D feature map, the network implicitly encodes the camera parameters. To adapt the network to an arbitrary number of input views, a variance-based metric is proposed that maps multiple feature volumes into a single cost volume. Multi-scale 3D convolutions are applied to the cost volume to regress an initial depth map. Finally, the depth map is refined using the reference image to improve accuracy in boundary regions. This method differs from previous learning-based approaches in two key ways. First, for the purpose of depth map estimation, the 3D cost volume is constructed based on the camera frustum rather than a regular Euclidean space. Second, MVS reconstruction is divided into a per-view depth estimation problem, making large-scale reconstruction possible.

[0116] Step S5: Reconstructing a three-dimensional model based on the point cloud data and three-dimensional reconstruction technology to complete the virtual reality realization of the ancient building.

[0117] like Figure 3 The figure shows an example of the implementation process of the ancient building three-dimensional reconstruction method based on multimodal data fusion. Figure 2 The whole process is Figure 1 The image and concrete description and display.

[0118] The present invention also includes a 3D reconstruction system for ancient buildings based on multimodal data fusion, such as Figure 4 As shown, it specifically includes a multimodal database server, a GPU server, a Web server, a query request interpreter and a client.

[0119] ① Multimodal database server, which mainly stores multimodal data of ancient buildings and their historical culture (including pictures, voice and text, etc.).

[0120] ②GPU server, mainly performs computational processing such as multimodal data fusion depth calculation, point cloud generation and 3D reconstruction.

[0121] ③ Web server and query request interpreter, which are mainly responsible for accepting user requests and normalizing user requests through the query request interpreter.

[0122] ④ Display the 3D reconstruction virtual reality results for the client and user requests.

[0123] 3D reconstruction virtual reality of ancient buildings based on single data carries certain risks and significant limitations. However, this invention utilizes a multimodal data fusion 3D reconstruction method based on artificial intelligence, which not only effectively reconstructs the historical features of ancient buildings but also supplements important historical and cultural details that may have been lost.

[0124] Throughout this specification, references to terms such as "one embodiment," "example," or "specific example" indicate that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of these terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0125] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A method for 3D reconstruction of ancient buildings based on multimodal data fusion, characterized in that: include: Acquire multimodal data on ancient architecture and historical culture, and build a data warehouse to store the acquired multimodal data on ancient architecture and historical culture; Performing semantic alignment processing on the multimodal data of the ancient buildings and historical culture to obtain multimodal fusion feature information; Input the multimodal fusion feature information into a well-trained artificial intelligence neural network model for depth calculation and estimation judgment to generate depth map data; Point cloud data is obtained based on the depth map data, and a three-dimensional model is reconstructed according to the obtained point cloud data and three-dimensional reconstruction technology to complete the virtual reality realization of the ancient building.

2. The method for 3D reconstruction of ancient buildings based on multimodal data fusion according to claim 1, characterized in that: The multimodal data of ancient buildings and historical culture includes three groups of modal data: laser scanning image data, voice data and text data.

3. The method for 3D reconstruction of ancient buildings based on multimodal data fusion according to claim 2, characterized in that: The multimodal data of the ancient buildings and historical culture are semantically aligned to obtain multimodal fusion feature information including: Get any two sets of modal data and use the feature extractor corresponding to each modality to obtain the corresponding serialized modal features; Add a special annotation to each corresponding serialized modal feature, and add a global annotation to the two sets of modal data obtained; Adding a modality type embedding vector and a position embedding vector to each modality feature to obtain an input feature vector, wherein the input feature vector includes the serialized modality features, the annotation features, the modality type embedding vector, and the position embedding vector of each modality data; The input feature vector is input into the transformer network for feature alignment operation. Based on the annotation information, it is determined whether the two sets of modal semantics of the input match. The two sets of modal features are further matched by learning the language model. After completing the matching between the multimodal data of ancient buildings and historical culture, multimodal fusion feature information is obtained.

4. The method for 3D reconstruction of ancient buildings based on multimodal data fusion according to claim 1, characterized in that: The artificial intelligence neural network model is constructed based on the multi-view depth prediction network MVSNet; The artificial intelligence neural network model includes an image feature extraction module, a cost volume generation module, an initial depth estimation module and a depth map optimization module. The image feature extraction module is used to output N 32-channel feature maps through a 2D convolutional neural network; the cost volume generation module is used to construct a 3D cost volume based on the extracted feature map and camera parameters; the initial depth estimation module is used to perform pixel-by-pixel depth estimation based on the expectation in the depth direction on the basis of constructing the 3D cost volume; the depth map optimization module is used to use a reference image to optimize the predicted depth map based on the residual network.

5. The method for 3D reconstruction of ancient buildings based on multimodal data fusion according to claim 1, characterized in that: The three-dimensional model reconstruction based on the obtained point cloud data and the three-dimensional reconstruction technology includes: Preprocess point cloud data; The point cloud data is segmented and clustered into multiple point clouds. Each point cloud corresponds to an independent object. A point is selected and a sphere centered on the point is calculated using a kd tree. All points within the sphere belong to the object. The radius of the sphere is set to a threshold, and the remaining points of the object are traversed to segment the point cloud into individual objects. The point cloud is triangulated to obtain the spatial topological structure of the point cloud, and mesh rendering is performed to complete 3D modeling to obtain a three-dimensional reconstructed model.

Images