Method for reconstructing three-dimensional scene based on oblique photography and multi-source data fusion

Abstract

The application discloses a method for aerial photogrammetry real scene three-dimensional reconstruction based on multi-source data fusion, relates to the technical field of computer vision and geographic spatial information, and comprises the following steps: acquiring first multi-source data of a target area; pre-processing and registering the first multi-source data to obtain second multi-source data; the registration comprises coarse registration and fine registration; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; performing three-dimensional reconstruction on the fused features, optimizing the result of the three-dimensional reconstruction, and performing texture mapping to obtain a three-dimensional model. The application establishes a complete end-to-end three-dimensional reconstruction framework, overcomes the problems of single data source, difficult registration and insufficient feature fusion in the prior art, and realizes high-precision and high-quality three-dimensional scene reconstruction.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and geospatial information, in particular to a method for real scene three-dimensional reconstruction based on multi-source data fusion. BACKGROUND

[0002] With the continuous development of three-dimensional reconstruction technology, multi-source data fusion is increasingly valued in real scene three-dimensional reconstruction. Traditional three-dimensional reconstruction methods mainly rely on a single data source, such as using only oblique photography or laser scanning data, which is difficult to fully capture the geometric and texture information of the scene. In recent years, researchers have begun to try to combine oblique photography, laser scanning and ground real scene multi-source data for three-dimensional reconstruction to improve the accuracy and integrity of the reconstruction results.

[0003] However, the current three-dimensional reconstruction technology still has some shortcomings. First, in terms of multi-source data acquisition, there is a lack of systematic collection strategies, making it difficult to ensure the integrity and complementarity of the data. Second, multi-source data preprocessing and registration face many challenges, and how to effectively handle the noise of different modal data and how to achieve high-precision multi-modal data registration are still difficult problems. Third, existing methods are not effective in multi-source data feature extraction and fusion, and have not fully tapped and integrated the complementary information of different data sources. Fourth, in the three-dimensional reconstruction process, how to generate high-quality point clouds and mesh models based on fused features, and how to achieve fine texture mapping, all need further research. Finally, there is a lack of a unified, end-to-end optimization framework, resulting in error accumulation in each link, affecting the final reconstruction effect.

[0004] Therefore, there is an urgent need for a three-dimensional reconstruction method that can systematically solve the above problems, achieving efficient processing from multi-source data acquisition, preprocessing and registration, feature extraction and fusion to three-dimensional reconstruction optimization, to improve the accuracy, integrity and visual quality of three-dimensional models. SUMMARY

[0005] In view of the problems existing in the prior art method for real scene three-dimensional reconstruction based on multi-source data fusion of oblique photography, the present application is proposed.

[0006] Therefore, the problem to be solved by the present application is that the existing three-dimensional reconstruction technology is difficult to achieve high-precision and high-quality real scene three-dimensional reconstruction, especially in complex scenes.

[0007] To solve the above technical problems, the present application provides the following technical solutions:

[0008] In a first aspect, the embodiments of the present application provide a method for aerial photogrammetry real scene three-dimensional reconstruction based on multi-source data fusion, which comprises: acquiring first multi-source data of a target area; the first multi-source data comprises aerial photogrammetry data, laser scanning data and ground real scene data; pre-processing and registering the first multi-source data to obtain second multi-source data; the registration comprises coarse registration and fine registration; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; performing three-dimensional reconstruction on the fused features, and optimizing and texture mapping the result of the three-dimensional reconstruction to obtain a three-dimensional model.

[0009] As a preferred scheme of the method for aerial photogrammetry real scene three-dimensional reconstruction based on multi-source data fusion, the pre-processing comprises: applying an adaptive filtering algorithm to the first multi-source data respectively to perform noise reduction processing; performing normalization processing on the first multi-source data after the noise reduction processing to make the data ranges of different modalities consistent; and after the pre-processing and before the registration, further comprising extracting multi-modality feature points and descriptors of the pre-processed first multi-source data.

[0010] As a preferred scheme of the method for aerial photogrammetry real scene three-dimensional reconstruction based on multi-source data fusion, the coarse registration of the multi-modality feature points and descriptors is performed using a random sample consensus (RANSAC) algorithm; the coarse registration comprises: setting a maximum number of iterations, an inlier threshold and a minimum consistent set size; randomly selecting a minimum sample set, calculating a transformation model, the transformation model being used to transform source data points from a source coordinate system to a target coordinate system; applying the transformation model to all matching points, calculating the Euclidean distance error between each pair of matching points; judging inliers based on an error threshold, and assigning a reliability weight to each matching point pair according to the reliability difference of the multi-modality features, the reliability weight being calculated based on the relationship between the distance between the matching point descriptors and the overall distance distribution; calculating a weighted consistent set score, i.e. dividing the total weight of the inliers by the total weight of all points; if the score of the current model is higher than the previous best score, updating the best model and the best score, and re-estimating the transformation model using all inliers; after iteration, a transformation matrix of the coarse registration is obtained, comprising a rotation matrix and a translation vector.

[0011] As a preferred scheme of the multi-source data fusion based oblique photography real scene three-dimensional reconstruction method described in the application, the weighted ICP algorithm is used to perform fine registration on the coarse registration result to obtain the registered multi-source data; the fine registration includes: using the transformation matrix of the coarse registration as an initial transformation, setting a maximum number of iterations and a convergence threshold; for each source data point, finding the nearest target point in the target point set to establish a point correspondence relationship; calculating the registration weight of each pair of corresponding points; constructing a weighted error function and minimizing it; calculating the weighted centroids of the source data point set and the target point set to convert the translation problem into a rotation problem; calculating the weighted covariance matrix after centroid removal; singular value decomposition is performed on the covariance matrix to obtain a singular value decomposition result; based on the singular value decomposition result, an optimal rotation matrix is calculated to minimize the squared distance between the source data point set and the target point set; an optimal translation vector is calculated to make the centers of the rotated source data point set and the target point set coincide; check whether the changes of the rotation matrix and the translation vector between the adjacent two iterations are less than the convergence threshold, if the condition is met, stop iteration, and output the fine registration transformation matrix; and convert the source data points to the target coordinate system to complete the alignment of the multi-source data in the same coordinate system.

[0012] As a preferred scheme of the multi-source data fusion based oblique photography real scene three-dimensional reconstruction method described in the application, multi-scale feature extraction and feature fusion are performed on the second multi-source data to obtain fused features, including: using a pre-trained deep convolutional neural network to extract multi-scale features from the second multi-source data; applying a self-attention mechanism and a cross-attention mechanism to fuse the extracted features; and connecting and integrating the self-attention features and the cross-attention features to form the fused features.

[0013] As a preferred scheme of the multi-source data fusion based oblique photography real scene three-dimensional reconstruction method described in the application, the first multi-source data includes oblique photography data, laser scanning data and ground real scene data.

[0014] As a preferred scheme of the multi-source data fusion based oblique photography real scene three-dimensional reconstruction method described in the application, the fused features are subjected to three-dimensional reconstruction, and the result of the three-dimensional reconstruction is optimized and texture mapped to obtain a three-dimensional model, including: performing depth estimation based on the fused features through an MVSNet network to generate dense point cloud data; applying a Poisson surface reconstruction algorithm to the generated point cloud data to construct a three-dimensional grid model and perform local optimization; and performing multi-resolution texture mapping on the optimized three-dimensional model based on the oblique photography data and the ground real scene data to obtain the three-dimensional model.

[0015] In a second aspect, the embodiments of the present application provide a tilt photography real scene three-dimensional reconstruction system based on multi-source data fusion, which comprises: a collection module that acquires first multi-source data of a target area; the first multi-source data comprises tilt photography data, laser scanning data and ground real scene data; a processing module that pre-processes and registers the first multi-source data to obtain second multi-source data; the registration comprises coarse registration and fine registration; a fusion module that extracts multi-scale features and fuses the features of the second multi-source data to obtain fused features; and a construction module that performs three-dimensional reconstruction on the fused features, optimizes the result of the three-dimensional reconstruction and performs texture mapping to obtain a three-dimensional model.

[0016] In a third aspect, the embodiments of the present application provide a computer device comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements any step of the tilt photography real scene three-dimensional reconstruction method based on multi-source data fusion described above when executing the computer program.

[0017] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any step of the tilt photography real scene three-dimensional reconstruction method based on multi-source data fusion described above.

[0018] The present application has the beneficial effects that the present application improves the completeness and accuracy of three-dimensional reconstruction by comprehensively utilizing multi-source data of tilt photography, laser scanning and ground real scene; realizes high-quality data preprocessing by using adaptive filtering denoising and multi-modal data registration technology; realizes effective feature fusion between different modal data by combining self-attention and cross-attention mechanism; constructs a high-quality three-dimensional model based on fused features by using MVSNet network and Poisson surface reconstruction algorithm; proposes a multi-resolution texture mapping strategy based on visual quality to generate seamless texture effect; establishes a complete end-to-end three-dimensional reconstruction framework, and overcomes the problems of single data source, difficult registration and insufficient feature fusion in the prior art, and realizes high-precision and high-quality three-dimensional scene reconstruction. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor. Among them:

[0020] Figure 1 The flowchart of the tilt photography real scene three-dimensional reconstruction method based on multi-source data fusion. DETAILED DESCRIPTION

[0021] In order to make the above objectives, features and advantages of the present application more clear and easily understood, the specific embodiments of the present application will be described in detail below with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work should belong to the protection scope of the present application.

[0022] In the following description, a large number of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar generalizations without departing from the connotation of the present application, therefore the present application is not limited by the specific embodiments disclosed below.

[0023] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is the embodiment alone or selectively exclusive of other embodiments.

[0024] Embodiment 1, refer to Figure 1 For the first embodiment of the present application, the embodiment provides a method for photogrammetry real scene three-dimensional reconstruction based on multi-source data fusion, comprising:

[0025] S1: obtaining first multi-source data of a target area.

[0026] The first multi-source data includes photogrammetry data, laser scanning data and ground real scene data.

[0027] In an optional embodiment, the photogrammetry data can be obtained by a UAV or an airplane carrying multiple tilt cameras, and the target area can be taken from different angles to capture high-resolution orthographic images and oblique images. These images can provide rich texture information and preliminary geometric structure information, providing a comprehensive visual data basis for three-dimensional reconstruction.

[0028] In an optional embodiment, the laser scanning data can be obtained by an airborne or ground laser radar system, and the target area can be scanned with high precision to generate dense point cloud data. These point cloud data can accurately describe the geometric shapes of the terrain and buildings, ensuring the geometric accuracy of the three-dimensional reconstruction results.

[0029] In an optional embodiment, the ground real scene data can be obtained by a ground mobile acquisition device, and high-resolution close-up images and panoramic images can be collected around streets and buildings. These data supplement the information of building bottoms, facade details and street levels, making up for the data blind area of photogrammetry in low-rise buildings and narrow streets.

[0030] The three types of data are distinguished from the traditional single-source data reconstruction method, each has advantages and complements each other. The oblique photography data provides wide visual coverage; the laser scanning data ensures high precision of geometric measurement; the ground real scene data supplements the near-ground details. By comprehensively utilizing these complementary multi-source data, a more complete, accurate and real three-dimensional scene model can be constructed.

[0031] S2: pre-processing and registration of the first multi-source data to obtain second multi-source data.

[0032] In an optional embodiment, the pre-processing includes applying an adaptive filtering algorithm to the first multi-source data, i.e. oblique photography data, laser scanning data and ground real scene data, respectively, for noise reduction processing to improve data quality.

[0033] Specifically, for noise reduction processing, the present application adopts an adaptive filtering algorithm, which is expressed as the following formula:

[0034] Y(x) = X(x) + K(x) x [X(x) - μ(x)];

[0035] Wherein, the adaptive gain coefficient K(x) is defined as:

[0036]

[0037] Wherein, Y(x) is the filtered output data, X(x) is the original input data, μ(x) is the mean value in the local window, σ 2 (x) is the variance in the local window, is a preset maximum variance threshold, and a is an adjustable control parameter.

[0038] It should be noted that the advantage of this algorithm is that it can automatically adjust the filtering strength according to the local statistical characteristics of the data: more detailed information is retained in high-detail areas (such as building edges, feature contours, etc.); stronger smoothing processing is performed in low-detail areas (such as flat ground, uniform wall surface, etc.). For data of different modalities, by adjusting the value of the parameter a, targeted processing can be achieved:

[0039] Oblique photography data: a image = 0.7, window size 5x5 pixels;

[0040] Laser scanning data: a lidar = 0.5, window size 50 adjacent points;

[0041] Ground real scene data: a ground = 0.8, window size 7x7 pixels.

[0042] Through the above parameter settings, the noise can be effectively removed while the unique information of various data is reserved, thereby providing high-quality input for subsequent data registration and feature extraction.

[0043] After the noise reduction processing, normalization processing is performed on the modal data, the numerical range of different types of data is unified to the interval [0, 1], and the scale difference between different modal data is eliminated, thereby laying a foundation for subsequent registration and feature extraction.

[0044] In an optional embodiment, after the preprocessing is completed, although the data is subjected to normalization processing, the oblique photography data, the laser scanning data and the ground real scene data still maintain respective different data modal characteristics. In order to perform subsequent registration, suitable feature points and descriptors need to be extracted according to the characteristics of the modal data:

[0045] For the oblique photography data: SIFT (Scale Invariant Feature Transform) algorithm is used to extract feature points and descriptors;

[0046] For the laser scanning data: ISS (Intrinsic Shape Signature) or Harris 3D corner point detection algorithm is used to extract feature points, and FPFH (Fast Point Feature Histogram) is used to generate descriptors;

[0047] For the ground real scene data: SIFT algorithm is also used to extract feature points and descriptors.

[0048] It should be noted that the above feature points constitute multi-modal feature points.

[0049] In an optional embodiment, the registration includes coarse registration using a RANSAC algorithm and fine registration using an ICP algorithm.

[0050] Specifically, the RANSAC algorithm is used to perform coarse registration on the extracted multi-modal feature points and descriptors, including:

[0051] First, the nearest neighbor algorithm is used to match the extracted multi-modal feature points and descriptors, and then the RANSAC algorithm is used to perform coarse registration, including setting the maximum number of iterations N max , setting the inlier threshold τ, and setting the minimum consistent set size M min ;

[0052] A minimum sample set (at least 3 pairs of matched points) is randomly selected, a transformation model T is calculated, the transformation model is a rigid transformation matrix including a rotation and a translation part, and is used to transform a point in source data from a source coordinate system to a target coordinate system. After the transformation model is applied to all matched points, the error is calculated:

[0053] e j =‖T(p j )-q j ‖2;

[0054] where e j is the Euclidean distance error of the jth matched point pair after applying the transformation model, T(p j ) is the transformed coordinate of the source feature point p j by the transformation model T, q j is the corresponding target feature point coordinate, and ||·||2is the Euclidean distance norm.

[0055] The weighted consensus set score is calculated as:

[0056]

[0057] where S i is the model score obtained in the ith iteration, δ(·) is an indicator function that takes value 1 (indicating that the point pair is considered as an inlier) if the condition e j < τ is true, otherwise it takes value 0; w j is the reliability weight of the jth matched point pair, and M is the total number of feature point matches.

[0058] If the model score S i of the current iteration is greater than the current best score S best , the best model T best and the best score S best are updated, and the final transformation model is re-estimated using all matched point pairs that are determined as inliers:

[0059]

[0060] where T best is the updated best transformation model, argmin denotes the transformation matrix T that minimizes the expression behind it, and I denotes the index set of all inliers, i.e., the set of indices j of all point pairs that satisfy the condition e j < τ.

[0061] Based on the reliability difference of multi-modal features, the reliability weight of each matched point pair is calculated as:

[0062]

[0063] where w j is the reliability weight of the jth matched point pair, d(p j , q j ) is the distance between the descriptors (e.g., Euclidean distance for SIFT descriptors, and Chi-square distance for FPFH descriptors), and μ and σ are the mean and standard deviation of the distances of all matched point pairs, respectively. This weight design gives higher weight to the matched point pairs whose descriptor distance is close to the mean of the overall distribution, thus reducing the impact of outliers.

[0064] After the above RANSAC iteration, the final transformation matrix T of rough registration is obtained coarse The matrix can be expressed as:

[0065]

[0066] wherein, is a rotation matrix, is a translation vector. By applying this transformation matrix, the source data points can be transformed to the target data coordinate system:

[0067] p trans =R·p+t;

[0068] wherein, p trans is the transformed point coordinate, and p is the original point coordinate in the source data.

[0069] It should be noted that the RANSAC algorithm has the following advantages: 1. By introducing the credibility weight based on the descriptor distance, the uncertainty in the multi-modal data is better handled; 2. For different types of feature points and descriptors (such as image SIFT and point cloud FPFH), the weight can be adaptively assigned; 3. By re-estimating the inlier model, the accuracy of the final transformation matrix is improved; 4. For different modal data, the parameters can be adjusted to obtain the best effect:

[0070] Oblique photography and ground real scene matching: τ img =2.0 pixels;

[0071] Laser scanning point cloud internal matching: τ lidar =0.05 meters;

[0072] Image and point cloud cross-modal matching: τ cross =1.5 pixels.

[0073] Further, the weighted ICP algorithm is used to fine register the rough registration result to obtain the registered multi-source data.

[0074] The rough registration result transformation matrix T coarse is used as the initial transformation, the maximum number of iterations K max is set, and the convergence threshold ∈ is set; for each source point x i , the nearest target point y i is found:

[0075] y i =argmin y∈Y ‖R k-1 x i +t k-1 -y‖2;

[0076] wherein, yi Let Y represent the nearest target point found, and R represent the set of target points. k-1 and t k-1 These are the rotation matrix and translation vector for the (k-1)th iteration, respectively, and ||·|2 represents the Euclidean distance norm. This step establishes point correspondences by finding the nearest neighbor of each source point in the target point set under the current transformation.

[0077] The registration weights for each pair of corresponding points are calculated and expressed as follows:

[0078]

[0079] Where, ω i It is the registration weight of the i-th pair of points. This weight formula is designed so that closer point pairs receive higher weights during the registration process, thereby reducing the impact of outliers and mismatches and improving the algorithm's robustness to noise and outliers.

[0080] The function that minimizes the error is represented as:

[0081]

[0082] Among them, E icp (R k ,t k R is the weighted error function that needs to be minimized in the current iteration, where P is the total number of matched point pairs, and R is the weighted error function that needs to be minimized in the current iteration. k and t k It is the rotation matrix and translation vector sought in the current iteration.

[0083] To solve for the error function using singular value decomposition, we first calculate the weighted centroid, which is expressed as:

[0084]

[0085] in, and These are the centroids of the weighted source and target point sets, respectively. By using these centroids, the translation problem can be transformed into a problem that only solves the rotation problem, thus simplifying the calculation process.

[0086] The covariance matrix is ​​calculated and expressed as:

[0087]

[0088] Among them, H cov It is the weighted covariance matrix of the source and target point sets after centroid removal, reflecting the spatial relationship between the two point sets. This matrix contains correlation information between point pairs and is key to solving for the optimal rotation.

[0089] The covariance matrix is ​​decomposed using SVD and expressed as follows:

[0090]

[0091] where U svd and V svd are orthogonal matrices, and Σ svd is a diagonal matrix containing singular values. The covariance matrix H cov is decomposed into the product of three matrices to facilitate subsequent computation of the optimal rotation matrix.

[0092] The optimal rotation matrix is solved such that the sum of squared distances between the source point cloud and the target point cloud is minimized, denoted as:

[0093]

[0094] The optimal translation vector is calculated such that the centers of the rotated source point cloud and the target point cloud coincide, denoted as:

[0095]

[0096] By finding the best rigid transformation (rotation + translation), the transformed source point cloud and the target point cloud are made to coincide as much as possible, ensuring the correctness of the solution, and minimizing the weighted sum of squared distances between point pairs.

[0097] If ‖t k -t k-1 ‖2< ε and ‖R k -R k-1 ‖ F < ε, stop iteration and output the fine registration transformation matrix T fine = [R k |t k ]. Where ‖t k -t k-1 ‖2 represents the Euclidean distance of the translation vector between adjacent iterations, ‖R k -R k-1 ‖ F represents the Frobenius norm, calculated as the square root of the sum of squares of all elements of the matrix, used to measure the difference between two rotation matrices.

[0098] It should be noted that the weighted ICP algorithm has the following advantages compared with the traditional ICP algorithm:

[0099] 1. By introducing a distance inversely proportional adaptive weight mechanism, the influence of outliers and incorrect matches on the final result is effectively reduced;

[0100] 2. Using SVD decomposition to directly solve the optimal rotation matrix, avoiding the possible local optimal problem of iterative solution;

[0101] 3. For the characteristics of multi-modal data, the following optimizations are made:

[0102] For the region with varying point cloud density: adopt adaptive weight mechanism to reduce the deviation caused by uneven density;

[0103] For multi-source data with low overlap: set a reasonable iteration termination condition ∈ overlap = 0.001;

[0104] For data of different scales: adopt normalization processing strategy, use ∈ image = 0.0005 for oblique photography data and ground reality data, and use ∈ lidar = 0.005 for laser point cloud data.

[0105] The fine registration algorithm combines the initial transformation provided by the coarse registration, continuously optimizes the transformation parameters through multiple iterations, and finally obtains a high-precision registration result T fine , laying a foundation for subsequent multi-source data fusion. The transformed data points can be represented as:

[0106] x result = R k ·x+t k ;

[0107] Where x result is the final transformed point coordinates, realizing the accurate alignment of multi-source data in the same coordinate system.

[0108] Preferably, the present application can effectively handle the registration problem of multi-source multi-dimensional data through the cooperation of coarse registration and fine registration, and improve the accuracy and reliability of the final result. Through the RANSAC algorithm, the abnormal values and mismatching in multi-modal data can be effectively handled, and the robustness of the algorithm is improved through the weighted scoring mechanism. The weighted ICP algorithm further optimizes the registration result, considers the reliability of different data sources, and realizes high-precision alignment.

[0109] S3: Multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features.

[0110] For the second multi-source data after preprocessing and registration, a pre-trained deep convolutional neural network is used to extract multi-scale features from oblique photography data, laser scanning data and ground reality data, so as to capture rich information from low-level texture to high-level semantics.

[0111] In an optional embodiment, multi-scale feature extraction can use pre-trained networks such as VGG16 or ResNet50 to extract feature maps of different levels. Low-level feature maps contain rich edge and texture information, and high-level feature maps contain more abstract semantic information.

[0112] The extracted features are applied with a self-attention mechanism and a cross-attention mechanism. The self-attention mechanism is used to highlight important features within the same data source, and the cross-attention mechanism is used to capture complementary information between different modalities of data, including the interaction between oblique photography and laser scanning, oblique photography and ground reality, and laser scanning and ground reality.

[0113] In an optional embodiment, the self-attention and cross-attention calculations can be implemented using the multi-head attention mechanism in the Transformer architecture. The features are converted into queries, keys, and values through linear projection, the attention weight distribution is calculated, and then the weighted feature representation is obtained.

[0114] The obtained self-attention features and cross-attention features are connected and integrated to form fusion features, which are used for subsequent three-dimensional reconstruction. This fusion method fully utilizes the texture information of oblique photography data, the geometric accuracy of laser scanning data, and the close-range details of ground reality data, laying a foundation for constructing high-quality three-dimensional models.

[0115] In an optional embodiment, all feature maps can be connected in the channel dimension using the feature concatenation operation, and the dimension can be reduced through a 1x1 convolutional layer to eliminate redundant information and obtain a compact fusion feature representation.

[0116] S4: Perform three-dimensional reconstruction on the fusion features, optimize and texture map the results of the three-dimensional reconstruction, and obtain a three-dimensional model.

[0117] Based on the fusion features, perform depth estimation through the MVSNet network to generate dense point cloud data.

[0118] In an optional embodiment, the MVSNet (Multi-View Stereo Network) algorithm can be used. This network introduces a multi-modal data constraint term, generates high-precision depth maps using fusion features, and converts the depth maps into point cloud data through a back-projection method, providing a basis for subsequent reconstruction.

[0119] Apply the Poisson surface reconstruction algorithm to the generated point cloud data to construct a three-dimensional mesh model and perform local optimization.

[0120] In an optional embodiment, the surface can be reconstructed through the Poisson equation, a closed mesh can be generated based on the vector field defined by the point cloud, and then the mesh can be simplified and smoothed to eliminate noise and artifacts, improving the geometric accuracy and topological rationality of the model.

[0121] Based on oblique photography data and ground reality data, perform multi-resolution texture mapping on the optimized three-dimensional model to obtain a three-dimensional model.

[0122] In an optional embodiment, a visual quality-based texture selection algorithm can be employed to select the best texture image for each triangular patch according to the angle between the line of sight and the patch normal, image resolution, camera distance, and other factors, and to fuse the textures in the overlapping areas using a weighted average method to generate a high-quality, seamless texture mapping result.

[0123] To sum up, the present application improves the integrity and accuracy of three-dimensional reconstruction by comprehensively utilizing oblique photography, laser scanning and ground real scene multi-source data; realizes high-quality data preprocessing by using adaptive filtering and noise reduction and multi-modal data registration technology; realizes effective feature fusion between different modal data by combining self-attention and cross-attention mechanism; constructs high-quality three-dimensional model based on fused features by using MVSNet network and Poisson surface reconstruction algorithm; proposes a multi-resolution texture mapping strategy based on visual quality to generate seamless texture effect; establishes a complete end-to-end three-dimensional reconstruction framework, which overcomes the problems of single data source, difficult registration and insufficient feature fusion in the prior art, and realizes high-precision and high-quality three-dimensional scene reconstruction.

[0124] Embodiment 2, on the basis of the first embodiment, the present embodiment further provides an oblique photography real scene three-dimensional reconstruction system based on multi-source data fusion, comprising:

[0125] A collection module is configured to acquire first multi-source data of a target area; the first multi-source data includes oblique photography data, laser scanning data and ground real scene data;

[0126] A processing module is configured to pre-process and register the first multi-source data to obtain second multi-source data; the registration includes coarse registration and fine registration;

[0127] A fusion module is configured to extract multi-scale features and fuse the features of the second multi-source data to obtain fused features;

[0128] A construction module is configured to perform three-dimensional reconstruction on the fused features, optimize and texture map the results of the three-dimensional reconstruction, and obtain a three-dimensional model.

[0129] The present embodiment further provides a computer device suitable for the oblique photography real scene three-dimensional reconstruction method based on multi-source data fusion, comprising a memory and a processor; the memory is configured to store computer executable instructions, and the processor is configured to execute the computer executable instructions to realize the oblique photography real scene three-dimensional reconstruction method based on multi-source data fusion as proposed in the above embodiments.

[0130] The computer device can be a terminal, and the computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is configured to perform wired or wireless communication with an external terminal. The wireless communication can be achieved through WIFI, an operator network, NFC (Near Field Communication) or other technologies. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0131] The embodiment also provides a storage medium having a computer program stored thereon, and the computer program is executed by a processor to implement the method for realizing oblique photography real scene three-dimensional reconstruction based on multi-source data fusion.

[0132] The storage medium provided by the embodiment belongs to the same inventive concept as the data storage method provided by the above embodiment, and the technical details not described in detail in the embodiment can be referred to the above embodiment, and the embodiment has the same beneficial effects as the above embodiment.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and all modifications or replacements should be included in the scope of the claims of the present application.

Claims

1. A method for oblique photography real scene three-dimensional reconstruction based on multi-source data fusion, characterized in that: The method comprises the following steps: acquiring first multi-source data of a target region; preprocessing and registering the first multi-source data to obtain second multi-source data; the registration comprises coarse registration and fine registration; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; performing three-dimensional reconstruction on the fused features, and optimizing and texture-mapping the result of the three-dimensional reconstruction to obtain a three-dimensional model; after the preprocessing and before the registration, the method further comprises extracting multi-modal feature points and descriptors of the preprocessed first multi-source data; performing coarse registration on the multi-modal feature points and descriptors by using a random sample consensus (RANSAC) algorithm; the coarse registration comprises the following steps: setting a maximum number of iterations, an inlier threshold, and a minimum consistent set size; randomly selecting a minimum sample set, and calculating a transformation model, which is used to transform source data points from a source coordinate system to a target coordinate system; applying the transformation model to all matching points, and calculating the Euclidean distance error between each pair of matching points; judging inliers based on an error threshold, and assigning a reliability weight to each pair of matching points based on the reliability difference of the multi-modal features; the reliability weight is calculated based on the relationship between the distance between the matching point descriptors and the overall distance distribution; calculating the reliability weight of each pair of matching points: ； wherein, denotes the a confidence weight for a matching point, is the distance between the descriptors, and are the mean and standard deviation of the descriptor distances for all matching points, respectively. calculating a weighted consistent set score, i.e., dividing the total weight of the inliers by the total weight of all points; if the score of the current model is higher than the previous best score, updating the best model and the best score, and re-estimating the transformation model using all inliers; after iteration, a transformation matrix of the coarse registration is obtained, which comprises a rotation matrix and a translation vector.

2. The multi-source data fusion based oblique photography real scene three-dimensional reconstruction method according to claim 1, characterized in that: The preprocessing comprises the following steps: applying an adaptive filtering algorithm to the first multi-source data respectively to perform noise reduction processing; performing normalization processing on the first multi-source data after the noise reduction processing to make the data ranges of different modalities consistent.

3. The multi-source data fusion based oblique photography real scene three-dimensional reconstruction method according to claim 2, characterized in that: performing fine registration on the coarse registration result by using a weighted ICP algorithm to obtain registered multi-source data; the fine registration comprises the following steps: using the transformation matrix of the coarse registration as an initial transformation, setting a maximum number of iterations and a convergence threshold; for each source data point, finding the nearest target point in the target point set to establish a point correspondence relationship; calculating the registration weight of each pair of corresponding points; constructing a weighted error function and minimizing it; calculating the weighted centroids of the source data point set and the target point set to convert the translation problem into a rotation problem; calculating the weighted covariance matrix after the centroids are removed; performing singular value decomposition on the covariance matrix to obtain a singular value decomposition result; calculating an optimal rotation matrix based on the singular value decomposition result to minimize the squared distance between the source data point set and the target point set; calculating an optimal translation vector to make the centers of the source data point set and the target point set coincide after rotation; checking whether the changes of the rotation matrix and the translation vector between two adjacent iterations are less than the convergence threshold; if the condition is met, stopping iteration and outputting the transformation matrix of the fine registration; converting the source data points to the target coordinate system to complete the alignment of the multi-source data in the same coordinate system. The formula for calculating the registration weight of each pair of corresponding points is as follows: ； wherein, is the first registration weight for the point, denotes the found closest target point, and are the rotation matrix and translation vector, respectively, of the iteration, is the source point.

4. The multi-source data fusion based oblique photography real scene three-dimensional reconstruction method of claim 3, wherein: performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features comprises the following steps: extracting multi-scale features from the second multi-source data using a pre-trained deep convolutional neural network; applying a self-attention mechanism and a cross-attention mechanism to the extracted features for fusion; connecting and integrating the self-attention features and the cross-attention features to form fusion features.

5. The multi-source data fusion based oblique photography real scene three-dimensional reconstruction method according to claim 4, characterized in that: The first multi-source data includes oblique photography data, laser scanning data, and ground real scene data.

6. The multi-source data fusion based oblique photography real scene 3D reconstruction method of claim 5, wherein: The fusion features are subjected to three-dimensional reconstruction, and the results of the three-dimensional reconstruction are optimized and texture-mapped to obtain a three-dimensional model, including: based on the fusion features, performing depth estimation through an MVSNet network to generate dense point cloud data; applying a Poisson surface reconstruction algorithm to the generated point cloud data to construct a three-dimensional mesh model and perform local optimization; based on the oblique photography data and the ground real scene data, performing multi-resolution texture mapping on the optimized three-dimensional model to obtain a three-dimensional model.

7. The system for real scene 3D reconstruction based on multi-source data fusion according to any one of claims 1-6, characterized in that: including, a collection module that obtains first multi-source data of a target area; The first multi-source data includes oblique photography data, laser scanning data, and ground real scene data. A processing module pre-processes and registers the first multi-source data to obtain second multi-source data; the registration includes coarse registration and fine registration. A fusion module extracts multi-scale features and fuses the features from the second multi-source data to obtain fusion features. A construction module performs three-dimensional reconstruction on the fusion features, and optimizes and texture-maps the results of the three-dimensional reconstruction to obtain a three-dimensional model.

8. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that: The processor executes the computer program to implement the steps of the oblique photography real scene three-dimensional reconstruction method based on multi-source data fusion according to any one of claims 1-6.

9. A computer readable storage medium having stored thereon a computer program, characterized in that: The computer program is executed by the processor to implement the steps of the oblique photography real scene three-dimensional reconstruction method based on multi-source data fusion according to any one of claims 1-6.

Images