Oblique photography live-action three-dimensional reconstruction method based on multi-source data fusion

Abstract

The invention discloses an oblique photography live-action three-dimensional reconstruction method based on multi-source data fusion, and relates to the technical field of computer vision and geographic space information, and the method comprises the steps: obtaining first multi-source data of a target region; preprocessing and registering the first multi-source data to obtain second multi-source data; the registration comprises coarse registration and fine registration; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fusion features; and performing three-dimensional reconstruction on the fused features, and performing optimization and texture mapping on a three-dimensional reconstruction result to obtain a three-dimensional model. According to the method, a complete end-to-end three-dimensional reconstruction framework is established, the problems of single data source, difficult registration, insufficient feature fusion and the like in the prior art are solved, and high-precision and high-quality three-dimensional scene reconstruction is realized.

Description

Technical Field

[0001] The present application relates to the fields of computer vision and geospatial information technology, and in particular to a method for three-dimensional reconstruction of real scenes using oblique photography based on multi-source data fusion. Background Art

[0002] With the continuous development of 3D reconstruction technology, multi-source data fusion is gaining increasing attention in the field of real-world 3D reconstruction. Traditional 3D reconstruction methods rely primarily on a single data source, such as oblique photography or laser scanning data, which makes it difficult to fully capture the scene's geometry and texture information. In recent years, researchers have begun to experiment with combining multi-source data such as oblique photography, laser scanning, and ground truth for 3D reconstruction to improve the accuracy and completeness of the reconstruction results.

[0003] However, the current 3D reconstruction technology still has some shortcomings. First, in terms of multi-source data acquisition, there is a lack of systematic collection strategies, and it is difficult to ensure the integrity and complementarity of the data. Secondly, the preprocessing and registration of multi-source data face many challenges. How to effectively deal with the noise of different modal data and how to achieve high-precision multi-modal data registration are still difficult problems. Third, the existing methods are not effective in extracting and fusion of multi-source data features, and fail to fully explore and integrate the complementary information of different data sources. Fourth, in the 3D reconstruction process, how to generate high-quality point clouds and mesh models based on fused features, and how to achieve fine texture mapping, need further research. Finally, the lack of a unified, end-to-end optimization framework leads to the accumulation of errors in each link, affecting the final reconstruction effect.

[0004] Therefore, there is an urgent need for a 3D reconstruction method that can systematically solve the above problems and realize efficient processing of the entire process from multi-source data acquisition, preprocessing and registration, feature extraction and fusion to 3D reconstruction optimization, so as to improve the accuracy, integrity and visual quality of the 3D model. Summary of the Invention

[0005] In view of the problems existing in the existing methods for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion, this application is proposed.

[0006] Therefore, the problem to be solved by this application is that the existing 3D reconstruction technology is difficult to achieve high-precision, high-quality 3D reconstruction of real scenes, especially the adaptability and performance in complex scenes are not ideal.

[0007] To solve the above technical problems, this application provides the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion, which includes obtaining first multi-source data of a target area; the first multi-source data includes oblique photography data, laser scanning data and ground real scene data; preprocessing and aligning the first multi-source data to obtain second multi-source data; the alignment includes coarse alignment and fine alignment; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; performing three-dimensional reconstruction on the fused features, optimizing and texture mapping the results of the three-dimensional reconstruction to obtain a three-dimensional model.

[0009] As a preferred solution of the method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion described in the present application, the preprocessing includes: applying an adaptive filtering algorithm to the first multi-source data for noise reduction processing; normalizing the first multi-source data after noise reduction processing to make the data ranges of different modalities consistent; after the preprocessing and before alignment, it also includes extracting multimodal feature points and descriptors of the preprocessed first multi-source data.

[0010] As a preferred solution of the method for 3D reconstruction of real scenes based on oblique photography based on multi-source data fusion described in the present application, the method comprises: using a random sampling consistency RANSAC algorithm to perform coarse alignment on the multimodal feature points and descriptors; the coarse alignment includes: setting a maximum number of iterations, an inlier threshold and a minimum consistent set size; randomly selecting a minimum sample set and calculating a transformation model, wherein the transformation model is used to transform the source data points from the source coordinate system to the target coordinate system; applying the transformation model to all matching points and calculating the Euclidean distance error between each pair of matching points; judging the inliers based on the error threshold, and assigning a credibility weight to each matching point pair based on the reliability difference of the multimodal features, wherein the credibility weight is calculated based on the relationship between the distance between the matching point descriptors and the overall distance distribution; calculating a weighted consistent set score, i.e. dividing the sum of the weights of the inliers by the sum of the weights of all points; if the score of the current model is higher than the previous best score, updating the best model and the best score, and re-estimating the transformation model using all inliers; after iteration, obtaining a transformation matrix for coarse alignment, which includes a rotation matrix and a translation vector.

[0011] As a preferred solution of the method for 3D reconstruction of real scenes based on oblique photography of the present application based on multi-source data fusion, wherein: a weighted ICP algorithm is used to perform fine registration on the coarse registration result to obtain the multi-source data after registration; the fine registration includes: using the transformation matrix of the coarse registration as the initial transformation, setting the maximum number of iterations and the convergence threshold; for each source data point, finding the nearest target point in the target point set and establishing a point correspondence relationship; calculating the registration weight of each pair of corresponding points; constructing a weighted error function and minimizing it; calculating the weighted centroid of the source data point set and the target point set to convert the translation problem into a rotation problem Problem; calculate the weighted covariance matrix after removing the centroid; perform singular value decomposition on the covariance matrix to obtain the singular value decomposition result; calculate the optimal rotation matrix based on the singular value decomposition result to minimize the sum of the square distances between the source data point set and the target point set; calculate the optimal translation vector to make the center of the rotated source data point set coincide with the center of the target point set; check whether the change of the rotation matrix and the translation vector between two adjacent iterations is less than the convergence threshold. If the condition is met, stop the iteration and output the precisely aligned transformation matrix; convert the source data points to the target coordinate system to complete the alignment of multi-source data in the same coordinate system.

[0012] As a preferred solution of the method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion described in the present application, wherein: multi-scale feature extraction and feature fusion are performed on the second multi-source data to obtain fused features, including: using a pre-trained deep convolutional neural network to extract multi-scale features from the second multi-source data; applying a self-attention mechanism and a cross-attention mechanism to fuse the extracted features; and connecting and integrating the self-attention features and the cross-attention features to form fused features.

[0013] As a preferred solution of the method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion described in the present application, the first multi-source data includes oblique photography data, laser scanning data and ground real scene data.

[0014] As a preferred solution of the method for three-dimensional reconstruction of real scenes from oblique photography based on multi-source data fusion described in the present application, the method comprises: performing three-dimensional reconstruction on the fused features, optimizing and texture mapping the results of the three-dimensional reconstruction, and obtaining a three-dimensional model, including: performing depth estimation through the MVSNet network based on the fused features to generate dense point cloud data; applying the Poisson surface reconstruction algorithm to the generated point cloud data to construct a three-dimensional mesh model and perform local optimization; and performing multi-resolution texture mapping on the optimized three-dimensional model based on the oblique photography data and the ground real scene data to obtain a three-dimensional model.

[0015] In the second aspect, an embodiment of the present application provides an oblique photography real-scene three-dimensional reconstruction system based on multi-source data fusion, which includes: a collection module, which acquires first multi-source data of the target area; the first multi-source data includes oblique photography data, laser scanning data and ground real-scene data; a processing module, which preprocesses and aligns the first multi-source data to obtain second multi-source data; the alignment includes coarse alignment and fine alignment; a fusion module, which performs multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; a construction module, which performs three-dimensional reconstruction on the fused features, optimizes and texture maps the results of the three-dimensional reconstruction, and obtains a three-dimensional model.

[0016] In a third aspect, an embodiment of the present application provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: when the processor executes the computer program, any step of the above-mentioned method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion is implemented.

[0017] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, wherein: when the computer program is executed by a processor, any step of the above-mentioned method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion is implemented.

[0018] A 3D image could be a very accurate 3D reconstruction of a scene, but it doesn't have to be a pixelated image, so it has to be a really good 3D image with high quality. A 3D image could be a very accurate 3D image with high quality. It doesn't have to be a very hard 3D image, but it does have to be a very well-made 3D image with high quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be derived from these drawings without inventive effort. Among them:

[0020] Figure 1 This is a flowchart of the method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion. DETAILED DESCRIPTION

[0021] To make the above-mentioned purposes, features, and advantages of this application more clearly understood, the following detailed description of the specific embodiments of this application is given in conjunction with the accompanying drawings. It is obvious that the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary persons in this field without creative work should fall within the scope of protection of this application.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application may also be implemented in other ways different from those described herein. Those skilled in the art may make similar generalizations without violating the connotation of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0023] Secondly, the term "one embodiment" or "embodiment" herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present application. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to the same embodiment, nor does it refer to a separate or selective embodiment that is mutually exclusive with other embodiments.

[0024] Example 1, reference Figure 1 , which is the first embodiment of this application, provides a method for 3D reconstruction of real scenes using oblique photography based on multi-source data fusion, including:

[0025] S1: Acquire the first multi-source data of the target area.

[0026] The first multi-source data includes oblique photography data, laser scanning data and ground truth data.

[0027] In an alternative embodiment, oblique photography data can be acquired using drones or aircraft equipped with multiple oblique cameras. This allows for aerial photography of the target area from different angles, capturing high-resolution orthophotos and oblique images. These images provide rich texture information and preliminary geometric structure information, providing a comprehensive visual data foundation for 3D reconstruction.

[0028] In an alternative embodiment, laser scanning data can be acquired through an airborne or ground-based LiDAR system, which performs a high-precision 3D scan of the target area and generates dense point cloud data. This point cloud data accurately describes the geometry of terrain and buildings, ensuring the geometric accuracy of the 3D reconstruction results.

[0029] In an alternative embodiment, ground truth data can be acquired using ground-based mobile acquisition equipment, collecting high-resolution close-up and panoramic images around streets and buildings. This data complements building bases, facade details, and street-level information, addressing the data gaps of oblique photography around low-rise buildings and narrow streets.

[0030] These three types of data, distinct from traditional single-source data reconstruction methods, each offer their own advantages and complement each other. Oblique photography provides wide-area visual coverage; laser scanning ensures high-precision geometric measurements; and ground truth data provides additional near-surface detail. By combining these complementary multi-source data, a more complete, accurate, and realistic 3D scene model can be constructed.

[0031] S2: Preprocess and register the first multi-source data to obtain second multi-source data.

[0032] In an optional embodiment, the preprocessing includes applying an adaptive filtering algorithm to the first multi-source data, namely, the oblique photography data, the laser scanning data, and the ground truth data, to perform noise reduction processing to improve the data quality.

[0033] Specifically, for noise reduction processing, this application adopts an adaptive filtering algorithm, which is expressed as the following formula:

[0034] Y(x)=X(x)+K(x)×[X(x)-μ(x)];

[0035] Among them, the adaptive gain coefficient K(x) is defined as:

[0036]

[0037] Among them, Y(x) is the output data after filtering, X(x) is the original input data, μ(x) is the mean value in the local window, σ 2 (x) is the variance within the local window, is the preset maximum variance threshold, and α is an adjustable control parameter.

[0038] It should be noted that the advantage of this algorithm is that it can automatically adjust the filtering strength based on the local statistical characteristics of the data: it retains more detailed information in high-detail areas (such as building edges and ground contours); and performs stronger smoothing in low-detail areas (such as flat ground and uniform walls). For data of different modes, targeted processing can be achieved by adjusting the value of parameter α:

[0039] Oblique photography data: α image =0.7, window size 5×5 pixels;

[0040] Laser scanning data: α lidar =0.5, the window size is 50 adjacent points;

[0041] Ground truth data: α ground =0.8, window size 7×7 pixels.

[0042] By setting the above parameters, it is possible to effectively remove noise while retaining the unique information of various types of data, providing high-quality input for subsequent data alignment and feature extraction.

[0043] After denoising, each modal data is normalized to unify the numerical range of different types of data into the interval [0,1], eliminating the scale differences between different modal data and laying the foundation for subsequent registration and feature extraction.

[0044] In an optional embodiment, after preprocessing, although the data has been normalized, the oblique photography, laser scanning, and ground truth data still retain their respective modal characteristics. In order to perform subsequent registration, it is necessary to extract appropriate feature points and descriptors based on the characteristics of each modality:

[0045] For oblique photography data: use SIFT (Scale Invariant Feature Transform) algorithm to extract feature points and descriptors;

[0046] For laser scanning data: ISS (Intrinsic Shape Features) or Harris 3D corner detection algorithm is used to extract feature points, and FPFH (Fast Point Feature Histogram) is used to generate descriptors;

[0047] For ground truth data: SIFT algorithm is also used to extract feature points and descriptors.

[0048] It should be noted that the above-mentioned feature points constitute multimodal feature points.

[0049] In an optional embodiment, the registration includes performing coarse registration using a RANSAC algorithm and performing fine registration using an ICP algorithm.

[0050] Specifically, the RANSAC algorithm is used to perform coarse registration on the extracted multimodal feature points and descriptors, including:

[0051] First, the nearest neighbor algorithm is used to match the extracted multimodal feature points and descriptors, and then the RANSAC algorithm is used for coarse registration, including setting the maximum number of iterations N. max , set the inlier threshold τ, set the minimum consistent set size M min ;

[0052] Randomly select a minimum sample set (at least 3 pairs of matching points) and calculate the transformation model T, which is a rigid body transformation matrix containing rotation and translation parts, used to transform the points in the source data from the source coordinate system to the target coordinate system. After applying the transformation model to all matching points, calculate the error:

[0053] e j =‖T(p j )-q j ‖2;

[0054] Among them, e j represents the Euclidean distance error of the jth pair of matching points after applying the transformation model, T(p j ) represents the source feature point p j The coordinates after transformation by the transformation model T, q j are the corresponding target feature point coordinates, and ‖·‖2 represents the Euclidean distance norm.

[0055] Compute the weighted consensus set score:

[0056]

[0057] Among them, S i represents the model score obtained at the i-th iteration, δ(·) is the indicator function, when the condition e j <τ takes the value 1 when it is true (indicating that the point pair is considered an interior point), otherwise it takes the value 0; w j is the credibility weight of the jth matching point pair, and M is the total number of feature point matching pairs.

[0058] If the model score of the current iteration is S i Greater than the current best score S best , then update the best model T best and best score S best , and re-estimate the final transformation model using all matching pairs that are judged to be inliers:

[0059]

[0060] Among them, T best represents the updated optimal transformation model, argmin represents the transformation matrix T that minimizes the following expression, and I represents the index set of all internal points, that is, the condition e is satisfied. j The set of indices j for all pairs of points <τ.

[0061] Based on the reliability differences of multimodal features, the credibility weight of each matching point pair is calculated:

[0062]

[0063] Among them, w j represents the credibility weight of the jth pair of matching points, d(p j ,q j ) is the distance between descriptors (for example, Euclidean distance is used for SIFT descriptors and chi-square distance is used for FPFH descriptors), μ and σ are the mean and standard deviation of the distances between all matching points and descriptors, respectively. This weighting design gives higher weights to matching points whose descriptor distance is close to the mean of the overall distribution, thereby reducing the impact of outliers.

[0064] After the above RANSAC iterations, the final rough registration transformation matrix T is obtained. coarse , the matrix can be expressed as:

[0065]

[0066] in, is the rotation matrix, Is the translation vector. Applying this transformation matrix can transform the source data point to the target data coordinate system:

[0067] p trans =R·p+t;

[0068] Among them, p trans are the transformed point coordinates, and p are the original point coordinates in the source data.

[0069] It should be noted that the RANSAC algorithm has the following advantages: 1. By introducing a credibility weight based on the distance between descriptors, it can better handle the uncertainty in multimodal data; 2. It can adaptively assign weights to different types of feature points and descriptors (such as image SIFT and point cloud FPFH); 3. It improves the accuracy of the final transformation matrix by re-estimating the interior point model; 4. For different modal data, the parameters can be adjusted to obtain the best effect:

[0070] Matching oblique photography with ground truth: τ img =2.0 pixels;

[0071] Laser scanning point cloud interior matching: τ lidar =0.05 m;

[0072] Cross-modal matching between images and point clouds: τ cross =1.5 pixels.

[0073] Furthermore, the weighted ICP algorithm is used to perform fine registration on the coarse registration results to obtain the registered multi-source data.

[0074] Use the transformation matrix T resulting from the coarse registration coarse As the initial transformation, set the maximum number of iterations K max , set the convergence threshold ∈; for each source point x i , find the nearest target point y i :

[0075] y i =argmin y∈Y ‖R k-1 x i +t k-1 -y‖2;

[0076] Among them, yi Indicates the nearest target point found, Y represents the target point set, R k-1 and t k-1 are the rotation matrix and translation vector of the k-1th iteration, respectively, and ‖·‖2 represents the Euclidean distance norm. This step establishes point correspondences by finding the nearest neighbor of each source point in the target point set under the current transformation.

[0077] Calculate the registration weight of each pair of corresponding points, expressed as:

[0078]

[0079] Among them, ω i is the registration weight of the i-th pair of points. This weight formula is designed so that closer pairs of points receive higher weights during the registration process, thereby reducing the impact of outliers and false matches and improving the algorithm’s robustness to noise and outliers.

[0080] Construct a function to minimize the error, expressed as:

[0081]

[0082] Among them, E icp (R k ,t k ) is the weighted error function that needs to be minimized in the current iteration, P is the total number of matching point pairs, R k and t k are the rotation matrix and translation vector sought in the current iteration.

[0083] Use singular value decomposition to solve the error function. First, calculate the weighted centroid, which is expressed as:

[0084]

[0085] in, and are the weighted centroids of the source point set and the target point set, respectively. Through these centroids, the translation problem can be transformed into a problem that only solves the rotation, simplifying the calculation process.

[0086] Calculate the covariance matrix, expressed as:

[0087]

[0088] Among them, H cov It is the weighted covariance matrix of the source and target point sets after removing the centroid, reflecting the spatial relationship between the two point sets. This matrix contains the correlation information between point pairs and is the key to solving the optimal rotation.

[0089] The covariance matrix is ​​decomposed by SVD and expressed as:

[0090]

[0091] Among them, U svd and V svd is an orthogonal matrix, Σ svd Is a diagonal matrix containing singular values. cov Decompose it into the product of three matrices to facilitate the subsequent calculation of the optimal rotation matrix.

[0092] Solve the optimal rotation matrix so that the sum of the square distances between the source point cloud and the target point cloud is minimized, which is expressed as:

[0093]

[0094] Calculate the optimal translation vector so that the center of the rotated source point cloud coincides with the center of the target point cloud, which is expressed as:

[0095]

[0096] By finding the optimal rigid body transformation (rotation + translation), the transformed source point cloud and the target point cloud are made to coincide as much as possible, ensuring the correctness of the solution and minimizing the sum of the weighted square distances between point pairs.

[0097] If ‖t k -t k-1 ‖2<∈ and ‖R k -R k-1 ‖ F <∈, then stop the iteration and output the transformation matrix T of the precise registration fine =[R k |t k ]. Among them, ‖t k -t k-1 ‖2 represents the Euclidean distance between two adjacent iterative translation vectors, ‖R k -R k-1 ‖ F Represents the Frobenius norm, which is calculated as the square root of the sum of the squares of all matrix elements and is used to measure the degree of difference between two rotation matrices.

[0098] It should be noted that the weighted ICP algorithm has the following advantages over the traditional ICP algorithm:

[0099] 1. By introducing an adaptive weighting mechanism that is inversely proportional to distance, the impact of outliers and mismatches on the final results is effectively reduced;

[0100] 2. Use SVD decomposition to directly solve the optimal rotation matrix, avoiding the possible local optimal problem of iterative solution;

[0101] 3. The following optimizations were made to address the characteristics of multimodal data:

[0102] For areas with varying point cloud density: an adaptive weighting mechanism is used to mitigate the deviation caused by uneven density.

[0103] For multi-source data with low overlap: set a reasonable iteration termination condition ∈ overlap =0.001;

[0104] For data of different scales: adopt normalization processing strategy, oblique photography data and ground truth data use ∈ image =0.0005, laser point cloud data uses∈ lidar =0.005.

[0105] The fine registration algorithm combines the initial transformation provided by the coarse registration and continuously optimizes the transformation parameters through multiple iterations to finally obtain a high-precision registration result T fine , which lays the foundation for subsequent multi-source data fusion. The transformed data points can be expressed as:

[0106] x result =R k ·x+t k ;

[0107] Among them, x result It is the final transformed point coordinate, which realizes the precise alignment of multi-source data in the same coordinate system.

[0108] Optimally, this application can effectively handle the registration problem of multi-source, multi-dimensional data through the combination of coarse and fine registration, improving the accuracy and reliability of the final results. The RANSAC algorithm can effectively handle outliers and mismatches in multimodal data, and the robustness of the algorithm is improved through a weighted scoring mechanism. The weighted ICP algorithm further optimizes the registration results, taking into account the reliability of different data sources, and achieving high-precision alignment.

[0109] S3: Perform multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features.

[0110] For the second multi-source data after preprocessing and registration, a pre-trained deep convolutional neural network is used to extract multi-scale features from oblique photography data, laser scanning data and ground truth data respectively, thereby capturing rich information from low-level textures to high-level semantics.

[0111] In an optional embodiment, multi-scale feature extraction can use pre-trained networks such as VGG16 or ResNet50 to extract feature maps at different levels, where low-level feature maps contain rich edge and texture information, and high-level feature maps contain more abstract semantic information.

[0112] The extracted features are fused with the self-attention mechanism and the cross-attention mechanism. The self-attention mechanism is used to highlight the important features within the same data source, while the cross-attention mechanism is used to capture the complementary information between different modal data, including the feature interactions between oblique photography and laser scanning, oblique photography and ground reality, and laser scanning and ground reality.

[0113] In an optional embodiment, the multi-head attention mechanism in the Transformer architecture can be used to implement self-attention and cross-attention calculations, convert features into queries, keys and values ​​through linear projection, calculate the attention weight distribution, and then obtain weighted feature representation.

[0114] The obtained self-attention features and cross-attention features are connected and integrated to form fusion features for subsequent 3D reconstruction. This fusion method makes full use of the texture information of oblique photography data, the geometric accuracy of laser scanning data and the close-up details of ground-truth data, laying the foundation for building high-quality 3D models.

[0115] In an optional embodiment, a feature concatenation operation can be used to connect all feature maps in the channel dimension, and the dimension can be reduced through a 1×1 convolution layer to eliminate redundant information and obtain a compact fused feature representation.

[0116] S4: Perform three-dimensional reconstruction on the fused features, optimize and texture map the results of the three-dimensional reconstruction, and obtain a three-dimensional model.

[0117] Based on the fusion features, depth estimation is performed through the MVSNet network to generate dense point cloud data.

[0118] In an optional embodiment, the MVSNet (Multi-View Stereo Network) algorithm can be used. The network introduces multimodal data constraints, uses fusion features to generate a high-precision depth map, and converts the depth map into point cloud data through a back-projection method, providing a basis for subsequent reconstruction.

[0119] The Poisson surface reconstruction algorithm is applied to the generated point cloud data to construct a three-dimensional mesh model and perform local optimization.

[0120] In an optional embodiment, the surface can be reconstructed using the Poisson equation, and a closed mesh can be generated based on the vector field defined by the point cloud. The mesh can then be simplified and smoothed to eliminate noise and artifacts, thereby improving the geometric accuracy and topological rationality of the model.

[0121] Multi-resolution texture mapping is performed on the optimized three-dimensional model based on the oblique photography data and the ground reality data to obtain a three-dimensional model.

[0122] In an optional embodiment, a texture selection algorithm based on visual quality can be used to select the best texture image for each triangular facet based on factors such as the angle between the line of sight and the facet normal, image resolution, and camera distance, and a weighted average method is used to fuse the textures of overlapping areas to generate a high-quality, seamless texture mapping result.

[0123] In summary, this application improves the integrity and accuracy of 3D reconstruction by comprehensively utilizing multi-source data from oblique photography, laser scanning and ground reality; adopts adaptive filtering denoising and multimodal data registration technology to achieve high-quality data preprocessing; combines self-attention and cross-attention mechanisms to achieve effective feature fusion between different modal data; based on the fused features, the MVSNet network and Poisson surface reconstruction algorithm are used to construct a high-quality 3D model; proposes a multi-resolution texture mapping strategy based on visual quality to generate a seamless texture effect; establishes a complete end-to-end 3D reconstruction framework, which overcomes the problems of single data source, difficult registration, insufficient feature fusion, and other problems in the existing technology, and achieves high-precision and high-quality 3D scene reconstruction.

[0124] Example 2: Based on the first example, this example further provides a system for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion, including:

[0125] A collection module is used to obtain first multi-source data of the target area; the first multi-source data includes oblique photography data, laser scanning data and ground truth data;

[0126] A processing module, configured to pre-process and register the first multi-source data to obtain second multi-source data; the registration includes coarse registration and fine registration;

[0127] A fusion module, configured to perform multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features;

[0128] The construction module is used to perform three-dimensional reconstruction of the fused features, optimize and texture map the results of the three-dimensional reconstruction, and obtain a three-dimensional model.

[0129] This embodiment also provides a computer device, which is suitable for the case of a method for three-dimensional reconstruction of real scenes from oblique photography based on multi-source data fusion, and includes a memory and a processor; the memory is used to store computer-executable instructions, and the processor is used to execute computer-executable instructions to implement the method for three-dimensional reconstruction of real scenes from oblique photography based on multi-source data fusion as proposed in the above embodiment.

[0130] The computer device may be a terminal, comprising a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse.

[0131] This embodiment further provides a storage medium having a computer program stored thereon. When the program is executed by a processor, the method for 3D reconstruction of real scenes using oblique photography based on multi-source data fusion as proposed in the above embodiment is implemented.

[0132] The storage medium proposed in this embodiment and the data storage method proposed in the above embodiment belong to the same inventive concept. Technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0133] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application and are not intended to limit the present application. Although the present application has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present application, and all of these should be included in the scope of the claims of the present application.

Claims

1. A method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion, characterized by: include, Acquire first multi-source data of a target area; Preprocessing and registering the first multi-source data to obtain second multi-source data; the registration includes coarse registration and fine registration; performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; The fused features are three-dimensionally reconstructed, and the result of the three-dimensional reconstruction is optimized and texture mapped to obtain a three-dimensional model.

2. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 1, characterized in that: The pretreatment includes: Applying an adaptive filtering algorithm to the first multi-source data to perform noise reduction processing; Normalizing the first multi-source data after noise reduction to make the data ranges of different modalities consistent; After the preprocessing and before the registration, the method further includes extracting multimodal feature points and descriptors of the preprocessed first multi-source data.

3. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 2, characterized in that: The multimodal feature points and the descriptors are coarsely aligned using a random sampling consensus RANSAC algorithm; the coarse alignment includes: Set the maximum number of iterations, inlier threshold, and minimum consistent set size; Randomly selecting a minimum sample set and calculating a transformation model, wherein the transformation model is used to transform source data points from a source coordinate system to a target coordinate system; Applying the transformation model to all matching points, and calculating the Euclidean distance error between each pair of matching points; Determine the inliers based on the error threshold and assign a credibility weight to each matching point pair according to the reliability difference of the multimodal features. The credibility weight is calculated based on the relationship between the distance between the matching point descriptors and the overall distance distribution. Calculate the weighted consistent set score, that is, divide the sum of the weights of the inliers by the sum of the weights of all points; if the score of the current model is higher than the previous best score, update the best model and the best score, and re-estimate the transformation model using all inliers; after iteration, obtain the transformation matrix of the coarse registration, which includes the rotation matrix and the translation vector.

4. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 3, characterized in that: The weighted ICP algorithm is used to perform fine registration on the coarse registration results to obtain the registered multi-source data; the fine registration includes: Use the transformation matrix of the rough registration as the initial transformation, set the maximum number of iterations and the convergence threshold; For each source data point, find the nearest target point in the target point set and establish a point correspondence; Calculate the registration weight of each pair of corresponding points; Construct a weighted error function and minimize it; Calculate the weighted centroid of the source data point set and the target point set to transform the translation problem into a rotation problem; Calculate the weighted covariance matrix after removing the centroid; Perform singular value decomposition on the covariance matrix to obtain the singular value decomposition result; Calculating an optimal rotation matrix based on the singular value decomposition result to minimize the sum of square distances between the source data point set and the target point set; Calculate the optimal translation vector so that the center of the rotated source data point set coincides with the center of the target point set; Check whether the change of the rotation matrix and translation vector between two adjacent iterations is less than the convergence threshold. If the condition is met, stop the iteration and output the transformation matrix of the precise registration; Convert the source data points to the target coordinate system to complete the alignment of multiple source data in the same coordinate system.

5. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 4, characterized in that: The multi-scale feature extraction and feature fusion are performed on the second multi-source data to obtain fused features, including: extracting multi-scale features from the second multi-source data using a pre-trained deep convolutional neural network; Apply the self-attention mechanism and the cross-attention mechanism to fuse the extracted features; The self-attention features and cross-attention features are connected and integrated to form fusion features.

6. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 5, characterized in that: The first multi-source data includes oblique photography data, laser scanning data and ground truth data.

7. The method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to claim 6, characterized in that: Performing three-dimensional reconstruction on the fused features, optimizing and texture mapping the results of the three-dimensional reconstruction to obtain a three-dimensional model, including: Based on the fusion features, the depth is estimated through the MVSNet network to generate dense point cloud data; Apply the Poisson surface reconstruction algorithm to the generated point cloud data to construct a three-dimensional mesh model and perform local optimization; Multi-resolution texture mapping is performed on the optimized three-dimensional model based on the oblique photography data and the ground reality data to obtain a three-dimensional model.

8. A system for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion, based on the method for 3D reconstruction of real scenes from oblique photography based on multi-source data fusion according to any one of claims 1 to 7, characterized in that: include, A collection module, acquiring first multi-source data of a target area; The first multi-source data includes oblique photography data, laser scanning data and ground truth data; A processing module, performing preprocessing and registration on the first multi-source data to obtain second multi-source data; the registration includes coarse registration and fine registration; a fusion module, performing multi-scale feature extraction and feature fusion on the second multi-source data to obtain fused features; A construction module is used to perform three-dimensional reconstruction on the fused features, and to optimize and texture map the results of the three-dimensional reconstruction to obtain a three-dimensional model.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for three-dimensional reconstruction of real scenes based on oblique photography based on multi-source data fusion according to any one of claims 1 to 7 are implemented.

Images