A real scene three-dimensional reconstruction method based on remote sensing image and oblique photography data

Abstract

This application discloses a method for real-scene 3D reconstruction based on remote sensing images and oblique photogrammetry data, belonging to the field of photogrammetric image processing technology. The method includes acquiring and preprocessing multi-source remote sensing images and oblique photogrammetry data; performing spatiotemporal unification based on a unified spatiotemporal coordinate system; extracting global features from remote sensing images and local features from oblique photogrammetry; constructing a multi-scale feature pyramid based on the extracted global and local features; iteratively optimizing the multi-source remote sensing images and oblique photogrammetry data; constructing a 3D mesh model based on the iteratively optimized multi-source remote sensing images and oblique photogrammetry data; and finally generating a 3D real-scene scene by combining the spatiotemporal unification results. This method ensures the spatial consistency and visual continuity of the fused data, improves the perception and reconstruction capabilities of large-scale terrain structures and small component details, and enhances the geometric accuracy of the model.

Description

Technical Field

[0001] This application belongs to the field of photographic image processing technology, specifically, it relates to a method for real-scene 3D reconstruction based on remote sensing images and oblique photographic data. Background Technology

[0002] Real-world 3D reconstruction is a core technology in fields such as digital twins, smart cities, cultural heritage protection, and autonomous driving. Currently, mainstream real-world 3D reconstruction technologies mainly rely on two types of data sources: first, remote sensing imagery carried by satellites or drones, which can acquire large-scale, macroscopic land cover and terrain information; and second, oblique photogrammetry data, which uses multi-angle cameras to acquire dense point clouds and texture details of building facades and small components. Ideally, by combining the macroscopic advantages of remote sensing imagery with the microscopic advantages of oblique photogrammetry, a real-world 3D scene can be constructed that possesses both a large-scale geographical pattern and rich local fine structure. Therefore, multi-source data fusion 3D reconstruction has become a current research hotspot and direction.

[0003] However, in existing multi-source data fusion 3D reconstruction, the multi-source data come from different sources, and their spatial coordinate systems, time bases, and radiometric characteristics vary significantly. Existing methods usually perform spatial registration first and then process radiometric differences separately, which easily leads to geometric misalignment problems at the data fusion boundary, affecting the visual realism and geometric accuracy of the 3D model. Moreover, remote sensing images and oblique photogrammetry data have plug-ins at the feature scale. Traditional methods often use single-branch networks or simple feature stitching, lacking cross-modal interaction mechanisms. This results in reconstruction results that are either accurate in macro structure but lose micro details, or detailed locally but disordered in global topological relationships, leading to obvious visual defects in the final 3D scene. Summary of the Invention

[0004] To address the aforementioned problems and technical deficiencies, this application adopts the following technical solution: a method for real-scene 3D reconstruction based on remote sensing imagery and oblique photogrammetry data, comprising the following steps: Multi-source remote sensing images and oblique photography data are collected and preprocessed. Spatiotemporal unification is performed based on a unified spatiotemporal coordinate system, and then global features of remote sensing images and local features of oblique photography are extracted. A multi-scale feature pyramid is constructed based on the extracted global and local features, and iterative optimization is performed on multi-source remote sensing images and oblique photogrammetry data. A 3D mesh model is constructed based on the iteratively optimized multi-source remote sensing images and oblique photography data, and then a 3D real-world scene is generated by combining the spatiotemporal unified results.

[0005] Preferably, the preprocessing involves performing radiometric correction, geometric correction, and denoising on the raw data of multi-source remote sensing images and oblique photogrammetry data, and then unifying the coordinate system and resolution to eliminate errors between different data sources. After unifying the coordinate system and resolution, a two-way registration model based on minimizing spatiotemporal residuals is constructed, while spatiotemporal alignment and radiometric consistency correction are performed simultaneously. A graph optimization method is used to establish the constraint relationship between sensor pose and timestamp. With minimizing reprojection error and time offset error as the objective function, a joint optimization Huber loss function is constructed. Based on a unified geographic coordinate system, multi-source data is matched, and the two types of data are mapped to the same spatial reference. Spatiotemporal registration and radiometric correction are performed to achieve spatiotemporal unification.

[0006] Preferably, the extraction of global features from remote sensing images and local features from oblique photography is performed using a dual-branch heterogeneous feature extraction network to extract deep features from preprocessed multi-source remote sensing images and oblique photography data, respectively. The multi-scale feature pyramid is a cross-modal feature transfer network based on the combination of macroscopic features of remote sensing images and microscopic features of oblique photography. It includes a global semantic feature layer of remote sensing images and a local geometric feature layer of oblique photography. The global semantic feature layer is based on the global encoder of remote sensing images. It performs global self-attention modeling on remote sensing images and outputs a global semantic feature tensor. The local geometric feature layer is based on the oblique photogrammetry local geometric encoder, which extracts the multi-scale geometric neighborhood features of points in the oblique photogrammetry data and generates a local geometric feature tensor.

[0007] Preferably, the cross-modal feature transfer network is designed based on the Transformer cross-attention architecture, enabling multi-level cross-modal information interaction between the global semantic feature layer and the local geometric feature layer; The output of the cross-modal feature transfer network is subjected to cascaded upsampling and downsampling operations to construct a multi-scale heterogeneous feature pyramid. Each level of the multi-scale heterogeneous feature pyramid simultaneously contains global semantic information of remote sensing images and local geometric information of oblique photogrammetry.

[0008] Furthermore, the method involves extracting global pose features from multi-source data based on a cross-modal feature encoder, encoding them in the feature embedding space, calculating the optimal transmission cost matrix, and achieving coarse alignment of cross-modal data at the global level. Multi-scale pixel-level local feature descriptions are extracted based on coarse alignment results. Graph convolutional networks are used to encode the spatial correlation between adjacent pixels to obtain pixel-level dense matching relationships. Local geometric optimization is performed through a differentiable spatial transformation module. A multi-task joint loss function is constructed with spatial consistency, photometric consistency, and cross-view normal consistency constraints. The Adam optimizer is used to perform joint iterative optimization of the three constraints until convergence, resulting in optimized and aligned multi-source feature data.

[0009] Preferably, the process of generating the three-dimensional real-scene includes: First, an initial sparse point cloud is constructed based on the optimized multi-source image features, and then feature matching and bundle adjustment are used to optimize and obtain high-precision camera pose. Based on camera pose, generate 3D point coordinate trajectories, perform multi-view dense matching and depth estimation on the 3D point coordinate trajectories to generate dense point clouds, and then use the Poisson surface reconstruction algorithm to construct a 3D mesh model. Finally, by fusing the texture mapping and spatiotemporally unified semantic segmentation results, a 3D real-world scene with semantic attributes is generated.

[0010] Furthermore, in the macroscopic layer of the three-dimensional real-world scene, remote sensing image data is used to construct a large-scale terrain and landform framework, while oblique photography data is used to supplement the details of small components in the microscopic layer. The occluded area is filled by combining macroscopic topographic information from remote sensing imagery with local detail information from oblique photography using a dual-data source complementary filling algorithm to restore the true shape of the occluded area. For small components and dense areas, the model is constructed by designing modeling parameters using optimized acquisition perspective and modeling algorithm.

[0011] Furthermore, the 3D real-world scene will be texture optimized for different application scenarios. Corresponding texture optimization parameters are preset. First, the 3D real-world scene will be texture extracted, filtered and quality evaluated. Then, the evaluated texture will be optimized based on the optimal viewpoint. Texture optimization involves correcting color differences and eliminating seams in cross-source textures, followed by texture completion and super-resolution reconstruction based on a generative model.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the content of a real-scene 3D reconstruction method based on remote sensing images and oblique photogrammetry data as described above.

[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content of a real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data as described above.

[0014] Compared to existing technologies, the beneficial effects of this application are as follows: (1) This application constructs a bidirectional registration model based on minimizing spatiotemporal residuals and jointly optimizes spatiotemporal alignment and radiometric correction. It uses graph optimization methods to establish constraints on sensor pose and timestamps, eliminates geometric misalignment and radiometric differences between multi-source data, and ensures spatial consistency and visual continuity of fused data. (2) This application designs a dual-branch heterogeneous feature extraction network and a cross-modal feature transfer network based on Transformer cross-attention, so that the global semantic features of remote sensing images and the local geometric features of oblique photography can perform multi-level, bi-directional information interaction, thereby improving the ability to perceive and reconstruct large-scale terrain structures and small component details. (3) This application uses the optimal transmission cost matrix to perform coarse alignment at the global level in the feature embedding space, then obtains pixel-level dense matching relationships, and performs local geometric optimization through the differentiable space transformation module. At the same time, iterative optimization is performed by jointly constructing loss functions with three constraints: spatial consistency, photometric consistency and cross-view normal consistency, thereby improving the geometric accuracy of the model. Attached Figure Description

[0015] In the attached diagram: Figure 1 This is a schematic diagram of the method steps in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments. Generally, the components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0017] Example 1, as Figure 1 As shown, a method for real-scene 3D reconstruction based on remote sensing imagery and oblique photogrammetry data includes the following steps: Multi-source remote sensing images and oblique photography data are collected and preprocessed. Spatiotemporal unification is performed based on a unified spatiotemporal coordinate system, and then global features of remote sensing images and local features of oblique photography are extracted. Preprocessing involves radiometric correction, geometric correction, and denoising of the raw data from multi-source remote sensing images and oblique photogrammetry data, followed by unifying the coordinate system and resolution to eliminate errors between different data sources. After unifying the coordinate system and resolution, a two-way registration model based on minimizing spatiotemporal residuals is constructed, while spatiotemporal alignment and radiometric consistency correction are performed simultaneously. A graph optimization method is used to establish the constraint relationship between sensor pose and timestamp. With minimizing reprojection error and time offset error as the objective function, a joint optimization Huber loss function is constructed. Based on a unified geographic coordinate system, multi-source data is matched, and the two types of data are mapped to the same spatial reference. Spatiotemporal registration and radiometric correction are performed to achieve spatiotemporal unification.

[0018] A multi-scale feature pyramid is constructed based on the extracted global and local features, and iterative optimization is performed on multi-source remote sensing images and oblique photogrammetry data. Extracting global features from remote sensing images and local features from oblique photography employs a dual-branch heterogeneous feature extraction network to extract deep features from preprocessed multi-source remote sensing images and oblique photography data, respectively. The multi-scale feature pyramid is a cross-modal feature transfer network based on the combination of macroscopic features of remote sensing images and microscopic features of oblique photography. It includes a global semantic feature layer of remote sensing images and a local geometric feature layer of oblique photography. The global semantic feature layer is based on the global encoder of remote sensing images. It performs global self-attention modeling on remote sensing images and outputs a global semantic feature tensor. The local geometric feature layer is based on the oblique photogrammetry local geometric encoder, which extracts the multi-scale geometric neighborhood features of points in the oblique photogrammetry data and generates a local geometric feature tensor.

[0019] The cross-modal feature transfer network is designed based on the Transformer cross-attention architecture, enabling multi-level cross-modal information interaction between the global semantic feature layer and the local geometric feature layer; The output of the cross-modal feature transfer network is subjected to cascaded upsampling and downsampling operations to construct a multi-scale heterogeneous feature pyramid. Each level of the multi-scale heterogeneous feature pyramid simultaneously contains global semantic information of remote sensing images and local geometric information of oblique photogrammetry.

[0020] Global pose features of multi-source data are extracted based on a cross-modal feature encoder, encoded in the feature embedding space, and the optimal transmission cost matrix is ​​calculated to achieve coarse alignment of cross-modal data at the global level. Multi-scale pixel-level local feature descriptions are extracted based on coarse alignment results. Graph convolutional networks are used to encode the spatial correlation between adjacent pixels to obtain pixel-level dense matching relationships. Local geometric optimization is performed through a differentiable spatial transformation module. A multi-task joint loss function is constructed with spatial consistency, photometric consistency, and cross-view normal consistency constraints. The Adam optimizer is used to perform joint iterative optimization of the three constraints until convergence, resulting in optimized and aligned multi-source feature data.

[0021] A 3D mesh model is constructed based on the iteratively optimized multi-source remote sensing images and oblique photography data, and then a 3D real-world scene is generated by combining the spatiotemporal unified results.

[0022] The process of generating a 3D real-world scene includes: First, an initial sparse point cloud is constructed based on the optimized multi-source image features, and then feature matching and bundle adjustment are used to optimize and obtain high-precision camera pose. Based on camera pose, generate 3D point coordinate trajectories, perform multi-view dense matching and depth estimation on the 3D point coordinate trajectories to generate dense point clouds, and then use the Poisson surface reconstruction algorithm to construct a 3D mesh model. Finally, by fusing the texture mapping and spatiotemporally unified semantic segmentation results, a 3D real-world scene with semantic attributes is generated.

[0023] In the 3D real-world scene, the macro layer uses remote sensing image data to construct a large-scale terrain and landform framework, while the micro layer uses oblique photogrammetry data to supplement the details of small components. The occluded area is filled by combining macroscopic topographic information from remote sensing imagery with local detail information from oblique photography using a dual-data source complementary filling algorithm to restore the true shape of the occluded area. For small components and dense areas, the model is constructed by designing modeling parameters using optimized acquisition perspective and modeling algorithm.

[0024] The 3D real scene will be textured for different application scenarios. The corresponding texture optimization parameters are preset. First, the texture of the 3D real scene is extracted, filtered and quality evaluated. Then, the texture of the evaluated texture is optimized based on the optimal viewpoint. Texture optimization involves correcting color differences and eliminating seams in cross-source textures, followed by texture completion and super-resolution reconstruction based on a generative model.

[0025] Candidate texture images are filtered based on view visibility and texture clarity, and low-quality textures are eliminated. Calculate the structural similarity index and blur evaluation of texture blocks for the preserved texture images to obtain a comprehensive quality score for each texture block; The optimal texture association viewpoint for each patch is determined based on the comprehensive quality score, and the initial texture mapping is completed.

[0026] A combined objective function is constructed, which includes cross-source texture color difference correction, seam elimination, local structure preservation, and perceptual consistency constraints. The optimal seam partitioning is solved using a graph cut optimization algorithm. For regions with missing textures, a pre-trained diffusion generation model is used for semantically guided texture completion, and super-resolution reconstruction is performed on the completed texture regions to generate high-resolution completed textures.

[0027] Example 2, from a hardware perspective, this application provides an embodiment of an electronic device containing all or part of a real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data. The electronic device includes a service processor and a distributed memory. The service processor is connected to the memory. The distributed memory stores a service self-management program configured to store machine-readable instructions. The service processor executes the service self-management program. When the instructions are executed by the processor, a real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data as described above can be implemented.

[0028] Example 3: This application also provides a computer-readable storage medium capable of implementing a real-scene 3D reconstruction method based on remote sensing images and oblique photogrammetry data, where the execution subject is a server or client as described in the above embodiments. The computer-readable storage medium stores a computer program, which, when executed by a processor, implements all the contents of the real-scene 3D reconstruction method based on remote sensing images and oblique photogrammetry data, where the execution subject is a server or client as described in the above embodiments.

[0029] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for real-scene 3D reconstruction based on remote sensing imagery and oblique photogrammetry data, characterized in that, Includes the following steps: Multi-source remote sensing images and oblique photography data are collected and preprocessed. Spatiotemporal unification is performed based on a unified spatiotemporal coordinate system, and then global features of remote sensing images and local features of oblique photography are extracted. A multi-scale feature pyramid is constructed based on the extracted global and local features, and iterative optimization is performed on multi-source remote sensing images and oblique photogrammetry data. A 3D mesh model is constructed based on the iteratively optimized multi-source remote sensing images and oblique photography data, and then a 3D real-world scene is generated by combining the spatiotemporal unified results.

2. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 1, characterized in that, The preprocessing involves performing radiometric correction, geometric correction, and noise reduction on the raw data of multi-source remote sensing images and oblique photogrammetry data, and then unifying the coordinate system and resolution to eliminate errors between different data sources. After unifying the coordinate system and resolution, a two-way registration model based on minimizing spatiotemporal residuals is constructed, while spatiotemporal alignment and radiometric consistency correction are performed simultaneously. A graph optimization method is used to establish the constraint relationship between sensor pose and timestamp. With minimizing reprojection error and time offset error as the objective function, a joint optimization Huber loss function is constructed. Based on a unified geographic coordinate system, multi-source data is matched, and the two types of data are mapped to the same spatial reference. Spatiotemporal registration and radiometric correction are performed to achieve spatiotemporal unification.

3. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 1, characterized in that, The extraction of global features from remote sensing images and local features from oblique photography is achieved by using a dual-branch heterogeneous feature extraction network to extract deep features from multi-source remote sensing images and oblique photography data after data preprocessing. The multi-scale feature pyramid is a cross-modal feature transfer network based on the combination of macroscopic features of remote sensing images and microscopic features of oblique photography. It includes a global semantic feature layer of remote sensing images and a local geometric feature layer of oblique photography. The global semantic feature layer is based on the global encoder of remote sensing images. It performs global self-attention modeling on remote sensing images and outputs a global semantic feature tensor. The local geometric feature layer is based on the oblique photogrammetry local geometric encoder, which extracts the multi-scale geometric neighborhood features of points in the oblique photogrammetry data and generates a local geometric feature tensor.

4. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 3, characterized in that, The cross-modal feature transfer network is designed based on the Transformer cross-attention architecture, enabling multi-level cross-modal information interaction between the global semantic feature layer and the local geometric feature layer; The output of the cross-modal feature transfer network is subjected to cascaded upsampling and downsampling operations to construct a multi-scale heterogeneous feature pyramid. Each level of the multi-scale heterogeneous feature pyramid simultaneously contains global semantic information of remote sensing images and local geometric information of oblique photogrammetry.

5. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 4, characterized in that, The method involves extracting global pose features from multi-source data using a cross-modal feature encoder, encoding them in the feature embedding space, calculating the optimal transmission cost matrix, and achieving coarse alignment of cross-modal data at the global level. Multi-scale pixel-level local feature descriptions are extracted based on coarse alignment results. Graph convolutional networks are used to encode the spatial correlation between adjacent pixels to obtain pixel-level dense matching relationships. Local geometric optimization is performed through a differentiable spatial transformation module. A multi-task joint loss function is constructed with spatial consistency, photometric consistency, and cross-view normal consistency constraints. The Adam optimizer is used to perform joint iterative optimization of the three constraints until convergence, resulting in optimized and aligned multi-source feature data.

6. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 1, characterized in that, The process of generating the three-dimensional real-scene includes: First, an initial sparse point cloud is constructed based on the optimized multi-source image features, and then feature matching and bundle adjustment are used to optimize and obtain high-precision camera pose. Based on camera pose, generate 3D point coordinate trajectories, perform multi-view dense matching and depth estimation on the 3D point coordinate trajectories to generate dense point clouds, and then use the Poisson surface reconstruction algorithm to construct a 3D mesh model. Finally, by fusing the texture mapping and spatiotemporally unified semantic segmentation results, a 3D real-world scene with semantic attributes is generated.

7. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 6, characterized in that, The macroscopic layer of the three-dimensional real scene uses remote sensing image data to construct a large-scale terrain and landform framework, while the microscopic layer uses oblique photography data to supplement the details of small components. The occluded area is filled by combining macroscopic topographic information from remote sensing imagery with local detail information from oblique photography using a dual-data source complementary filling algorithm to restore the true shape of the occluded area. For small components and dense areas, the model is constructed by designing modeling parameters using optimized acquisition perspective and modeling algorithm.

8. The real-scene 3D reconstruction method based on remote sensing imagery and oblique photogrammetry data according to claim 7, characterized in that, The 3D real scene will be texture optimized for different application scenarios. The corresponding texture optimization parameters are preset. First, the 3D real scene will be texture extracted, filtered and quality evaluated. Then, the evaluated texture will be optimized based on the optimal viewpoint. Texture optimization involves correcting color differences and eliminating seams in cross-source textures, followed by texture completion and super-resolution reconstruction based on a generative model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the content of the real-scene 3D reconstruction method based on remote sensing images and oblique photogrammetry data as described in claim 1.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the content of the real-scene 3D reconstruction method based on remote sensing images and oblique photogrammetry data as described in claim 1.

Images