Multi-source remote sensing fusion method and system based on visible light image and sar image

Abstract

The application provides a multi-source remote sensing fusion method and system based on visible light images and SAR images, comprising: acquiring a visible light image pair and a SAR image pair of a target region in the same space-time; performing feature matching and orientation on the visible light image pair to obtain an oriented image pair; performing dense matching on the oriented image pair by using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map; extracting a visible light point cloud from the disparity map; performing dense matching on the SAR image pair by using object elevation search to obtain a SAR point cloud; generating a multi-source digital surface model by performing point cloud gridding interpolation on the visible light point cloud and the SAR point cloud; after unifying coordinates and elevations of the multi-source digital surface model, performing multi-source elevation fusion to obtain a fused digital surface model; and performing point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target region, so that the accuracy and generalization of multi-source data fusion are realized.

Description

Technical Field

[0001] This invention relates to the field of multi-source remote sensing fusion technology, specifically to a multi-source remote sensing fusion method and system based on visible light images and SAR images. Background Technology

[0002] With the rapid development of Earth observation technology, modern remote sensing systems have established an integrated three-dimensional monitoring network encompassing air, space, and ground. Multi-platform, multi-sensor collaborative observation systems have achieved all-weather, all-time dynamic monitoring capabilities, from macro to micro perspectives, providing unprecedented data support for applications such as resource surveys, environmental monitoring, and disaster early warning. In particular, the network operation of Earth observation systems such as the Gaofen series satellites and Sentinel satellites has enabled a qualitative leap in the acquisition of high spatiotemporal resolution remote sensing data, laying a solid foundation for the automated production of 3D geographic information products. However, how to fully leverage the synergistic advantages of multi-source heterogeneous remote sensing data remains a key technical challenge that urgently needs to be addressed in the field of remote sensing information processing.

[0003] Due to significant differences in their imaging mechanisms, multi-source remote sensing devices acquire diverse images with varying physical meanings, spatial resolutions, measurement ranges, and error characteristics. This makes effective integration of these heterogeneous data during the fusion process challenging, severely limiting the accuracy and completeness of geographic information extracted from the fused data. Furthermore, multi-source remote sensing data typically contains massive amounts of information, resulting in low data fusion efficiency. Summary of the Invention

[0004] To overcome the shortcomings of the prior art, this invention proposes a multi-source remote sensing fusion method based on visible light imagery and SAR imagery, comprising:

[0005] Acquire visible light image pairs and SAR image pairs of the same time and space in the target area;

[0006] The visible light image pairs are subjected to feature matching and orientation to obtain oriented image pairs. The oriented image pairs are densely matched using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map. Visible light point clouds are extracted from the disparity map.

[0007] The SAR image pairs are densely matched using object elevation search to obtain SAR point clouds;

[0008] A multi-source digital surface model is generated by performing point cloud rasterization interpolation on the visible light point cloud and SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. The fused surface model is then transformed into a point cloud to obtain the three-dimensional point cloud of the target area.

[0009] Optionally, the step of using a pyramid matching strategy combined with a semi-global algorithm to perform dense matching on the directional image pairs to obtain a disparity map includes:

[0010] Epipolar line correction is performed on the directional image pairs to obtain epipolar line image pairs. The pyramid matching strategy is used to divide the epipolar line image pairs into multi-resolution levels and determine the matching disparity range of the initial low-resolution level.

[0011] The pixel similarity of candidate pixels in the matching disparity range is calculated by combining the matching costs, and an initial cost cube is generated.

[0012] The optimal disparity of the candidate pixel is obtained by optimizing the initial cost cube using a semi-global matching algorithm. Based on the optimal disparity, the matching disparity range of the next level is determined. The optimal disparity of the candidate pixel in the matching disparity range of the next level is determined by combining the matching cost with the semi-global matching algorithm. This process is passed down level by level to determine the optimal disparity of the candidate pixel in the matching disparity range of each level.

[0013] The disparity map of the epipolar image pair is obtained based on the optimal disparity of candidate pixels within the matching disparity range of each level.

[0014] Optionally, extracting the visible light point cloud from the disparity map includes:

[0015] Obtain multiple pairs of identical image points corresponding to the directional image pairs in the disparity map;

[0016] Based on each pair of corresponding image points, a nonlinear error equation is established using the rational function model parameters of the two orientation images and the collinearity equation corresponding to the image coordinates of the corresponding image points. The average value of the regularized translation parameters of the two orientation images is used as the initial value of the object coordinates. The nonlinear error equation is iteratively calculated using a nonlinear least squares algorithm until the convergence condition is met, and the object coordinates of the corresponding image points are obtained.

[0017] The visible light point cloud of the visible light image pair is determined based on the object-space coordinates of each pair of image points with the same name.

[0018] Optionally, the step of performing feature matching and orientation on the visible light image pair to obtain an oriented image pair includes:

[0019] By using the SIFT algorithm combined with a distance metric function, based on the grayscale change characteristics and structural information of visible light image pairs, feature points of the visible light image pairs are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs.

[0020] Initialize the model parameters of the rational function model that introduces image-square compensation parameters;

[0021] For each pair of connection points, an error equation is established based on the image compensation parameters and model parameters. By combining the error equations of multiple pairs of connection points, an overdetermined set of equations is obtained.

[0022] The overdetermined system of equations is solved using regional network adjustment techniques to obtain the image-side compensation parameters and model parameters. Based on the image-side compensation parameters and model parameters, the final rational function model is determined.

[0023] The object coordinates of the visible light image pair are determined using the final rational function model, and the orientationd image pair is obtained based on the same object coordinates.

[0024] Optionally, before performing dense matching of the SAR image pairs using object elevation search, the method further includes:

[0025] Using an atmospheric propagation delay correction model, atmospheric propagation delay corrections are generated based on neutral atmospheric and ionospheric parameters of the target region;

[0026] Based on each SAR image in the SAR image pair, the calibration parameters in the pre-built calibration model are solved by least squares using the difference between the geographic coordinates of the corner reflectors deployed in the calibration field of the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction. The SAR image is then corrected based on the calibration parameters to obtain a geometrically calibrated SAR image.

[0027] Optionally, the step of performing dense matching of the SAR image pairs using object-space elevation search to obtain SAR point clouds includes:

[0028] Multiple candidate elevation values ​​are initialized based on the terrain features of the target area, and multiple grid points representing the candidate locations are established in the object space of the target area.

[0029] Based on each grid point representing a candidate location, multiple elevation candidate values ​​are traversed, and the grid points are projected onto the SAR image pair to obtain the corresponding image point coordinates; based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point.

[0030] The SAR point cloud is obtained based on the object coordinates of each grid point in the image point coordinates and the true elevation of the SAR image pair.

[0031] Optionally, the step of unifying the coordinates and elevations of the multi-source digital surface model and then performing multi-source elevation fusion to obtain a fused digital surface model includes:

[0032] In the multi-source digital surface model, corresponding control points are selected, and the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system are solved by least squares fitting. Based on the coordinate transformation parameters, coordinate transformation is performed to obtain a multi-source digital surface model under the same target geographic coordinate system.

[0033] The multi-source digital surface model under the same target geographic coordinate system is transformed from the original elevation to the preset target elevation datum to obtain the final multi-source digital surface model;

[0034] Using the reliable elevation values ​​of each point in the visible light point cloud and SAR point cloud determined by the image matching cost and coherence coefficient, and combined with the pre-calibrated multi-source weight ratio, the final multi-source digital surface model is subjected to elevation fusion to generate a fused digital surface model.

[0035] Optionally, the step of performing point cloud transformation on the fused surface model to obtain the three-dimensional point cloud of the target region includes:

[0036] The fused digital surface model is converted from grid elevation to three-dimensional spatial coordinates pixel by pixel to generate an initial fused three-dimensional point cloud. The initial fused three-dimensional point cloud is then interpolated to obtain a fused three-dimensional point cloud.

[0037] Radiometric correction and Walls filtering are performed on multiple original images to obtain grayscale normalized original images, and the global weight of each original image is determined based on the resolution and texture complexity of each original image.

[0038] Based on the global weight and pixel value of each original image, and the correspondence between the points of the fused 3D point cloud and the pixels in each original image, the pixel value of each point in each band of the fused 3D point cloud is calculated respectively. Each original image includes the visible light image pair and the SAR image pair.

[0039] The fused 3D point cloud is obtained by coloring the fused 3D point cloud based on the pixel values ​​of each point in each band.

[0040] Optionally, the step of interpolating and supplementing the initial fused 3D point cloud to obtain the fused 3D point cloud includes:

[0041] Point cloud density analysis was performed on the initial fused 3D point cloud to detect point cloud void regions;

[0042] Image segmentation technology is used to perform semantic segmentation and connected component analysis on the visible light image pairs and SAR image pairs, and candidate regions are obtained by combining screening rules.

[0043] The candidate region is spatially associated with the point cloud hole region by spatial indexing to obtain the point cloud region to be supplemented and the key regions in the visible light image pair and SAR image pair.

[0044] In the key area, feature matching is performed using the visible light image pairs and SAR image pairs to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a new point cloud.

[0045] The newly added point cloud is used to interpolate the area of ​​the point cloud to be supplemented to obtain a fused 3D point cloud.

[0046] Based on the same inventive concept, this invention proposes a multi-source remote sensing fusion system based on visible light images and SAR images, comprising:

[0047] The data acquisition module is used to acquire visible light image pairs and SAR image pairs of the same time and space in the target area;

[0048] The visible light point cloud extraction module is used to perform feature matching and orientation on the visible light image pairs to obtain oriented image pairs, and to perform dense matching on the oriented image pairs using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and to extract visible light point clouds from the disparity map.

[0049] The SAR point cloud extraction module is used to perform dense matching of the SAR image pairs using object-space elevation search to obtain SAR point clouds.

[0050] The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud raster interpolation on the visible light point cloud and SAR point cloud, and after unifying the coordinates and elevations of the multi-source digital surface model, perform multi-source elevation fusion to obtain a fused digital surface model; and perform point cloud transformation on the fused surface model to obtain the three-dimensional point cloud of the target area.

[0051] Optionally, the visible light point cloud extraction module is specifically used for:

[0052] Epipolar line correction is performed on the directional image pairs to obtain epipolar line image pairs. The pyramid matching strategy is used to divide the epipolar line image pairs into multi-resolution levels and determine the matching disparity range of the initial low-resolution level.

[0053] The pixel similarity of candidate pixels in the matching disparity range is calculated by combining the matching costs, and an initial cost cube is generated.

[0054] The optimal disparity of the candidate pixel is obtained by optimizing the initial cost cube using a semi-global matching algorithm. Based on the optimal disparity, the matching disparity range of the next level is determined. The optimal disparity of the candidate pixel in the matching disparity range of the next level is determined by combining the matching cost with the semi-global matching algorithm. This process is passed down level by level to determine the optimal disparity of the candidate pixel in the matching disparity range of each level.

[0055] The disparity map of the epipolar image pair is obtained based on the optimal disparity of candidate pixels within the matching disparity range of each level.

[0056] Optionally, the visible light point cloud extraction module is specifically used for:

[0057] Obtain multiple pairs of identical image points corresponding to the directional image pairs in the disparity map;

[0058] Based on each pair of corresponding image points, a nonlinear error equation is established using the rational function model parameters of the two orientation images and the collinearity equation corresponding to the image coordinates of the corresponding image points. The average value of the regularized translation parameters of the two orientation images is used as the initial value of the object coordinates. The nonlinear error equation is iteratively calculated using a nonlinear least squares algorithm until the convergence condition is met, and the object coordinates of the corresponding image points are obtained.

[0059] The visible light point cloud of the visible light image pair is determined based on the object-space coordinates of each pair of image points with the same name.

[0060] Optionally, the visible light point cloud extraction module is specifically used for:

[0061] By using the SIFT algorithm combined with a distance metric function, based on the grayscale change characteristics and structural information of visible light image pairs, feature points of the visible light image pairs are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs.

[0062] Initialize the model parameters of the rational function model that introduces image-square compensation parameters;

[0063] For each pair of connection points, an error equation is established based on the image compensation parameters and model parameters. By combining the error equations of multiple pairs of connection points, an overdetermined set of equations is obtained.

[0064] The overdetermined equations are solved using regional network adjustment techniques to obtain the image-side compensation parameters and model parameters. Based on the image-side compensation parameters and model parameters, the final rational function model is determined.

[0065] The object coordinates of the visible light image pair are determined using the final rational function model, and the orientationd image pair is obtained based on the same object coordinates.

[0066] Optionally, the SAR point cloud extraction module is further used for:

[0067] Based on each SAR image in the SAR image pair:

[0068] Using an atmospheric propagation delay correction model, atmospheric propagation delay corrections are generated based on neutral atmospheric and ionospheric parameters of the target region;

[0069] Based on each SAR image in the SAR image pair, the calibration parameters in the pre-built calibration model are solved by least squares using the difference between the geographic coordinates of the corner reflectors deployed in the calibration field of the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction. The SAR image is then corrected based on the calibration parameters to obtain a geometrically calibrated SAR image.

[0070] Optionally, the SAR point cloud extraction module is specifically used for:

[0071] Multiple candidate elevation values ​​are initialized based on the terrain features of the target area, and multiple grid points representing the candidate locations are established in the object space of the target area.

[0072] Based on each grid point representing a candidate location, multiple elevation candidate values ​​are traversed, and the grid points are projected onto the SAR image pair to obtain the corresponding image point coordinates; based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point.

[0073] The SAR point cloud is obtained based on the object coordinates of each grid point in the image point coordinates and the true elevation of the SAR image pair.

[0074] Optionally, the point cloud fusion module is specifically used for:

[0075] In the multi-source digital surface model, corresponding control points are selected, and the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system are solved by least squares fitting. Based on the coordinate transformation parameters, coordinate transformation is performed to obtain a multi-source digital surface model under the same target geographic coordinate system.

[0076] The multi-source digital surface model under the same target geographic coordinate system is transformed from the original elevation to the preset target elevation datum to obtain the final multi-source digital surface model;

[0077] Using the reliable elevation values ​​of each point in the visible light point cloud and SAR point cloud determined by the image matching cost and coherence coefficient, and combined with the pre-calibrated multi-source weight ratio, the final multi-source digital surface model is subjected to elevation fusion to generate a fused digital surface model.

[0078] Optionally, the point cloud fusion module is specifically used for:

[0079] The fused digital surface model is converted from grid elevation to three-dimensional spatial coordinates pixel by pixel to generate an initial fused three-dimensional point cloud. The initial fused three-dimensional point cloud is then interpolated to obtain a fused three-dimensional point cloud.

[0080] Radiometric correction and Walls filtering are performed on multiple original images to obtain grayscale normalized original images, and the global weight of each original image is determined based on the resolution and texture complexity of each original image.

[0081] Based on the global weight and pixel value of each original image, and the correspondence between the points of the fused 3D point cloud and the pixels in each original image, the pixel value of each point in each band of the fused 3D point cloud is calculated respectively. Each original image includes the visible light image pair and the SAR image pair.

[0082] The fused 3D point cloud is obtained by coloring the fused 3D point cloud based on the pixel values ​​of each point in each band.

[0083] Optionally, the point cloud fusion module is specifically used for:

[0084] Point cloud density analysis was performed on the initial fused 3D point cloud to detect point cloud void regions;

[0085] Image segmentation technology is used to perform semantic segmentation and connected component analysis on the visible light image pairs and SAR image pairs, and candidate regions are obtained by combining screening rules.

[0086] The candidate region is spatially associated with the point cloud hole region by spatial indexing to obtain the point cloud region to be supplemented and the key regions in the visible light image pair and SAR image pair.

[0087] In the key area, feature matching is performed using the visible light image pairs and SAR image pairs to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a new point cloud.

[0088] The newly added point cloud is used to interpolate the area of ​​the point cloud to be supplemented to obtain a fused 3D point cloud.

[0089] In another aspect, this application also provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0090] The memory is used to store one or more programs;

[0091] When the one or more programs are executed by the at least one processor, the multi-source remote sensing fusion method based on visible light images and SAR images as described above is implemented.

[0092] In another aspect, this application also provides a computer-readable storage medium having an executable program stored thereon, which, when executed, implements the multi-source remote sensing fusion method based on visible light images and SAR images as described above.

[0093] Compared with the closest existing technology, the present invention has the following beneficial effects:

[0094] This invention provides a multi-source remote sensing fusion method and system based on visible light imagery and SAR imagery, comprising: acquiring visible light image pairs and SAR image pairs of the same time and space in a target area; performing feature matching and orientation on the visible light image pairs to obtain oriented image pairs; using a pyramid matching strategy combined with a semi-global algorithm to perform dense matching on the oriented image pairs to obtain a disparity map; extracting visible light point clouds from the disparity map; performing dense matching on the SAR image pairs using object-space elevation search to obtain SAR point clouds; generating a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point clouds and SAR point clouds; and performing multi-source elevation fusion on the multi-source digital surface model after unifying the coordinates and elevations to obtain a fused digital surface model. The fused surface model is converted into a point cloud to obtain the three-dimensional point cloud of the target area. This invention utilizes a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, which facilitates the extraction of three-dimensional point clouds from the image. At the same time, the efficiency of data processing can be improved by "reducing the matching disparity range" through pyramid matching and "only searching the same row of pixels during dense matching in epipolar correction, avoiding searching the entire image". Extracting three-dimensional point clouds from the image and fusing them solves the problem of fusion difficulties caused by different imaging mechanisms. Traditional digital surface models that store elevation are not conducive to importing multi-source remote sensing information. In this scheme, multi-source digital surface models are converted to the same elevation datum and coordinate system for multi-source elevation fusion, realizing the accuracy and generalization of multi-source data fusion. Attached Figure Description

[0095] Figure 1 A flowchart illustrating a multi-source remote sensing fusion method based on visible light images and SAR images provided by the present invention;

[0096] Figure 2 A schematic diagram of the radar stereo image acquisition method provided by the present invention;

[0097] Figure 3 This is a schematic diagram of the visible light image three-dimensional information extraction process provided by the present invention;

[0098] Figure 4 A schematic diagram of the visible light image geometric preprocessing workflow provided by the present invention;

[0099] Figure 5 This is a schematic diagram of the visible light image 3D point cloud acquisition process provided by the present invention;

[0100] Figure 6 A schematic diagram of the pyramid strategy for dense matching of visible light images provided by the present invention;

[0101] Figure 7The original image for superpixel segmentation provided by this invention;

[0102] Figure 8 The surface region obtained by superpixel segmentation provided by this invention;

[0103] Figure 9 A schematic diagram illustrating the extraction of 3D point clouds based on the principle of forward intersection provided by this invention;

[0104] Figure 10 This is a schematic diagram of the SAR image three-dimensional point cloud acquisition process provided by the present invention;

[0105] Figure 11 This is a schematic diagram illustrating the working principle of the SAR system provided by the present invention.

[0106] Figure 12 This is a schematic diagram of dense matching of SAR stereo images provided by the present invention;

[0107] Figure 13 A flowchart of the multi-source remote sensing three-dimensional information fusion technology provided by this invention;

[0108] Figure 14 A schematic diagram of the structure of a multi-source remote sensing fusion system based on visible light images and SAR images provided by the present invention;

[0109] Figure 15 This is a schematic diagram of the structure of an electronic device provided by the present invention. Detailed Implementation

[0110] The specific embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0111] Example 1

[0112] This invention provides a multi-source remote sensing fusion method based on visible light imagery and SAR imagery, such as... Figure 1 As shown, it includes:

[0113] S1. Acquire visible light image pairs and SAR image pairs of the same time and space in the target area;

[0114] S2. Perform feature matching and orientation on the visible light image pairs to obtain oriented image pairs. Use a pyramid matching strategy combined with a semi-global algorithm to perform dense matching on the oriented image pairs to obtain a disparity map. Extract visible light point clouds from the disparity map.

[0115] S3. Perform dense matching on the SAR image pairs using object elevation search to obtain SAR point clouds;

[0116] S4. A multi-source digital surface model is generated by performing point cloud raster interpolation on the visible light point cloud and SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. The fused surface model is then converted into a point cloud to obtain the three-dimensional point cloud of the target area.

[0117] In step S1, visible light image pairs and SAR image pairs of the target area in the same time and space are acquired.

[0118] The visible light image pairs and SAR image pairs of the target area acquired by remote sensing equipment are both high-resolution images. The high-resolution visible light remote sensing image data and SAR image data are used for rapid pre-extraction of ground feature information in the target area, providing a foundation for subsequent data fusion.

[0119] The method for acquiring SAR image pairs is as follows: based on the observation requirements of the target area, the SAR stereo image imaging mechanism and the radar response time difference determine the stereo image observation method to acquire SAR image pairs.

[0120] Specifically, considering that SAR stereo imaging using a different-side stereo observation method is not suitable for acquiring high-quality radar stereo images in mountainous areas with significant terrain undulations, the SAR image imaging mechanism should be fully considered, and the acquisition methods of SAR stereo images (such as same-side stereo observation and different-side stereo observation) should be tailored accordingly. Figure 2 As shown, s1 and s2 both represent radar positions, time differences of radar images, etc. Based on the requirements to be measured, the optimal strategy is selected, and SAR image pairs with better stereo observation methods and smaller time differences are selected.

[0121] In step S2, feature matching and orientation are performed on the visible light image pairs to obtain oriented image pairs. A pyramid matching strategy combined with a semi-global algorithm is used to perform dense matching on the oriented image pairs to obtain a disparity map. Visible light point clouds are then extracted from the disparity map.

[0122] Considering the unique image distortion issues caused by CCD linear array pushbroom imaging of visible light images, and taking into account the imaging characteristics of satellite payloads, geometric error types, and error propagation laws, geometric preprocessing is required before information pre-extraction from the visible light images.

[0123] Geometric preprocessing is performed on each visible light image based on tie point matching technology and regional network adjustment and geometric correction technology to obtain the geometrically preprocessed visible light image, thus refining the visible light image imaging model. The geometric preprocessing process includes: studying and utilizing automatic extraction technology of high-quality tie points and control points during regional network adjustment to achieve high-precision automatic matching of multiple satellite images. Based on the visible light image feature matching principle, robust tie points between visible light images are obtained using the classic SIFT algorithm (Scale-Invariant Feature Transform, a scale-space-based image local feature descriptor). Simultaneously, the matching results of multiple-view visible light images within the region are combined (optionally, if it is necessary to fuse images of the same region at different times, the matching results of multiple temporal and multi-view visible light images within the region are combined), along with auxiliary control data. Based on the principles of regional network adjustment and geometric correction, and considering the characteristics of the general RFM (Rational Function Model) imaging model of satellite remote sensing images, geometric correction of visible light satellite images is performed to refine the visible light satellite image imaging model and improve the Earth positioning accuracy of visible light images. Figure 3 and Figure 4 As shown, specifically:

[0124] (1) Using the SIFT algorithm combined with the distance metric function, based on the gray-scale change characteristics and structural information of the visible light image pairs, the feature points of the visible light image pairs are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs.

[0125] (1-1) Based on the grayscale change characteristics and structural information of each visible light image, feature point detection is performed to obtain multiple feature points of each visible light image;

[0126] Optionally, the feature point detection method includes Harris corner detection technology and edge detection technology. Harris corner detection technology can extract corner points in the image based on the gray-level gradient and structural information of the image, while edge detection technology can obtain other key feature points. The combination of the two yields multiple feature points in each visible light image.

[0127] (1-2) Using the SIFT algorithm, a scale-invariant feature description vector is constructed for each feature point based on its position;

[0128] If Harris corner detection is used alone, it cannot resist scale changes. The SIFT algorithm can construct a local feature description vector with scale / rotation invariance based on the feature point positions extracted by corner detection, that is, it can construct a robust matching description that maintains local invariance.

[0129] (1-3) Using a distance metric function based on the feature description vector of each feature point, calculate the global similarity between multiple feature points, and determine multiple matching feature points based on the global similarity as connection points between multiple visible light images.

[0130] The distance metric functions include Euclidean distance function and Mahalanobis distance function.

[0131] Optionally, after determining multiple matching feature points based on global similarity, the method further includes: using classic robust estimation techniques to filter out matching errors caused by ground cover occlusion, texture similarity, etc. in visible light images, to obtain more robust matching feature points, and to provide more accurate connection points for subsequent visible light image regional network adjustment and geometric correction.

[0132] (2) Using the image-side compensation-based RFM regional network adjustment technique, based on the connection points between multiple visible light images, system error compensation is performed on multiple visible light images to obtain multiple directional images with completed error compensation.

[0133] That is, based on the theory of satellite image regional network adjustment in photogrammetry, and according to the fundamental principle that corresponding image points (i.e., the tie points obtained above) intersect at the same object point, regional network adjustment is performed jointly by the visible light image tie points within the region. Regional network adjustment should consider the characteristics of the general RFM imaging model of satellite remote sensing imagery, and use more rigorous image-side compensation-based RFM regional network adjustment techniques to compensate for systematic errors, refine the visible light satellite imagery imaging model, and improve the Earth positioning accuracy of visible light satellite imagery.

[0134] Specifically, the model parameters of the rational function model that introduces image-side compensation parameters are initialized; based on the connection points between multiple visible light images, the imaging geometric relationship of multiple visible light images is constructed using the rational function model (RFM), that is, the mapping relationship between the image-side coordinates and the object-side coordinates of the visible light images;

[0135] For each pair of connection points, an error equation is established based on the image compensation parameters and model parameters. By combining the error equations of multiple pairs of connection points, an overdetermined set of equations is obtained.

[0136] The overdetermined equations are solved using the least squares method with the regional network adjustment technique to obtain the image-side compensation parameters and model parameters. Based on the image-side compensation parameters and model parameters, the final rational function model is determined.

[0137] The object coordinates of the visible light image pair are determined using the final rational function model, and the orientationd image pair is obtained based on the same object coordinates.

[0138] Orientation refers to the precise calculation of the exterior orientation elements (position, attitude) of each image using adjustment techniques (such as RFM + image-side compensation) to ensure strict alignment with the real geographic coordinate system.

[0139] Furthermore, taking into full account the scenario of performing regional network adjustment and geometric correction using visible light satellite imagery combined with auxiliary control data under high positioning accuracy requirements, appropriate auxiliary control data is selected to control errors that cannot be eliminated by regional network adjustment, making the subsequent pre-extracted 3D point cloud closer to the real geographical location, and also providing a basic accuracy guarantee for high-precision 3D modeling, engineering facility target and structure identification, etc.

[0140] Therefore, optionally, during the regional network adjustment process, auxiliary control data can be selected to further compensate for errors that cannot be eliminated by the regional network adjustment. Specifically:

[0141] By analyzing the residuals after regional network adjustment (such as the residual distribution of tie points / control points, error ellipse, and spatial autocorrelation), the types of residual errors and their spatial distribution patterns can be identified.

[0142] Select auxiliary data based on the error type, transform the auxiliary data into constraints for regional network adjustment, and reprocess the visible light image using regional network adjustment to obtain an optimized directional image.

[0143] The auxiliary data includes attitude data and multi-source images, etc. The attitude data is used to correct system drift in time or orbital direction; the multi-source images are used to compensate for atmospheric or cross-sensor differences.

[0144] Meanwhile, taking into full account the characteristics of satellite imagery—large swath width and the inability of conventional geometric transformations to absorb image distortion—this scheme, in order to make the geometric correction of visible light images more stable, includes the following steps before using image-side compensation-based RFM regional network adjustment technology to compensate for systematic errors in visible light images:

[0145] Each visible light image is segmented according to the same specifications to obtain multiple block images for each visible image. Each block image has several connection points to ensure that the visible light image can be locally geometrically corrected in local blocks.

[0146] This completes the geometric preprocessing of the visible light image, resulting in an error-corrected visible light image.

[0147] S2-1. Using a pyramid matching strategy combined with a semi-global algorithm, dense matching is performed on the directional image pairs to obtain a disparity map.

[0148] Due to the wide swath characteristics of satellite imagery and the need for dense 3D point clouds, 3D point cloud reconstruction from visible light imagery requires improving the efficiency of 3D point cloud construction while ensuring the accuracy of the digital surface model. Based on these requirements, this paper proposes to construct visible light image epipolar line image pairs based on the classic optical image 3D reconstruction workflow, fully utilizing epipolar line constraints to accelerate dense matching. Based on the classic semi-global matching algorithm and referencing cutting-edge pyramid strategy acceleration methods, the matching search range is automatically reduced layer by layer, effectively ensuring a small but correct matching search range, improving the overall matching accuracy and stability of the target area satellite imagery, and reducing parameter dependence. Simultaneously, image segmentation technology is used to supplement mismatched and missing points, suppress matching noise, preserve parallax breaks, and constrain adjacent matching results to maintain smoothness as much as possible. Figure 5 As shown.

[0149] 1a) Perform epipolar line correction on the directional image pair to obtain an epipolar line image pair.

[0150] To address the issues of uncertain search range and excessively large search range in pixel-by-pixel dense matching of visible light images, this scheme constructs stereo image pairs of visible light satellite epipolar images to accelerate the dense matching process. By using epipolar line correction, the matching range of a single point is constrained to its epipolar line, providing a foundation for subsequent rapid dense matching.

[0151] Specifically, based on the imaging geometry model of the oriented image pair, a kernel curve equation is constructed;

[0152] The linear equation is simplified using piecewise linear fitting techniques to obtain an approximate kernel curve equation.

[0153] Based on the simplified kernel curve equation, the directional image pair is resampled to obtain the corrected epipolar image pair.

[0154] The above method fully considers the problem of irregular epipolar lines caused by the pushbroom imaging characteristics of visible light satellite imagery. It selects an approximate fitting method to replace the high-order epipolar curve equation, thereby improving the efficiency of epipolar line correction. In the corrected epipolar line image pair, each pixel only needs to search the same row position in another image, thus avoiding searching the entire image.

[0155] Optionally, considering that errors may be introduced during the approximate fitting of the kernel curve, this scheme determines the effect of the kernel line correction by statistically analyzing the distance of the real corresponding points from the kernel line and experimentally determining the distance threshold. Visible light images with poor kernel line correction are appropriately divided into blocks to reduce the error of the approximate fitting of the kernel curve.

[0156] 1b) Using a pyramid matching strategy, each image in the epipolar image pair is divided into multiple resolution levels based on the same specifications. Based on the matching disparity range of the initial low-resolution level in the multiple resolution levels, the optimal disparity of each candidate pixel in the matching disparity range of each level is calculated step by step using a combination of matching cost and a semi-global matching algorithm to obtain the disparity map of the epipolar image pair.

[0157] In other words, this solution addresses the simultaneous requirements of matching speed and accuracy in dense matching of visible light images. Based on the classic semi-global matching algorithm, this solution comprehensively selects appropriate matching costs and explores and utilizes a pyramid matching strategy to accelerate matching, thereby increasing the matching efficiency of wide-swath satellite imagery while ensuring the accuracy of dense matching.

[0158] Specifically, the epipolar image pairs are divided into multi-resolution levels using a pyramid matching strategy to determine the matching disparity range of the initial low-resolution level.

[0159] The pixel similarity of candidate pixels in the matching disparity range is calculated using the matching cost combination to generate an initial cost cube. The matching cost combination includes Census transform cost and NCC (Normalized Cross-Correlation) cost.

[0160] The optimal disparity of the candidate pixels is obtained by optimizing the initial cost cube using a semi-global matching algorithm. Specifically, the semi-global matching algorithm aggregates neighborhood costs through multi-path dynamic programming, introduces smoothness constraints, optimizes the initial cost cube of the initial low-resolution level, and obtains the aggregated cost cube. The optimal disparity of each candidate pixel is then determined based on the cost cube.

[0161] Based on the optimal disparity, the matching disparity range of the next level is determined. The optimal disparity of candidate pixels in the matching disparity range of the next level is determined by combining the matching cost and the semi-global matching algorithm. This process is passed down level by level to determine the optimal disparity of candidate pixels in the matching disparity range of each level.

[0162] The disparity map of the epipolar image pair is obtained based on the optimal disparity of candidate pixels within the matching disparity range of each level.

[0163] Among these, regarding matching cost: To address the slow computation of conventional matching costs, experiments were conducted to obtain a combination of matching costs that guarantees both matching accuracy and computational efficiency, based on classic matching costs such as Census transform cost and NCC cost. To address the large storage space and low matching efficiency caused by the wide swath characteristics of satellite imagery, a pyramid multi-level matching strategy was employed. An appropriate algorithm was designed to automatically reduce the matching search range layer by layer, limiting the matching range of each level within certain limits. Through progressive reduction, the matching disparity range of each pixel was gradually decreased. This ensures that a smaller matching search range is available on the image without missing correct matching results, thus accelerating the matching speed of the target area and reducing gross errors in matching. Figure 6 As shown. To address the problems of high matching noise, uneven matching results, and poor matching performance in textureless areas in traditional photogrammetric matching techniques, a semi-global matching technique is used to constrain the matching results of adjacent pixels to be as smooth and parallel as possible, effectively preserving disparity breaks and maintaining the geometric structure of surface features. At the same time, the semi-global matching technique aggregates neighborhood information through multi-path dynamic programming and introduces smoothness constraints, thereby improving the detail preservation capability of the disparity map in continuous areas.

[0164] 1c) Optionally, the disparity map is segmented using superpixel segmentation technology to obtain multiple regions, and the disparity of the pixels in each region is fitted to a plane to obtain an optimized disparity map.

[0165] Since the point cloud obtained by the preliminary matching method often contains matching noise and mismatches in the difficult matching areas, in order to obtain a smooth matching result that preserves the edges, and at the same time provide more effective information for the subsequent fusion of multi-source remote sensing 3D information, the matching result, i.e., the disparity map, is pre-extracted from the established visible light image 3D information and refined after dense matching using image segmentation methods.

[0166] Image segmentation should fully consider the structural units of ground features, perceive spatial and grayscale proximity, and combine pixels into sub-regions of appropriate size. A suitable segmentation technique for refining matching results is selected. This scheme preferentially uses classic superpixel segmentation techniques such as SLIC (Simple Linear Iterative Clustering) or SEED (Superpixels Extracted via Energy-Driven Sampling) to segment locations such as building rooftops in satellite imagery into spatially connected and dense homogeneous regions, providing high-quality segmentation results for subsequent segmentation-based matching result refinement. Figure 7 and Figure 8As shown, segmentation-based matching result refinement refers to addressing the discrepancies and gaps in the initial matching results. Based on the fundamental principle that homogeneous regions are often coplanar in elevation, it utilizes the image segmentation result map to perform planar fitting on the matching disparity results within each segmentation class. This reduces matching noise, fills in missing pixels caused by erroneous matching, and obtains smoother and more accurate matching results. Simultaneously, considering the "step-like" phenomenon of non-coincident matching boundaries caused by excessively small segmentation, the study and utilization of the image segmentation domain system, selecting an appropriate sampling method, samples boundary pixels from adjacent segments and performs planar fitting together with its own pixels to mitigate the step-like effect.

[0167] S2-2. Extract visible light point cloud from the disparity map. Obtain the corresponding image points of the directional image pair corresponding to the disparity map, and perform forward intersection using the rational function model parameters of the two images and the image-side coordinates of the corresponding image points to obtain the three-dimensional point cloud of the visible light image pair.

[0168] In other words, the project aims to obtain corresponding image points on stereo satellite imagery based on the refined matching results (disparity map). Using the principle of forward intersection, these points will be intersected to the same ground point through collinearity equations, thereby obtaining the three-dimensional coordinates of that point. Based on the RPC (Rational Polynomial Coefficients) parameters of the left and right images of the stereo image pair, and using the image plane coordinates of the corresponding image points on the stereo image pair, direct ground positioning will be performed to obtain the object space coordinates (X, Y, Z) of the corresponding ground point, thus achieving forward intersection based on RFM. Figure 9 As shown, (r l ,c l ) and (r r ,c r ) represent the image plane coordinates of the corresponding image points in the left and right images, respectively.

[0169] Specifically, multiple pairs of corresponding image points of the directional image pairs corresponding to the disparity map are obtained;

[0170] Based on each pair of corresponding image points, a nonlinear error equation is established using the rational function model parameters of the two orientation images and the collinearity equation corresponding to the image coordinates of the corresponding image points. The average value of the regularized translation parameters of the two orientation images is used as the initial value of the object coordinates. The nonlinear error equation is iteratively calculated using a nonlinear least squares algorithm until the convergence condition is met, and the object coordinates of the corresponding image points are obtained.

[0171] The visible light point cloud of the visible light image pair is determined based on the object-space coordinates of each pair of image points with the same name.

[0172] In step S3, the SAR image pairs are densely matched using object elevation search to obtain SAR point clouds.

[0173] To address the problem of 3D information pre-extraction failure due to environmental factors such as clouds and rainfall hindering the acquisition of effective images, high-resolution SAR imagery, which can be acquired 24 / 7, effectively solves the data source issue. Achieving high-precision positioning of SAR imagery is a prerequisite for 3D information pre-extraction. Therefore, this project focuses on breakthroughs in high-precision positioning of SAR imagery, SAR image radiometric quality enhancement, and SAR dense matching, thereby enabling the generation of 3D reconstructed point clouds from SAR imagery based on 3D information pre-extraction of high-resolution SAR imagery. This provides data support for subsequent multi-source 3D information fusion, such as... Figure 10 As shown.

[0174] Unlike optical SAR images, SAR images suffer from two problems: inherent speckle noise degrades their quality, and side-looking imaging characteristics lead to inconsistent feature extents within the same window. These issues challenge dense matching and subsequent point cloud generation. Therefore, geometric preprocessing is necessary before pre-extracting 3D information from SAR images. Geometric preprocessing of SAR images mainly includes two aspects: improving the radiometric quality of SAR images through speckle suppression techniques, and improving the relative and absolute positioning accuracy of spaceborne SAR images through techniques such as geometric calibration and regional network adjustment. Figure 10 As shown.

[0175] Before performing dense matching, the SAR image is geometrically preprocessed: (1) Radiometric quality enhancement—SAR image speckle suppression.

[0176] SAR images suffer from inherent defects in their coherent imaging systems, resulting in a large amount of speckle noise. This speckle noise, a typical form of random multiplicative noise, severely limits the development and application of SAR image interpretation techniques. This scheme, based on the principle of SAR image speckle noise suppression and incorporating edge detection algorithms, accurately identifies homogeneous and heterogeneous regions during the speckle suppression process, improving the effectiveness of traditional speckle suppression algorithms. Addressing the difficulty in accurately estimating the noise mean and variance in traditional anisotropic diffusion methods, the following approach achieves optimal speckle suppression, thereby improving the radiometric quality of SAR images.

[0177] Specifically, based on each SAR image, the Euclidean distance between pixels in the SAR image is calculated, and the homogeneous and heterogeneous regions of the SAR image are determined based on the Euclidean distance between the pixels.

[0178] Adaptively construct the heterogeneous diffusion coefficients for homogeneous and heterogeneous regions, and smooth the noise in the homogeneous and heterogeneous regions based on the heterogeneous diffusion coefficients to obtain SAR images with completed speckle noise suppression.

[0179] (2) Improved absolute positioning accuracy—SAR geometric calibration.

[0180] Based on the working principle of SAR systems, considering the effects of SAR system time delay within the radar system and atmospheric propagation delay along the radar propagation path, the error forms of SAR system time delay, atmospheric propagation delay, etc., are determined, such as... Figure 11 As shown, Figure 11 In this diagram, Trans represents the transmitter, responsible for generating and transmitting electromagnetic waves; fc represents the center frequency of the transmitted signal; Rcvr represents the receiver, responsible for receiving reflected electromagnetic waves; Switch represents a switch, used to switch between transmit and receive modes; Antenna represents the antenna, used for transmitting and receiving electromagnetic waves; Energy pulse represents the energy of the transmitted electromagnetic wave; Target represents the object being detected; Indicator represents an indicator, used to display received signal information; and R represents the distance from the antenna to the target. The atmospheric propagation delay correction model is used to eliminate the errors based on the neutral atmospheric and ionospheric environmental parameters of the target point. For the geometric positioning errors of SAR imagery in the range and azimuth directions, a suitable calibration model is selected, using the range and azimuth directions as calibration parameters. Using corner reflectors deployed in the calibration field, the corresponding geographic coordinates of the corner reflectors and the atmospheric ionospheric content at the corresponding satellite transit time are obtained to solve for the calibration parameters. Specifically:

[0181] Using an atmospheric propagation delay correction model, atmospheric propagation delay corrections are generated based on neutral atmospheric and ionospheric parameters of the target region;

[0182] Based on each SAR image in the SAR image pair, the calibration parameters in the pre-built calibration model are solved by least squares using the difference between the geographic coordinates of the corner reflectors deployed in the calibration field of the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction. The SAR image is then corrected based on the calibration parameters to obtain a geometrically calibrated SAR image.

[0183] The geometric calibration model should have a certain degree of generalization. For SAR images in non-calibrated fields, calibration parameters can also be used for compensation, thereby achieving high-precision positioning of SAR images.

[0184] By introducing atmospheric propagation delay correction and corner reflector-assisted least squares calibration, the geometric accuracy and positioning reliability of SAR images are effectively improved, and the imaging stability under complex atmospheric conditions is enhanced.

[0185] (3) Improved relative positioning accuracy—SAR regional network adjustment.

[0186] Due to the influence of multiplicative noise in SAR images, the traditional SIFT algorithm performs poorly in SAR image registration, resulting in SAR images acquiring very large gradients in static areas with high reflectivity, leading to gradient bias. To address these issues with SIFT, this scheme utilizes the noise characteristics of SAR images and selects feature point detection and description algorithms suitable for multiplicative noise in SAR images. This ensures that the magnitude and direction of the gradient are robust to speckle noise, thereby determining the connection points between SAR images.

[0187] Because SAR imagery differs significantly from visible light imagery in its imaging method, its imaging model also differs. SAR imagery regional network adjustment should be based on a Range-Doppler (RD) model, using the image range-slant range, azimuth flight time, and the geographic coordinates of the tie points as the adjustment parameters to be solved, with the image coordinates of the tie points as known observations. The project proposes to use the condition that image points with the same name at tie points should correspond to the same ground coordinates to connect regional images as a whole, constructing error equations based on the RD model, with the image range-slant range, azimuth flight time, and the geographic coordinates (longitude, latitude, and elevation) of the tie points as unknown parameters. Two error equations can be listed for each tie point coordinate. All error equations are listed and represented in matrix form, then solved using an optimization algorithm to obtain the azimuth flight time and range-slant range corrections for each image, and the geographic coordinates of each tie point.

[0188] By adjusting the regional network based on the range-Doppler model, the geographic coordinates of each connection point are jointly solved and optimized to form consistent three-dimensional ground coordinates. SAR image pairs are spatially registered through common connection points, eliminating systematic geometric deviations between images.

[0189] At this point, we have obtained a SAR image pair with completed geometric preprocessing.

[0190] The SAR image pairs are densely matched using object elevation search to obtain SAR point clouds.

[0191] Specifically, this scheme addresses the characteristics of SAR image side-looking imaging and the weak intersection angle of stereo image pairs, as well as the need for 3D information extraction from SAR image data at different time phases and angles. The dense matching of SAR stereo images employs a matching strategy based on direct object-side elevation search to accelerate the matching efficiency of SAR image data at different time phases and angles. Figure 12 As shown.

[0192] Specifically, multiple candidate elevation values ​​are initialized based on the terrain features of the target area, and multiple grid points representing candidate locations are established in the object space of the target area;

[0193] Based on each grid point representing a candidate location, multiple elevation candidate values ​​are traversed, and the grid points are projected onto the SAR image pair to obtain the corresponding image point coordinates; based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point.

[0194] The SAR point cloud is obtained based on the object coordinates of each grid point in the image point coordinates and the true elevation of the SAR image pair.

[0195] Optionally, in the above matching process, to address the issue of large side-view tilt angles in SAR image imaging and sensitivity to geometric deformation between images, and to avoid inconsistencies in grayscale information during matching of SAR images with large tilt angles, an adaptively corrected matching window (grid points) is adopted. That is, based on the basic principles of correlation matching, the grayscale information within the matching window of two images is matched using an adaptive window, and the optimal adaptive adjustment strategy is selected to ensure that the grayscale information within matching windows of the same size in both images is consistent.

[0196] In step S4, a multi-source digital surface model is generated by performing point cloud raster interpolation on the visible light point cloud and SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. The fused surface model is then converted into a point cloud to obtain the three-dimensional point cloud of the target area.

[0197] Digital surface models (DSMs) are regularly stored "3D point clouds." For the Earth's surface, the difference between DSMs and 3D point clouds is minimal, and they are often interchangeable in engineering practice. The use of regular grids in DSMs for multi-point cloud fusion has been validated by some research. Considering the difficulty of directly fusing multiple sources in 3D point cloud form, and the fact that many open-source elevation products are regularly stored digital ground models / digital elevation models, which are not conducive to importing information obtained through other means, this paper considers constructing a multi-source DSM based on multi-source point clouds, combining it with information obtained through other means, and performing elevation fusion within regular grid cells. Finally, the DSM is used as an intermediate fusion medium, and the fused DSM and original imagery are combined to construct a color 3D point cloud that integrates multi-source remote sensing 3D information, serving subsequent engineering facility target and structure identification. Specifically, this is achieved through two stages: the construction and fusion of multi-source DSMs, and the construction of a fused 3D point cloud based on the fused DSM. Figure 13 As shown.

[0198] S4-1. A multi-source digital surface model is generated by performing point cloud rasterization interpolation on the visible light point cloud and SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, a multi-source elevation fusion is performed to obtain a fused digital surface model.

[0199] In summary, for 3D point clouds from different time phases, spatial locations, and imaging sources (visible light / SAR), following the point cloud fusion technology route based on multi-source fusion digital surface models, this approach integrates the advantages of multiple 3D point clouds, fully utilizes the 3D point clouds from each source, and constructs digital surface models for each 3D point cloud from each time phase, spatial location, and imaging source (visible light / SAR). Then, the constructed multiple digital surface models are registered and fused to obtain an accurate fused digital surface model, providing support for the subsequent construction of fused 3D point clouds. The steps include: multi-source digital surface model construction, multi-source digital surface model registration, and multi-source digital surface model fusion.

[0200] 1a) Using digital surface models to construct interpolation methods, interpolation processing is performed on the models generated after the visible light point cloud and SAR point cloud are rasterized to obtain the visible light digital surface model and the SAR digital surface model.

[0201] Specifically, a point cloud is rasterized using visible light point cloud to generate an initial digital surface model, and the empty areas in the initial digital surface model are marked.

[0202] Multiple non-empty regions are selected, and nearest neighbor interpolation, bilinear interpolation, and inverse distance interpolation are combined with the visible light point cloud to interpolate and fill in each non-empty region to obtain the interpolated region;

[0203] Calculate the elevation deviation between the interpolation region and the visible light point cloud for each interpolation method, and select the interpolation method corresponding to the minimum elevation deviation;

[0204] By using the interpolation method corresponding to the minimum elevation deviation and combining it with the visible light point cloud, the missing area is interpolated to obtain a visible light digital surface model.

[0205] Similarly, the SAR digital surface model corresponding to the SAR point cloud is obtained, and the visible light digital surface model and the SAR digital surface model together constitute a multi-source digital surface model.

[0206] Optionally, before constructing the digital surface model, a suitable geographic coordinate range can be selected as the range of the digital surface model by referring to the geographic distribution of each point cloud, so that the three-dimensional information of the point cloud is not omitted. Taking into account the resolution of the source image of the three-dimensional point cloud and the size of the target to be observed later, the resolution of the digital surface model can be unified in the construction stage to ensure that the subsequent fusion of digital surface models can be carried out more effectively, and to provide a foundation for the fusion of three-dimensional information.

[0207] For each point cloud that is irregularly distributed on the earth's surface in the form of three-dimensional coordinates of longitude, latitude, and elevation, the requirements of multiple aspects such as the range, resolution, and construction method of the digital surface model are fully considered. The three-dimensional information of the point cloud is regularized into a raster form, providing a suitable fusion basis for three-dimensional information fusion.

[0208] 1b) Select corresponding control points in the multi-source digital surface model, solve the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system by least squares fitting, and perform coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model under the same target geographic coordinate system.

[0209] The multi-source digital surface model under the same target geographic coordinate system is transformed from the original elevation to the preset target elevation datum to obtain the final multi-source digital surface model.

[0210] In this scheme, to eliminate geographical location deviations caused by satellite imagery positioning model errors and point cloud reconstruction errors in multiple digital surface models and to obtain positioning accuracy as close as possible to the real geographic coordinates, the scheme fully considers the elevation similarity between multiple digital surface models, references control data and other intelligence, and comprehensively utilizes image matching, coordinate correction and other methods to unify all digital surface models under the same geographic coordinate system. This ensures that the elevations of multi-source point clouds are corrected to the same raster pixel position / geographic location, guaranteeing elevation information fusion. By eliminating multi-source data system deviations through a unified coordinate framework and elevation benchmark, and achieving adaptive elevation fusion based on reliability weights, the geometric accuracy of the generated fused digital surface model is improved, while effectively preserving the advantageous characteristics of each data source.

[0211] To address the issue of different elevation datums for 3D point clouds from multi-source satellite imagery, this study fully considers the starting position of the elevation values ​​of the multi-source 3D point clouds and, with reference to engineering practice experience, selects a widely accepted and convenient elevation datum as the elevation datum for the digital surface model. This transforms the elevations of all digital surface models to the same elevation datum, avoiding fusion errors caused by inconsistent elevation datums.

[0212] 1c) Using the reliable elevation values ​​of each point in the visible light point cloud and SAR point cloud determined by image matching cost and coherence coefficient, and combining them with a pre-calibrated multi-source weight ratio, elevation fusion is performed on the final multi-source digital surface model to generate a fused digital surface model. Specifically:

[0213] Using image matching cost and coherence coefficient, outliers in the visible light 3D point cloud and SAR 3D point cloud are detected respectively, and the reliable elevation value weight of each point outside the outlier is determined. The elevation value of the outlier is the unreliable elevation value.

[0214] Based on the reliable elevation value weight of each point outside the anomaly point, and the pre-experimentally calibrated weight ratio of the visible light 3D point cloud to the SAR 3D point cloud, the elevation values ​​in the visible light digital surface model and the SAR digital surface model are used to perform elevation fusion to obtain a fused digital surface model.

[0215] Before fusing multi-source digital surface models, preprocessing of the multi-source digital surface models is performed. Since each registered multi-source digital surface model corresponds to an elevation value provided by one or more different source point clouds at each grid location (i.e., geographical location), the fusion of multi-source 3D information is converted into pixel-by-pixel elevation value fusion under a regular grid. For grid locations with only a single elevation value or whose elevation value is an invalid hole, no processing is performed to avoid disrupting the geometric structure of the 3D information as much as possible. For grid locations with multiple redundant elevation values, considering the principles of point cloud construction, the probability that each source elevation is close to the true elevation is estimated through point matching reliability and elevation statistical properties. Multiple elevation values ​​are then fused to obtain a final unique elevation value, completing the preprocessing before multi-source digital surface model fusion.

[0216] S4-2. Perform point cloud conversion on the fused surface model to obtain the three-dimensional point cloud of the target region.

[0217] Based on a fused digital surface model, and taking into account the specific characteristics of ground objects, a 3D spatial point cloud is constructed to ensure the geometric integrity of the 3D point cloud. Subsequently, considering the requirements for grayscale / color information in the point cloud, and taking into full account the differences in grayscale / color information caused by the imaging radiation environment and imaging angles of multi-temporal, multi-source, and multi-view satellite imagery, the relationship between the fused reference geographic / elevation coordinate system and the original geographic / elevation coordinate system of the image is utilized to endow the 3D spatial point cloud with image texture information, ensuring rich texture information in the 3D point cloud.

[0218] (1) 3D Point Cloud Construction. The fused digital surface model is converted into a 3D point cloud expressed in 3D spatial coordinates, and the raster elevation is converted to x, y, z spatial coordinates pixel by pixel. Since the fused digital surface model is derived from georegistered multi-source digital surface models, the resulting 3D point cloud does not require further registration or fusion operations. However, considering that a few side point clouds may be missed during the construction of the digital surface model due to the regularization of the original point cloud, appropriate strategies such as image segmentation or expert knowledge assistance are needed to supplement the missing point clouds in the conversion of the fused digital surface model into a 3D point cloud, so as to obtain a more complete 3D spatial point cloud.

[0219] Specifically, the fused digital surface model is converted pixel by pixel from raster elevation to three-dimensional spatial coordinates to generate an initial fused three-dimensional point cloud;

[0220] The initial fused 3D point cloud is interpolated to obtain a fused 3D point cloud, including:

[0221] Point cloud density analysis was performed on the initial fused 3D point cloud to detect point cloud void regions;

[0222] Image segmentation technology is used to perform semantic segmentation and connected component analysis on the visible light image pair and SAR image pair to obtain multiple connected components. Candidate regions are obtained by filtering the connected components based on preset filtering rules.

[0223] The candidate regions of the visible light image pair and the SAR image pair are spatially indexed and associated with the point cloud hole regions of the initial fused 3D point cloud. The point cloud hole regions are then filtered to obtain the point cloud regions to be supplemented and the key regions in the visible light image pair and the SAR image pair.

[0224] In the key area, feature matching is performed using visible light image pairs and SAR image pairs to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a new point cloud.

[0225] The newly added point cloud is used to interpolate the area of ​​the point cloud to be supplemented to obtain a fused 3D point cloud.

[0226] Point cloud density analysis primarily locates the extent of missing geometric data, but it cannot directly determine whether these areas belong to targets requiring completion, such as building facades (which could be ground, vegetation, etc.). Image segmentation assists by extracting key areas (such as building edges) through semantic information and further filtering effective candidate areas within the missing regions. That is, if point cloud holes exist in key areas, they are identified as effective candidate areas, avoiding unnecessary completion of non-candidate areas and improving the accuracy and relevance of the completion.

[0227] (2) Considering the actual use scenario of fused point cloud for grayscale / color information of point cloud, the texture information of visible light / SAR image is comprehensively considered to obtain the projection relationship between point cloud and image, so as to provide grayscale / color information for each spatial point cloud in the fused 3D point cloud, so as to ensure that more significant features can be extracted based on 3D point cloud.

[0228] The project should incorporate the geometric registration and transformation relationships during the digital surface model fusion process, comprehensively considering the relationship between the fusion reference geographic / elevation coordinate system and the original geographic / elevation coordinate system of the image, to back-calculate the image pixel positions from the 3D point cloud to the image from which the information is to be extracted. Considering that each 3D point cloud position may correspond to multiple temporal, multi-source, and multi-view satellite images, and that different satellite images may have different grayscale / color information due to differences in imaging radiometric environment and imaging angle, common image grayscale unification algorithms such as image radiometric correction and filtering should be used to fuse multi-view images and obtain unique color values ​​for each band.

[0229] Specifically, the coordinate system transformation process in the construction of the digital surface model is reversed to obtain the correspondence between the points in the fused 3D point cloud and the corresponding pixels in the original images.

[0230] Radiometric correction and Walls filtering are performed on multiple original images to obtain grayscale normalized original images, and the global weight of each original image is determined based on the resolution and texture complexity of each original image.

[0231] Based on the global weight and pixel value of each original image, and the correspondence between the points of the fused 3D point cloud and the pixels in each original image, the pixel value of each point in each band of the fused 3D point cloud is calculated respectively. Each original image includes the visible light image pair and the SAR image pair.

[0232] The fused 3D point cloud is obtained by coloring the fused 3D point cloud based on the pixel values ​​of each point in each band.

[0233] Meanwhile, considering that the texture information of visible light / SAR images may have their own uses, the project retains the information of visible light / SAR images while extracting grayscale / color information from the visible light / SAR images to avoid information loss due to fusion.

[0234] Example 2

[0235] Based on the same inventive concept, this invention also provides a multi-source remote sensing fusion system based on visible light images and SAR images, such as... Figure 14 As shown, it includes:

[0236] The data acquisition module is used to acquire visible light image pairs and SAR image pairs of the same time and space in the target area;

[0237] The visible light point cloud extraction module is used to perform feature matching and orientation on the visible light image pairs to obtain oriented image pairs, and to perform dense matching on the oriented image pairs using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and to extract visible light point clouds from the disparity map.

[0238] The SAR point cloud extraction module is used to perform dense matching of the SAR image pairs using object-space elevation search to obtain SAR point clouds.

[0239] The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud raster interpolation on the visible light point cloud and SAR point cloud, and after unifying the coordinates and elevations of the multi-source digital surface model, perform multi-source elevation fusion to obtain a fused digital surface model; and perform point cloud transformation on the fused surface model to obtain the three-dimensional point cloud of the target area.

[0240] In one possible implementation, the aforementioned visible light point cloud extraction module is specifically used for:

[0241] Epipolar line correction is performed on the directional image pairs to obtain epipolar line image pairs. The pyramid matching strategy is used to divide the epipolar line image pairs into multi-resolution levels and determine the matching disparity range of the initial low-resolution level.

[0242] The pixel similarity of candidate pixels in the matching disparity range is calculated by combining the matching costs, and an initial cost cube is generated.

[0243] The optimal disparity of the candidate pixel is obtained by optimizing the initial cost cube using a semi-global matching algorithm. Based on the optimal disparity, the matching disparity range of the next level is determined. The optimal disparity of the candidate pixel in the matching disparity range of the next level is determined by combining the matching cost with the semi-global matching algorithm. This process is passed down level by level to determine the optimal disparity of the candidate pixel in the matching disparity range of each level.

[0244] The disparity map of the epipolar image pair is obtained based on the optimal disparity of candidate pixels within the matching disparity range of each level.

[0245] In one possible implementation, the aforementioned visible light point cloud extraction module is specifically used for:

[0246] Obtain multiple pairs of identical image points corresponding to the directional image pairs in the disparity map;

[0247] Based on each pair of corresponding image points, a nonlinear error equation is established using the rational function model parameters of the two orientation images and the collinearity equation corresponding to the image coordinates of the corresponding image points. The average value of the regularized translation parameters of the two orientation images is used as the initial value of the object coordinates. The nonlinear error equation is iteratively calculated using a nonlinear least squares algorithm until the convergence condition is met, and the object coordinates of the corresponding image points are obtained.

[0248] The visible light point cloud of the visible light image pair is determined based on the object-space coordinates of each pair of image points with the same name.

[0249] In one possible implementation, the aforementioned visible light point cloud extraction module is specifically used for:

[0250] By using the SIFT algorithm combined with a distance metric function, based on the grayscale change characteristics and structural information of visible light image pairs, feature points of the visible light image pairs are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs.

[0251] Initialize the model parameters of the rational function model that introduces image-square compensation parameters;

[0252] For each pair of connection points, an error equation is established based on the image compensation parameters and model parameters. By combining the error equations of multiple pairs of connection points, an overdetermined set of equations is obtained.

[0253] The overdetermined system of equations is solved using regional network adjustment techniques to obtain the image-side compensation parameters and model parameters. Based on the image-side compensation parameters and model parameters, the final rational function model is determined.

[0254] The object coordinates of the visible light image pair are determined using the final rational function model, and the orientationd image pair is obtained based on the same object coordinates.

[0255] In one possible implementation, the aforementioned SAR point cloud extraction module is further used for:

[0256] Based on each SAR image in the SAR image pair:

[0257] Using an atmospheric propagation delay correction model, atmospheric propagation delay corrections are generated based on neutral atmospheric and ionospheric parameters of the target region;

[0258] Based on each SAR image in the SAR image pair, the calibration parameters in the pre-built calibration model are solved by least squares using the difference between the geographic coordinates of the corner reflectors deployed in the calibration field of the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction. The SAR image is then corrected based on the calibration parameters to obtain a geometrically calibrated SAR image.

[0259] In one possible implementation, the aforementioned SAR point cloud extraction module is specifically used for:

[0260] Multiple candidate elevation values ​​are initialized based on the terrain features of the target area, and multiple grid points representing the candidate locations are established in the object space of the target area.

[0261] Based on each grid point representing a candidate location, multiple elevation candidate values ​​are traversed, and the grid points are projected onto the SAR image pair to obtain the corresponding image point coordinates; based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point.

[0262] The SAR point cloud is obtained based on the object coordinates of each grid point in the image point coordinates and the true elevation of the SAR image pair.

[0263] In one possible implementation, the aforementioned point cloud fusion module is specifically used for:

[0264] In the multi-source digital surface model, corresponding control points are selected, and the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system are solved by least squares fitting. Based on the coordinate transformation parameters, coordinate transformation is performed to obtain a multi-source digital surface model under the same target geographic coordinate system.

[0265] The multi-source digital surface model under the same target geographic coordinate system is transformed from the original elevation to the preset target elevation datum to obtain the final multi-source digital surface model;

[0266] Using the reliable elevation values ​​of each point in the visible light point cloud and SAR point cloud determined by the image matching cost and coherence coefficient, and combined with the pre-calibrated multi-source weight ratio, the final multi-source digital surface model is subjected to elevation fusion to generate a fused digital surface model.

[0267] In one possible implementation, the aforementioned point cloud fusion module is specifically used for:

[0268] The fused digital surface model is converted from grid elevation to three-dimensional spatial coordinates pixel by pixel to generate an initial fused three-dimensional point cloud. The initial fused three-dimensional point cloud is then interpolated to obtain a fused three-dimensional point cloud.

[0269] Radiometric correction and Walls filtering are performed on multiple original images to obtain grayscale normalized original images, and the global weight of each original image is determined based on the resolution and texture complexity of each original image.

[0270] Based on the global weight and pixel value of each original image, and the correspondence between the points of the fused 3D point cloud and the pixels in each original image, the pixel value of each point in each band of the fused 3D point cloud is calculated respectively. Each original image includes the visible light image pair and the SAR image pair.

[0271] The fused 3D point cloud is obtained by coloring the fused 3D point cloud based on the pixel values ​​of each point in each band.

[0272] In one possible implementation, the aforementioned point cloud fusion module is specifically used for:

[0273] Point cloud density analysis was performed on the initial fused 3D point cloud to detect point cloud void regions;

[0274] Image segmentation technology is used to perform semantic segmentation and connected component analysis on the visible light image pairs and SAR image pairs, and candidate regions are obtained by combining screening rules.

[0275] The candidate region is spatially associated with the point cloud hole region by spatial indexing to obtain the point cloud region to be supplemented and the key regions in the visible light image pair and SAR image pair.

[0276] In the key area, feature matching is performed using the visible light image pairs and SAR image pairs to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a new point cloud.

[0277] The newly added point cloud is used to interpolate the area of ​​the point cloud to be supplemented to obtain a fused 3D point cloud.

[0278] Example 3

[0279] like Figure 15 As shown, the present invention also provides an electronic device, which may be a computer device, a microcontroller device, a smart mobile device, etc. The electronic device in this embodiment may include a processor, a memory, a transceiver component, etc. The memory, processor, and transceiver component are connected via a bus; the memory can be used to store executable programs, and an exemplary executable program may include instructions; the processor is used to execute the instructions stored in the memory. The memory can also be used to store data, which can be accessed and / or modified when instructions are executed.

[0280] The processor may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, and it is suitable for implementing one or more instructions. Specifically, it is suitable for loading and executing one or more instructions in the storage medium to realize the corresponding method flow or corresponding function, so as to realize the steps of the multi-source remote sensing fusion method based on visible light images and SAR images in the above embodiments.

[0281] Example 4

[0282] Based on the same inventive concept, this invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory). This readable storage medium is a memory device within an electronic device used to store programs and data. It is understood that the storage medium here can include both built-in storage media within the electronic device and extended storage media supported by the electronic device. The storage medium provides storage space, which stores the terminal's operating system. Furthermore, this storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more executable programs (including program code). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage device. Loading and executing one or more instructions stored in the storage medium by the processor can implement the steps of a multi-source remote sensing fusion method based on visible light imagery and SAR imagery in the above embodiments.

[0283] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0284] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0285] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0286] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0287] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit its scope of protection. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that after reading the present invention, they can still make various changes, modifications or equivalent substitutions to the specific implementation methods of the application, but these changes, modifications or equivalent substitutions are all within the scope of protection of the claims pending approval.

Claims

1. A multi-source remote sensing fusion method based on visible light imagery and SAR imagery, characterized in that, include: Acquire visible light image pairs and SAR image pairs of the same time and space in the target area; The visible light image pairs are subjected to feature matching and orientation to obtain oriented image pairs. The oriented image pairs are densely matched using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map. Visible light point clouds are extracted from the disparity map. The SAR image pairs are densely matched using object elevation search to obtain SAR point clouds; A multi-source digital surface model is generated by interpolating the visible light point cloud and SAR point cloud through point cloud rasterization. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. This includes: selecting corresponding control points in the multi-source digital surface model; solving the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system through least squares fitting; performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model under the same target geographic coordinate system; transforming the multi-source digital surface model under the same target geographic coordinate system from the original elevation to a preset target elevation benchmark to obtain the final multi-source digital surface model; determining the reliable elevation value weight of each point in the visible light point cloud and SAR point cloud using image matching cost and coherence coefficient; combining the pre-calibrated multi-source weight ratio; and performing elevation fusion on the final multi-source digital surface model to generate a fused digital surface model. The fused digital surface model is converted into a point cloud to obtain a three-dimensional point cloud of the target region.

2. The method as described in claim 1, characterized in that, The process of using a pyramid matching strategy combined with a semi-global algorithm to perform dense matching on the directional image pairs to obtain a disparity map includes: Epipolar line correction is performed on the directional image pairs to obtain epipolar line image pairs. The pyramid matching strategy is used to divide the epipolar line image pairs into multi-resolution levels and determine the matching disparity range of the initial low-resolution level. The pixel similarity of candidate pixels in the matching disparity range is calculated by combining the matching costs, and an initial cost cube is generated. The optimal disparity of the candidate pixel is obtained by optimizing the initial cost cube using a semi-global matching algorithm. Based on the optimal disparity, the matching disparity range of the next level is determined. The optimal disparity of the candidate pixel in the matching disparity range of the next level is determined by combining the matching cost with the semi-global matching algorithm. This process is passed down level by level to determine the optimal disparity of the candidate pixel in the matching disparity range of each level. The disparity map of the epipolar image pair is obtained based on the optimal disparity of candidate pixels within the matching disparity range of each level.

3. The method as described in claim 1 or 2, characterized in that, Extracting visible light point clouds from the disparity map includes: Obtain multiple pairs of identical image points corresponding to the directional image pairs in the disparity map; Based on each pair of corresponding image points, a nonlinear error equation is established using the rational function model parameters of the two orientation images and the collinearity equation corresponding to the image coordinates of the corresponding image points. The average value of the regularized translation parameters of the two orientation images is used as the initial value of the object coordinates. The nonlinear error equation is iteratively calculated using a nonlinear least squares algorithm until the convergence condition is met, and the object coordinates of the corresponding image points are obtained. The visible light point cloud of the visible light image pair is determined based on the object-space coordinates of each pair of image points with the same name.

4. The method as described in claim 1, characterized in that, The step of performing feature matching and orientation on the visible light image pairs to obtain oriented image pairs includes: By using the SIFT algorithm combined with a distance metric function, based on the grayscale change characteristics and structural information of visible light image pairs, feature points of the visible light image pairs are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs. Initialize the model parameters of the rational function model that introduces image-square compensation parameters; For each pair of connection points, an error equation is established based on the image compensation parameters and model parameters. By combining the error equations of multiple pairs of connection points, an overdetermined set of equations is obtained. The overdetermined equations are solved using regional network adjustment techniques to obtain the image-side compensation parameters and model parameters. Based on the image-side compensation parameters and model parameters, the final rational function model is determined. The object coordinates of the visible light image pair are determined using the final rational function model, and the orientationd image pair is obtained based on the same object coordinates.

5. The method as described in claim 1, characterized in that, Before performing dense matching of the SAR image pairs using object elevation search, the method further includes: Using an atmospheric propagation delay correction model, atmospheric propagation delay corrections are generated based on neutral atmospheric and ionospheric parameters of the target region; Based on each SAR image in the SAR image pair, the calibration parameters in the pre-built calibration model are solved by least squares using the difference between the geographic coordinates of the corner reflectors deployed in the calibration field of the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction. The SAR image is then corrected based on the calibration parameters to obtain a geometrically calibrated SAR image.

6. The method as described in claim 1 or 5, characterized in that, The method of using object-space elevation search to perform dense matching on the SAR image pairs to obtain SAR point clouds includes: Multiple candidate elevation values ​​are initialized based on the terrain features of the target area, and multiple grid points representing the candidate locations are established in the object space of the target area. Based on each grid point representing a candidate location, multiple elevation candidate values ​​are traversed, and the grid points are projected onto the SAR image pair to obtain the corresponding image point coordinates; based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point. The SAR point cloud is obtained based on the object coordinates of each grid point in the image point coordinates and the true elevation of the SAR image pair.

7. The method as described in claim 1, characterized in that, The step of performing point cloud conversion on the fused digital surface model to obtain the three-dimensional point cloud of the target region includes: The fused digital surface model is converted from grid elevation to three-dimensional spatial coordinates pixel by pixel to generate an initial fused three-dimensional point cloud. The initial fused three-dimensional point cloud is then interpolated to obtain a fused three-dimensional point cloud. Radiometric correction and Walls filtering are performed on multiple original images to obtain grayscale normalized original images, and the global weight of each original image is determined based on the resolution and texture complexity of each original image. Based on the global weight and pixel value of each original image, and the correspondence between the points of the fused 3D point cloud and the pixels in each original image, the pixel value of each point in each band of the fused 3D point cloud is calculated respectively. Each original image includes the visible light image pair and the SAR image pair. The fused 3D point cloud is obtained by coloring the fused 3D point cloud based on the pixel values ​​of each point in each band.

8. The method as described in claim 7, characterized in that, The process of interpolating and supplementing the initial fused 3D point cloud to obtain the fused 3D point cloud includes: Point cloud density analysis was performed on the initial fused 3D point cloud to detect point cloud void regions; Image segmentation technology is used to perform semantic segmentation and connected component analysis on the visible light image pairs and SAR image pairs, and candidate regions are obtained by combining screening rules. The candidate region is spatially associated with the point cloud hole region by spatial indexing to obtain the point cloud region to be supplemented and the key regions in the visible light image pair and SAR image pair. In the key area, feature matching is performed using the visible light image pairs and SAR image pairs to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a new point cloud. The newly added point cloud is used to interpolate the area of ​​the point cloud to be supplemented to obtain a fused 3D point cloud.

9. A multi-source remote sensing fusion system based on visible light imagery and SAR imagery, characterized in that, include: The data acquisition module is used to acquire visible light image pairs and SAR image pairs of the same time and space in the target area; The visible light point cloud extraction module is used to perform feature matching and orientation on the visible light image pairs to obtain oriented image pairs, and to perform dense matching on the oriented image pairs using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and to extract visible light point clouds from the disparity map. The SAR point cloud extraction module is used to perform dense matching of the SAR image pairs using object-space elevation search to obtain SAR point clouds. The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. This includes: selecting corresponding control points in the multi-source digital surface model; solving the coordinate transformation parameters from the corresponding control points to the target geographic coordinate system using least squares fitting; performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model under the same target geographic coordinate system; transforming the multi-source digital surface model under the same target geographic coordinate system from its original elevation to a preset target elevation benchmark to obtain the final multi-source digital surface model; determining the reliable elevation value weight of each point in the visible light point cloud and SAR point cloud using image matching cost and coherence coefficient; combining this with a pre-calibrated multi-source weight ratio; and performing elevation fusion on the final multi-source digital surface model to generate the fused digital surface model. The fused digital surface model is converted into a point cloud to obtain a three-dimensional point cloud of the target region.

Images