Multi-source remote sensing fusion method and system based on visible light image and SAR image

Abstract

The invention provides a multi-source remote sensing fusion method and system based on visible light images and SAR (Synthetic Aperture Radar) images. The multi-source remote sensing fusion method comprises the following steps: acquiring a visible light image pair and an SAR image pair of a target area in the same time and space; carrying out feature matching and orientation on the visible light image pair to obtain an oriented image pair, carrying out dense matching on the oriented image pair by utilizing a pyramid matching strategy in combination with a semi-global algorithm to obtain a disparity map, and extracting visible light point cloud from the disparity map; carrying out dense matching on the SAR image pair by utilizing object space elevation search to obtain an SAR point cloud; the method comprises the following steps: generating a multi-source digital surface model by performing point cloud rasterizing interpolation on visible light point cloud and SAR point cloud, unifying coordinates and elevations of the multi-source digital surface model, and performing multi-source elevation fusion to obtain a fused digital surface model; and performing point cloud conversion on the fusion surface model to obtain a three-dimensional point cloud of the target area, thereby realizing accuracy and generalization of multi-source data fusion.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-source remote sensing fusion, and in particular to a multi-source remote sensing fusion method and system based on visible light images and SAR images. Background Art

[0002] With the rapid development of Earth observation technology, modern remote sensing systems have established a three-dimensional monitoring network that integrates space, air, and land. Multi-platform, multi-sensor collaborative observation systems enable dynamic monitoring capabilities from the macro to the microscopic, around-the-clock, and in all weather conditions, providing unprecedented data support for applications such as resource surveys, environmental monitoring, and disaster warning. In particular, the networking and operation of Earth observation systems such as the Gaofen series of satellites and the Sentinel satellites has significantly advanced the acquisition of high-temporal and spatial resolution remote sensing data, laying a solid foundation for the automated production of three-dimensional geographic information products. However, fully leveraging the synergistic advantages of multi-source, heterogeneous remote sensing data remains a key technical challenge that urgently needs to be addressed in the field of remote sensing information processing.

[0003] Due to significant differences in imaging mechanisms, multi-source remote sensing equipment produces diverse images with varying physical meanings, spatial resolutions, measurement ranges, and error characteristics. This makes it difficult to effectively integrate these heterogeneous data during the fusion process, severely limiting the accuracy and integrity of the target area's geographic information extracted from the fused data. Furthermore, multi-source remote sensing data typically contains vast amounts of information, resulting in low data fusion efficiency. Summary of the Invention

[0004] To overcome the above-mentioned deficiencies of the prior art, the present invention proposes a multi-source remote sensing fusion method based on visible light images and SAR images, comprising:

[0005] Obtain visible light image pairs and SAR image pairs of the target area at the same time and space;

[0006] performing feature matching and orientation on the visible light image pair to obtain an oriented image pair, performing dense matching on the oriented image pair using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extracting a visible light point cloud from the disparity map;

[0007] densely matching the SAR image pairs using object space elevation search to obtain a SAR point cloud;

[0008] A multi-source digital surface model is generated by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. Point cloud conversion is performed on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0009] Optionally, the using a pyramid matching strategy in combination with a semi-global algorithm to perform dense matching on the directional image pairs to obtain a disparity map includes:

[0010] Performing epipolar correction on the directional image pair to obtain an epipolar image pair, dividing the epipolar image pair into multiple resolution levels using a pyramid matching strategy, and determining a matching disparity range of an initial low-resolution level;

[0011] The matching cost combination is used to calculate the pixel similarity of candidate pixels in the matching disparity range and generate the initial cost cube;

[0012] Optimizing the initial cost cube using a semi-global matching algorithm to obtain the optimal disparity of the candidate pixel point, determining the matching disparity range of the next level based on the optimal disparity, and determining the optimal disparity of the candidate pixel point in the matching disparity range of the next level using a matching cost combination combined with the semi-global matching algorithm, and transferring the results step by step to determine the optimal disparity of the candidate pixel point in the matching disparity range of each level;

[0013] A disparity map of the epipolar image pair is obtained based on the optimal disparity of the candidate pixel points in the matching disparity range of each level.

[0014] Optionally, extracting a visible light point cloud from the disparity map includes:

[0015] Obtain multiple pairs of image points with the same name in the directional image pairs corresponding to the disparity map;

[0016] Based on each pair of homonymous image points, a nonlinear error equation is established using the rational function model parameters of the two directional images and the collinearity equation corresponding to the image-space coordinates of the homonymous image points; the average value of the regularized translation parameters of the two directional images is used as the initial value of the object-space coordinates, and the nonlinear error equation is iteratively calculated in combination with a nonlinear least squares algorithm until a convergence condition is reached, thereby obtaining the object-space coordinates of the homonymous image points;

[0017] A visible light point cloud of the visible light image pair is determined based on the object space coordinates of each pair of image points with the same name.

[0018] Optionally, performing feature matching and orientation on the visible light image pair to obtain an oriented image pair includes:

[0019] Using the SIFT algorithm combined with the distance metric function, based on the grayscale change characteristics and structural information of the visible light image pair, the feature points of the visible light image pair are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs;

[0020] Initialize the model parameters of the rational function model that introduces image space compensation parameters;

[0021] Establishing an error equation based on the image space compensation parameters and the model parameters for each pair of connection points, and combining the error equations of multiple pairs of connection points to obtain an overdetermined equation group;

[0022] Solving the overdetermined set of equations using a block adjustment technique to obtain image square compensation parameters and model parameters, and determining a final rational function model based on the image square compensation parameters and model parameters;

[0023] The object space coordinates of the visible light image pair are determined using the final rational function model, and a pair of oriented images with completed orientation is obtained based on the same object space coordinates.

[0024] Optionally, before performing dense matching on the SAR image pairs using object-space elevation search, the method further includes:

[0025] The atmospheric propagation delay correction model is used to generate the atmospheric propagation delay correction based on the neutral atmosphere and ionosphere parameters of the target area.

[0026] Based on each SAR image in the SAR image pair, calibration parameters in a pre-constructed calibration model are solved by least squares using the difference between the geographic coordinates of a corner reflector arranged in a calibration field in the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction; and the SAR image is corrected based on the calibration parameters to obtain a SAR image that has completed geometric calibration.

[0027] Optionally, the dense matching of the SAR image pairs using object-space elevation search to obtain a SAR point cloud includes:

[0028] Initialize multiple candidate elevation values ​​according to the terrain characteristics of the target area, and establish multiple grid points representing the candidate positions in the object space of the target area;

[0029] Based on each grid point representing a candidate location, traverse the plurality of candidate elevation values ​​and project the grid point onto the SAR image pair to obtain corresponding image point coordinates; based on the image point coordinates, calculate the similarity of the grid point in the image pair at each candidate elevation value, and use the candidate elevation value corresponding to the highest similarity as the true elevation of the grid point;

[0030] The SAR point cloud is obtained based on the object space coordinates of each grid point, the image point coordinates in the SAR image pair and the true elevation.

[0031] Optionally, after unifying the coordinates and elevations of the multi-source digital surface models, performing multi-source elevation fusion to obtain a fused digital surface model includes:

[0032] Selecting control points with the same name in the multi-source digital surface model, solving coordinate transformation parameters of the control points with the same name to a target geographic coordinate system by least squares fitting, and performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model in the same target geographic coordinate system;

[0033] Converting the multi-source digital surface model under the same target geographic coordinate system from the original elevation to a preset target elevation benchmark to obtain a final multi-source digital surface model;

[0034] The weight of the reliable elevation value of each point in the visible light point cloud and the SAR point cloud determined by the image matching cost and the coherence coefficient is combined with the pre-calibrated multi-source weight ratio to perform elevation fusion on the final multi-source digital surface model to generate a fused digital surface model.

[0035] Optionally, performing point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area includes:

[0036] Converting the fused digital surface model from grid elevation to three-dimensional space coordinates pixel by pixel to generate an initial fused three-dimensional point cloud, and interpolating and supplementing the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud;

[0037] Performing radiation correction and walls filtering on the plurality of original images to obtain grayscale normalized original images, and determining a global weight of each original image based on the resolution and texture complexity of each original image;

[0038] Based on the global weight and pixel value of each original image and the correspondence between the points of the fused three-dimensional point cloud and the pixel points in each original image, respectively calculating the pixel value of each band of each point in the fused three-dimensional point cloud, each original image including the visible light image pair and the SAR image pair;

[0039] The fused three-dimensional point cloud is colored based on the pixel values ​​of each point in each band to obtain a fused three-dimensional point cloud.

[0040] Optionally, interpolating and supplementing the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud includes:

[0041] Performing point cloud density analysis and detection on the initial fused three-dimensional point cloud to obtain point cloud void areas;

[0042] Performing semantic segmentation and connected domain analysis on the visible light image pair and the SAR image pair using image segmentation technology, and obtaining candidate regions in combination with screening rules;

[0043] Associating the candidate area with the point cloud hole area through spatial indexing to obtain the point cloud area to be supplemented and the key area in the visible light image pair and the SAR image pair;

[0044] In the key area, feature matching is performed using the visible light image pair and the SAR image pair to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a newly added point cloud;

[0045] The newly added point cloud is used to interpolate the point cloud area to be supplemented to obtain a fused three-dimensional point cloud.

[0046] Based on the same inventive concept, the present invention proposes a multi-source remote sensing fusion system based on visible light images and SAR images, comprising:

[0047] The data acquisition module is used to obtain visible light image pairs and SAR image pairs of the target area at the same time and space;

[0048] a visible light point cloud extraction module, configured to perform feature matching and orientation on the visible light image pair to obtain an oriented image pair, perform dense matching on the oriented image pair using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extract a visible light point cloud from the disparity map;

[0049] A SAR point cloud extraction module is used to perform dense matching on the SAR image pairs using object space elevation search to obtain a SAR point cloud;

[0050] The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unify the coordinates and elevations of the multi-source digital surface model, and then perform multi-source elevation fusion to obtain a fused digital surface model; and perform point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0051] Optionally, the visible light point cloud extraction module is specifically used to:

[0052] Performing epipolar correction on the directional image pair to obtain an epipolar image pair, dividing the epipolar image pair into multiple resolution levels using a pyramid matching strategy, and determining a matching disparity range of an initial low-resolution level;

[0053] The matching cost combination is used to calculate the pixel similarity of candidate pixels in the matching disparity range and generate the initial cost cube;

[0054] Optimizing the initial cost cube using a semi-global matching algorithm to obtain the optimal disparity of the candidate pixel point, determining the matching disparity range of the next level based on the optimal disparity, and determining the optimal disparity of the candidate pixel point in the matching disparity range of the next level using a matching cost combination combined with the semi-global matching algorithm, and transferring the results step by step to determine the optimal disparity of the candidate pixel point in the matching disparity range of each level;

[0055] A disparity map of the epipolar image pair is obtained based on the optimal disparity of the candidate pixel points in the matching disparity range of each level.

[0056] Optionally, the visible light point cloud extraction module is specifically used to:

[0057] Obtain multiple pairs of image points with the same name in the directional image pairs corresponding to the disparity map;

[0058] Based on each pair of homonymous image points, a nonlinear error equation is established using the rational function model parameters of the two directional images and the collinearity equation corresponding to the image-space coordinates of the homonymous image points; the average value of the regularized translation parameters of the two directional images is used as the initial value of the object-space coordinates, and the nonlinear error equation is iteratively calculated in combination with a nonlinear least squares algorithm until a convergence condition is reached, thereby obtaining the object-space coordinates of the homonymous image points;

[0059] A visible light point cloud of the visible light image pair is determined based on the object space coordinates of each pair of image points with the same name.

[0060] Optionally, the visible light point cloud extraction module is specifically used to:

[0061] Using the SIFT algorithm combined with the distance metric function, based on the grayscale change characteristics and structural information of the visible light image pair, the feature points of the visible light image pair are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs;

[0062] Initialize the model parameters of the rational function model that introduces image space compensation parameters;

[0063] Establishing an error equation based on the image space compensation parameters and the model parameters for each pair of connection points, and combining the error equations of multiple pairs of connection points to obtain an overdetermined equation group;

[0064] Solving the overdetermined set of equations using a block adjustment technique to obtain image square compensation parameters and model parameters, and determining a final rational function model based on the image square compensation parameters and model parameters;

[0065] The object space coordinates of the visible light image pair are determined using the final rational function model, and a pair of oriented images with completed orientation is obtained based on the same object space coordinates.

[0066] Optionally, the SAR point cloud extraction module is further used to:

[0067] Based on each SAR image in a SAR image pair:

[0068] The atmospheric propagation delay correction model is used to generate the atmospheric propagation delay correction based on the neutral atmosphere and ionosphere parameters of the target area.

[0069] Based on each SAR image in the SAR image pair, calibration parameters in a pre-constructed calibration model are solved by least squares using the difference between the geographic coordinates of a corner reflector arranged in a calibration field in the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction; and the SAR image is corrected based on the calibration parameters to obtain a SAR image that has completed geometric calibration.

[0070] Optionally, the SAR point cloud extraction module is specifically used to:

[0071] Initialize multiple candidate elevation values ​​according to the terrain characteristics of the target area, and establish multiple grid points representing the candidate positions in the object space of the target area;

[0072] Based on each grid point representing a candidate location, traverse the plurality of candidate elevation values ​​and project the grid point onto the SAR image pair to obtain corresponding image point coordinates; based on the image point coordinates, calculate the similarity of the grid point in the image pair at each candidate elevation value, and use the candidate elevation value corresponding to the highest similarity as the true elevation of the grid point;

[0073] The SAR point cloud is obtained based on the object space coordinates of each grid point, the image point coordinates in the SAR image pair and the true elevation.

[0074] Optionally, the point cloud fusion module is specifically used to:

[0075] Selecting control points with the same name in the multi-source digital surface model, solving coordinate transformation parameters of the control points with the same name to a target geographic coordinate system by least squares fitting, and performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model in the same target geographic coordinate system;

[0076] Converting the multi-source digital surface model under the same target geographic coordinate system from the original elevation to a preset target elevation benchmark to obtain a final multi-source digital surface model;

[0077] The weight of the reliable elevation value of each point in the visible light point cloud and the SAR point cloud determined by the image matching cost and the coherence coefficient is combined with the pre-calibrated multi-source weight ratio to perform elevation fusion on the final multi-source digital surface model to generate a fused digital surface model.

[0078] Optionally, the point cloud fusion module is specifically used to:

[0079] Converting the fused digital surface model from grid elevation to three-dimensional space coordinates pixel by pixel to generate an initial fused three-dimensional point cloud, and interpolating and supplementing the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud;

[0080] Performing radiation correction and walls filtering on the plurality of original images to obtain grayscale normalized original images, and determining a global weight of each original image based on the resolution and texture complexity of each original image;

[0081] Based on the global weight and pixel value of each original image and the correspondence between the points of the fused three-dimensional point cloud and the pixel points in each original image, respectively calculating the pixel value of each band of each point in the fused three-dimensional point cloud, each original image including the visible light image pair and the SAR image pair;

[0082] The fused three-dimensional point cloud is colored based on the pixel values ​​of each point in each band to obtain a fused three-dimensional point cloud.

[0083] Optionally, the point cloud fusion module is specifically used to:

[0084] Performing point cloud density analysis and detection on the initial fused three-dimensional point cloud to obtain point cloud void areas;

[0085] Performing semantic segmentation and connected domain analysis on the visible light image pair and the SAR image pair using image segmentation technology, and obtaining candidate regions in combination with screening rules;

[0086] Associating the candidate area with the point cloud hole area through spatial indexing to obtain the point cloud area to be supplemented and the key area in the visible light image pair and the SAR image pair;

[0087] In the key area, feature matching is performed using the visible light image pair and the SAR image pair to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a newly added point cloud;

[0088] The newly added point cloud is used to interpolate the point cloud area to be supplemented to obtain a fused three-dimensional point cloud.

[0089] In another aspect, the present application further provides an electronic device, comprising: at least one processor and a memory; the memory and the processor are connected via a bus;

[0090] The memory is used to store one or more programs;

[0091] When the one or more programs are executed by the at least one processor, the multi-source remote sensing fusion method based on visible light images and SAR images as described above is implemented.

[0092] On the other hand, the present application also provides a computer-readable storage medium having an execution program stored thereon. When the execution program is executed, the multi-source remote sensing fusion method based on visible light images and SAR images as described above is implemented.

[0093] Compared with the closest prior art, the present invention has the following beneficial effects:

[0094] The present invention provides a multi-source remote sensing fusion method and system based on visible light images and SAR images, comprising: obtaining a visible light image pair and a SAR image pair of the target area at the same time and space; performing feature matching and orientation on the visible light image pair to obtain an oriented image pair, using a pyramid matching strategy combined with a semi-global algorithm to densely match the oriented image pair to obtain a disparity map, and extracting a visible light point cloud from the disparity map; using object-space elevation search to densely match the SAR image pair to obtain a SAR point cloud; generating a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unifying the coordinates and elevations of the multi-source digital surface model, and then performing multi-source elevation fusion to obtain a fused digital surface model. ; The fused surface model is converted into a point cloud to obtain a three-dimensional point cloud of the target area; the present invention utilizes a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, which is convenient for extracting three-dimensional point clouds from images, and at the same time, the efficiency of data processing can be improved by "reducing the matching disparity range" through pyramid matching and "only searching the same row of pixels during dense matching to avoid searching the entire image" in kernel correction; extracting three-dimensional point clouds from images for fusion solves the problem of fusion difficulties caused by different imaging mechanisms. Traditional digital surface models that store elevations are not conducive to importing multi-source remote sensing information. In this solution, multi-source digital surface models are converted to the same elevation reference and coordinate system for multi-source elevation fusion, thereby achieving the accuracy and generalization of multi-source data fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 A flowchart of a multi-source remote sensing fusion method based on visible light images and SAR images provided by the present invention;

[0096] Figure 2 A schematic diagram of a radar stereo image acquisition method provided by the present invention;

[0097] Figure 3 Schematic diagram of the process of extracting three-dimensional information from visible light images provided by the present invention;

[0098] Figure 4 A schematic diagram of the visible light image geometric preprocessing process provided by the present invention;

[0099] Figure 5 A schematic diagram of the process for obtaining a three-dimensional point cloud from a visible light image provided by the present invention;

[0100] Figure 6 A schematic diagram of the pyramid strategy for dense matching of visible light images provided by the present invention;

[0101] Figure 7The original image segmented by the superpixel provided by the application;

[0102] Figure 8 The surface area segmented by the superpixel provided by the application;

[0103] Figure 9 The schematic view of extracting the three-dimensional point cloud based on the principle of intersection provided by the application;

[0104] Figure 10 The schematic view of the SAR image three-dimensional point cloud acquisition process provided by the application;

[0105] Figure 11 The schematic view of the SAR system working principle provided by the application;

[0106] Figure 12 The schematic view of the SAR stereo image dense matching provided by the application;

[0107] Figure 13 The flow chart of the multi-source remote sensing three-dimensional information fusion technology provided by the application;

[0108] Figure 14 The structural schematic view of the multi-source remote sensing fusion system based on the visible light image and the SAR image provided by the application;

[0109] Figure 15 The structural schematic view of an electronic device provided by the application. DETAILED DESCRIPTION

[0110] The specific embodiments of the application will be further described in detail below with reference to the accompanying drawings.

[0111] Embodiment 1

[0112] The multi-source remote sensing fusion method based on the visible light image and the SAR image provided by the application, as shown in the figure, comprises: Figure 1

[0113] S1, acquiring a visible light image pair and a SAR image pair of a target region in the same space-time;

[0114] S2, performing feature matching and orientation on the visible light image pair to obtain an oriented image pair, performing dense matching on the oriented image pair by using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extracting a visible light point cloud from the disparity map;

[0115] S3, performing dense matching on the SAR image pair by using object elevation search to obtain a SAR point cloud;

[0116] ​S4. Generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unify the coordinates and elevations of the multi-source digital surface model, and then perform multi-source elevation fusion to obtain a fused digital surface model; perform point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0117] In step S1, a visible light image pair and a SAR image pair of the target area are obtained at the same time and space.

[0118] The visible light image pairs and SAR image pairs of the target area obtained by remote sensing equipment are both high-resolution images. The high-resolution visible light remote sensing image data and SAR image data are used for rapid pre-extraction of ground object information in the target area, providing a basis for subsequent data fusion.

[0119] The method of obtaining SAR image pairs is as follows: based on the observation requirements of the target area, the SAR stereo image imaging mechanism and the radar response time difference, the stereo image observation method is determined to obtain the SAR image pairs.

[0120] Specifically, considering that SAR stereo images adopt the opposite-side stereo observation method, it is not suitable to obtain high-quality radar stereo images in mountainous areas with large terrain undulations. The SAR imaging mechanism should be fully considered, and the acquisition methods of SAR stereo images such as the same-side stereo observation and the opposite-side stereo observation should be used. Figure 2 As shown in the figure, s1 and s2 both represent radar positions), the time difference of radar images, etc., the optimal strategy is selected based on the test requirements, and the SAR image pair with better stereo observation mode and smaller time difference is selected.

[0121] In step S2, feature matching and orientation are performed on the visible light image pair to obtain an oriented image pair, and a pyramid matching strategy combined with a semi-global algorithm is used to densely match the oriented image pair to obtain a disparity map, and a visible light point cloud is extracted from the disparity map.

[0122] Taking into account the special image distortion problem caused by the push-scan imaging of visible light images through CCD linear arrays, and considering the imaging characteristics of satellite payloads, geometric error types and error propagation laws, before pre-extracting information from the visible light images, it is also necessary to perform geometric pre-processing on the visible light images.

[0123] Based on the tie point matching technology and the regional network adjustment and geometric correction technology, each of the visible light images is geometrically preprocessed to obtain the visible light image after geometric preprocessing, and the visible light image imaging model is refined. The geometric preprocessing process includes: studying and utilizing the automatic extraction technology of high-quality tie points and control points in the regional network adjustment process to achieve high-precision automatic matching of multiple satellite images. According to the visible light image feature matching principle, the classic SIFT algorithm (Scale-Invariant Feature Transform, a scale-space-based image local feature description operator) is used to obtain robust tie points between visible light images. At the same time, the multi-perspective visible light image matching results in the region are combined (optionally, if it is necessary to fuse images of the same region at different times, the multi-phase and multi-perspective visible light image matching results in the region are combined), combined with auxiliary control data, based on the regional network adjustment and geometric correction principles, considering the characteristics of the RFM (Rational Function Model) general imaging model of satellite remote sensing images, geometric correction of visible light satellite images is performed, the visible light satellite image imaging model is refined, and the ground positioning accuracy of visible light images is improved. Figure 3 and Figure 4 As shown, specifically:

[0124] (1) Using the SIFT algorithm combined with the distance measurement function, based on the grayscale change characteristics and structural information of the visible light image pair, the feature points of the visible light image pair are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs.

[0125] (1-1) performing feature point detection based on the grayscale variation characteristics and structural information of each of the visible light images to obtain a plurality of feature points of each of the visible light images;

[0126] Optionally, the feature point detection method includes Harris corner detection technology and edge detection technology. Harris corner detection technology can be used to extract corner points in the image based on the grayscale gradient and structural information of the image, and edge detection technology can be used to obtain other key feature points. The two are combined to obtain multiple feature points in each of the visible light images.

[0127] (1-2) constructing a scale-invariant feature description vector for each feature point based on the position of each feature point using the SIFT algorithm;

[0128] Among them, if only Harris corner detection is used, it cannot resist scale changes. The SIFT algorithm can be used to construct a local feature description vector with scale / rotation invariance based on the feature point positions extracted by corner detection, that is, it can construct a robust and locally invariant matching description.

[0129] (1-3) Calculating the global similarity between the plurality of feature points based on the feature description vector of each feature point using a distance metric function, and determining a plurality of matching feature points based on the global similarity as connection points between the plurality of visible light images.

[0130] The distance metric function includes the Euclidean distance function and the Mahalanobis distance function.

[0131] Optionally, after determining multiple matching feature points based on global similarity, the above method also includes: using classic robust estimation technology to filter out matching errors caused by ground object occlusion, texture similarity, etc. on the visible light image, to obtain more robust matching feature points, and to provide more accurate connection points for subsequent visible light image regional network adjustment and geometric correction.

[0132] (2) Using the RFM block adjustment technology based on image space compensation, based on the connection points between the multiple visible light images, the system error compensation is performed on the multiple visible light images to obtain multiple directional images with completed error compensation.

[0133] Specifically, using visible light imagery, based on the theory of satellite image block adjustment in photogrammetry, and following the principle that image points with the same name (i.e., the tie points obtained above) intersect at the same object point, block adjustment is performed using the tie points of visible light images within the region. Block adjustment should consider the characteristics of the RFM universal imaging model for satellite remote sensing imagery. Systematic errors are compensated through a more rigorous RFM block adjustment technique based on image square compensation, refining the visible light satellite imagery imaging model and improving the ground positioning accuracy of visible light satellite imagery.

[0134] Specifically, the model parameters of the rational function model that introduces the image space compensation parameters are initialized; based on the connection points between multiple visible light images, the imaging geometric relationship of the multiple visible light images is constructed using the rational function model (RFM), that is, the mapping relationship between the image space coordinates and the object space coordinates of the visible light images;

[0135] Establishing an error equation based on the image space compensation parameters and the model parameters for each pair of connection points, and combining the error equations of multiple pairs of connection points to obtain an overdetermined equation group;

[0136] Solving the overdetermined set of equations using a least squares method using a block adjustment technique to obtain image square compensation parameters and model parameters, and determining a final rational function model based on the image square compensation parameters and the model parameters;

[0137] The object space coordinates of the visible light image pair are determined using the final rational function model, and a pair of oriented images with completed orientation is obtained based on the same object space coordinates.

[0138] Orientation refers to the precise calculation of the exterior orientation elements (position, attitude) of each image through adjustment techniques (such as RFM+image compensation) so that they are strictly aligned with the real geographic coordinate system.

[0139] In addition, full consideration is given to the situation where visible light satellite images are combined with auxiliary control data for regional block adjustment and geometric correction under high positioning accuracy requirements. Appropriate auxiliary control data is selected to control errors that cannot be eliminated by regional block adjustment, making the subsequent pre-extracted three-dimensional point cloud closer to the actual geographical location, and also providing basic accuracy guarantees for high-precision three-dimensional modeling, engineering facility target and structure identification, etc.

[0140] Therefore, during the block adjustment process, auxiliary control data can be selected to further compensate for errors that cannot be eliminated by the block adjustment.

[0141] By analyzing the residuals after block adjustment (such as tie point / control point residual distribution, error ellipse, spatial autocorrelation), the type of residual error and its spatial distribution pattern are identified;

[0142] Select auxiliary data according to the error type, convert the auxiliary data into constraints for block adjustment, re-perform block adjustment on the visible light image to obtain the optimized orientation image;

[0143] The auxiliary data includes attitude data and multi-source images, etc. The attitude data is used to correct system drift in time or orbit direction; the multi-source images are used to compensate for atmospheric or cross-sensor differences.

[0144] At the same time, taking into full consideration the characteristics of satellite images with large width and the inability of conventional geometric transformations to absorb image distortion, in order to make the geometric correction of visible light images more stable, this solution also includes the following before using the RFM block adjustment technology based on image square compensation to compensate for the systematic errors of visible light images:

[0145] Each visible light image is segmented based on the same specifications to obtain multiple block images of each visible image. Each block image has a number of connection points, ensuring that the visible light image can be locally geometrically corrected in the local block.

[0146] At this point, the geometric preprocessing of the visible light image is completed, and the visible light image with error elimination is obtained.

[0147] S2-1. Using a pyramid matching strategy combined with a semi-global algorithm, dense matching is performed on the directional image pairs to obtain a disparity map.

[0148] Due to the wide width characteristics of satellite images and the dense requirements of three-dimensional point clouds, the reconstruction of three-dimensional point clouds from visible light images needs to improve the efficiency of three-dimensional point cloud construction while ensuring the accuracy of digital surface models. Based on the above requirements, it is planned to construct visible light image nuclear line image pairs based on the classic optical image three-dimensional reconstruction process system, and make full use of nuclear line constraints to speed up dense matching. Based on the classic semi-global matching algorithm, with reference to the cutting-edge pyramid strategy acceleration method, the matching search range is automatically reduced layer by layer, effectively ensuring a small and correct matching search range, improving the overall matching accuracy and stability of satellite images in the target area, and reducing parameter dependence. At the same time, based on image segmentation technology, matching errors and missed points are supplemented, matching noise is suppressed, parallax breaks are retained, and adjacent matching results are constrained to remain as smooth as possible, such as Figure 5 shown.

[0149] 1a) performing epipolar correction on the oriented image pair to obtain an epipolar image pair.

[0150] To address the problems of uncertain search range and excessively large search range in pixel-by-pixel dense matching of visible light images, this scheme accelerates the dense matching process and constructs stereo pairs of visible light satellite epipolar images. By using epipolar correction, this scheme constrains the single-point matching range to its epipolar line, providing a basis for subsequent fast dense matching.

[0151] Specifically, based on the imaging geometry model of the directional image pair, a kernel curve equation is constructed;

[0152] Simplifying the linear equation using piecewise linear fitting technology to obtain an approximate kernel curve equation;

[0153] Based on the simplified kernel curve equation, image resampling is performed on the directional image pair to obtain a corrected epipolar line image pair.

[0154] This approach takes into account the irregular epipolar curves resulting from the push-broom imaging characteristics of visible satellite imagery. Approximate fitting replaces the higher-order epipolar curve equation to improve the efficiency of epipolar correction. For each pixel in a corrected epipolar image pair, it only needs to search the same row in the other image, avoiding searching the entire image.

[0155] Optionally, considering that errors may be introduced in the process of approximate fitting of the kernel curve, this scheme determines the effect of the kernel line correction by counting the distances of the real homonymous points from the kernel line and experimentally determining the distance threshold. The visible light image with poor kernel line correction effect is appropriately divided into blocks to reduce the error of the approximate fitting of the kernel curve.

[0156] 1b) using a pyramid matching strategy based on the same specifications, dividing each image in the epipolar image pair into multiple resolution levels, and based on the matching disparity range of an initial low-resolution level in the multi-resolution level, combining a matching cost combination and a semi-global matching algorithm, calculating the optimal disparity of each candidate pixel point in the matching disparity range of each level step by step to obtain a disparity map of the epipolar image pair.

[0157] In other words, for dense matching of visible light images, both matching speed and matching accuracy are required. This solution, based on the classic semi-global matching algorithm, comprehensively selects an appropriate matching cost and explores and utilizes a pyramid matching strategy for matching acceleration. This approach accelerates the matching efficiency of wide-band satellite imagery while ensuring dense matching accuracy.

[0158] Specifically, the epipolar image pair is divided into multiple resolution levels using a pyramid matching strategy, and a matching disparity range of an initial low-resolution level is determined;

[0159] Calculating the pixel similarity of candidate pixels in the matching disparity range using a matching cost combination, which includes a Census transformation cost and an NCC (Normalized Cross-Correlation) cost, to generate an initial cost cube.

[0160] Optimizing the initial cost cube using a semi-global matching algorithm to obtain the optimal disparity of the candidate pixel. Specifically, the semi-global matching algorithm is used to aggregate neighborhood costs through multi-path dynamic programming, introduce smoothness constraints, optimize the initial cost cube of the initial low-resolution layer, obtain an aggregated cost cube, and determine the optimal disparity of each candidate pixel based on the cost cube.

[0161] Determine the matching disparity range of the next level based on the optimal disparity, use the matching cost combination combined with the semi-global matching algorithm to determine the optimal disparity of the candidate pixel points in the matching disparity range of the next level, and pass it on level by level to determine the optimal disparity of the candidate pixel points in the matching disparity range of each level;

[0162] A disparity map of the epipolar image pair is obtained based on the optimal disparity of the candidate pixel points in the matching disparity range of each level.

[0163] Among them, matching cost: In order to solve the problem that conventional matching cost is too slow to be calculated, based on classic matching costs such as Census transformation cost and NCC cost, the experiment obtains a matching cost combination that guarantees both matching accuracy and computational efficiency. In order to solve the problems of large storage space and low matching efficiency caused by the wide width characteristics of satellite images, a pyramid multi-level matching strategy is used to design an appropriate algorithm for automatically reducing the matching search range layer by layer, limiting the matching range of each level within a certain limit, and slowly reducing the matching disparity range of each pixel through step-by-step transmission. This ensures that a smaller matching search range can be obtained on the image without missing the correct matching result, speeding up the matching speed of the target area and reducing the gross error of the matching, such as Figure 6 As shown in the figure. To address the problems of traditional photogrammetric matching technology, such as large matching noise, uneven matching results, and poor matching performance in textureless areas, a semi-global matching technique is used to constrain the matching results of adjacent pixels to be as smooth and parallel as possible, effectively preserving disparity breaks and the geometric structure of surface objects. Furthermore, the semi-global matching technique aggregates neighborhood information through multi-path dynamic programming and introduces smoothness constraints, improving the disparity map's ability to preserve details in continuous areas.

[0164] 1c) Optionally, the disparity map is segmented using a superpixel segmentation technique to obtain a plurality of regions, and plane fitting is performed on the disparity of the pixel points in each region to obtain an optimized disparity map.

[0165] Since the point cloud obtained by the preliminary matching method often contains matching noise and mismatching in difficult-to-match areas, in order to obtain smooth matching results that retain edges and provide more effective information for subsequent multi-source remote sensing three-dimensional information fusion, image segmentation is used after dense matching to refine the pre-extracted matching results of the established visible light image three-dimensional information, namely the disparity map.

[0166] Among them, image segmentation should fully consider the various structural units of the ground objects, perceive the proximity in space and grayscale, and combine pixels into sub-regions of appropriate size. Select a segmentation technology suitable for refining the matching results. In this solution, classic superpixel segmentation technologies such as SLIC (Simple Linear Iterative Clustering) superpixel segmentation or SEED (Superpixels Extracted via Energy-Driven Sampling) superpixel segmentation are preferred, so that the top surface of the building in the satellite image can be segmented into spatially connected and dense homogeneous areas, providing good quality segmentation results for the subsequent refinement of the matching results based on segmentation, such as Figure 7 and Figure 8As shown in the figure. Segmentation-based matching result refinement addresses the discreteness and loopholes in the initial matching results. Based on the fundamental principle that homogeneous regions often share common elevation planes, the image segmentation result map is used to perform plane fitting on the matching disparity results within each segmentation class. This reduces matching noise and fills in pixels missed due to mismatches, resulting in smoother and more accurate matching results. Furthermore, considering the "staircase" phenomenon caused by mismatched matching boundaries due to overly small segmentation, the domain system of image segmentation is studied and utilized. Appropriate sampling methods are selected to sample boundary pixels from adjacent segments and then perform plane fitting on the pixels within the segment to reduce the staircase effect.

[0167] S2-2: Extracting a visible light point cloud from the disparity map. Obtaining the same-name image points of the directional image pair corresponding to the disparity map, performing a forward intersection using the rational function model parameters of the two images and the image-space coordinates of the same-name image points to obtain a three-dimensional point cloud of the visible light image pair.

[0168] That is, the project intends to obtain the same-name image points on the stereo satellite image according to the refined matching results (disparity map), and intersect at the same point on the ground according to the principle of forward intersection through the collinear equation, thereby obtaining the three-dimensional coordinates of the point. According to the RPC (Rational Polynomial Coefficients) parameters of the left and right images of the stereo image pair, based on the image plane coordinates of the same-name image points on the stereo image pair, direct ground positioning is performed to obtain the object space coordinates (X, Y, Z) of the corresponding ground point to achieve forward intersection based on RFM, such as Figure 9 As shown, where (r l ,c l ) and (r r ,c r ) represent the image plane coordinates of the same-name image points in the left and right images, respectively.

[0169] Specifically, obtaining multiple pairs of image points with the same name of the directional image pairs corresponding to the disparity map;

[0170] Based on each pair of homonymous image points, a nonlinear error equation is established using the rational function model parameters of the two directional images and the collinearity equation corresponding to the image-space coordinates of the homonymous image points; the average value of the regularized translation parameters of the two directional images is used as the initial value of the object-space coordinates, and the nonlinear error equation is iteratively calculated in combination with a nonlinear least squares algorithm until a convergence condition is reached, thereby obtaining the object-space coordinates of the homonymous image points;

[0171] A visible light point cloud of the visible light image pair is determined based on the object space coordinates of each pair of image points with the same name.

[0172] In step S3, dense matching is performed on the SAR image pairs using object-space elevation search to obtain a SAR point cloud.

[0173] In order to solve the problem that the pre-extraction of 3D information from high-resolution visible light images cannot obtain effective images due to environmental factors such as clouds, fog, and rainfall, which leads to the failure of 3D information pre-extraction, high-resolution SAR images that can be obtained all day and all weather are used to effectively solve the data source problem. How to achieve high-precision positioning of SAR images is the prerequisite for 3D information pre-extraction. Therefore, this project focuses on breakthroughs in high-precision positioning of SAR images, improvement of SAR image radiation quality and SAR dense matching, so as to achieve SAR image 3D reconstruction point cloud based on high-resolution SAR image 3D information pre-extraction, and provide data support for subsequent multi-source 3D information fusion, such as Figure 10 shown.

[0174] Different from optics, SAR images have two problems: the inherent speckle noise reduces the quality of SAR images and the side-view imaging characteristics lead to inconsistent ranges of objects in the same window. These two problems bring challenges to dense matching and subsequent point cloud generation. Therefore, before pre-extracting 3D information from SAR images, they need to be geometrically pre-processed. The geometric pre-processing of SAR images mainly includes two aspects: improving the radiation quality of SAR images through SAR speckle suppression technology, and improving the relative and absolute positioning accuracy of spaceborne SAR images through geometric calibration, regional block adjustment and other technologies. Figure 10 shown.

[0175] Before dense matching, the SAR image is geometrically preprocessed: (1) Radiometric quality improvement—SAR image speckle suppression.

[0176] SAR images are plagued by a large amount of speckle noise due to inherent defects in coherent imaging systems. Speckle noise, a typical form of random multiplicative noise, severely limits the development and application of SAR image interpretation technology. This solution, based on the principles of SAR image speckle noise suppression and incorporating edge detection algorithms, accurately identifies homogeneous and heterogeneous regions during speckle suppression, improving the effectiveness of traditional speckle suppression algorithms. To address the difficulties in accurately estimating the noise mean and variance in traditional anisotropic diffusion methods, the following approach is used to achieve optimal speckle suppression, improving the radiometric quality of SAR images.

[0177] Specifically, based on each SAR image, the Euclidean distance between pixels in the SAR image is calculated, and the homogeneous area and the heterogeneous area of ​​the SAR image are determined based on the Euclidean distance between the pixels;

[0178] Adaptively constructing the respective anisotropic diffusion coefficients of the homogeneous region and the heterogeneous region, smoothing the noise in the homogeneous region and the heterogeneous region according to the anisotropic diffusion coefficients, and obtaining a SAR image with speckle noise suppression.

[0179] (2) Improvement of absolute positioning accuracy—SAR geometric calibration.

[0180] Based on the working principle of the SAR system, the influence of the SAR system delay inside the radar system and the influence of the atmospheric propagation delay on the radar propagation path are considered to determine the error forms such as the SAR system delay and the atmospheric propagation delay, such as Figure 11 As shown, Figure 11 Trans in it represents the transmitter, which is responsible for generating and transmitting electromagnetic waves, fc represents the center frequency of the transmitted signal; Rcvr represents the receiver, which is responsible for receiving the reflected electromagnetic waves; Switch represents the switch, which is used to switch between the transmitting and receiving modes; Antenna represents the antenna, which is used to transmit and receive electromagnetic waves; Energy pulse represents the energy of the emitted electromagnetic waves; Target represents the object to be detected; Indicator represents the indicator, which is used to display the received signal information; and R represents the distance from the antenna to the target. The atmospheric propagation delay correction model is used to eliminate the error according to the neutral atmosphere and ionosphere environmental parameters of the target point. In view of the geometric positioning error of the SAR image in the range and azimuth directions, a suitable calibration model is selected, and the range and azimuth directions are used as calibration parameters. The corresponding geographical coordinates of the corner reflector and the atmospheric ionosphere content at the corresponding satellite transit time are obtained through the corner reflectors deployed in the calibration field, and the calibration parameters are solved. Specifically:

[0181] The atmospheric propagation delay correction model is used to generate the atmospheric propagation delay correction based on the neutral atmosphere and ionosphere parameters of the target area.

[0182] Based on each SAR image in the SAR image pair, calibration parameters in a pre-constructed calibration model are solved by least squares using the difference between the geographic coordinates of a corner reflector arranged in a calibration field in the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction; and the SAR image is corrected based on the calibration parameters to obtain a SAR image that has completed geometric calibration.

[0183] The geometric calibration model should have a certain degree of generalization. For SAR images in non-calibrated areas, the calibration parameters can also be used for compensation, thereby achieving high-precision positioning of SAR images.

[0184] The introduction of atmospheric propagation delay correction and corner reflector-assisted least squares calibration effectively improves the geometric accuracy and positioning reliability of SAR images and enhances imaging stability under complex atmospheric conditions.

[0185] (3) Improvement of relative positioning accuracy—SAR regional network adjustment.

[0186] Due to the influence of multiplicative noise in SAR images, the traditional SIFT algorithm performs poorly for SAR image registration, resulting in very large gradients in static areas with high reflectivity, leading to gradient deviations. To address these issues with SIFT, this solution leverages the noise characteristics of SAR images and selects feature point detection and feature point description algorithms suitable for multiplicative noise in SAR images. This ensures that the gradient amplitude and direction are highly robust to speckle noise, thereby determining the connection points between SAR images.

[0187] Because SAR imaging is completely different from visible light imaging, its imaging model is also different from that of visible light imaging. SAR image block adjustment should be performed based on the Range-Doppler (RD) model, with the image range-slant distance, azimuth flight time, and the geographic coordinates corresponding to the tie points as the adjustment parameters to be solved, and the image point coordinates of the tie points as the known observations. The project intends to use the condition that image points with the same name at tie points should correspond to the same ground coordinates to connect the regional images as a whole, and construct an error equation based on the RD model, with the image range-slant distance, azimuth flight time, and the geographic coordinates (longitude, latitude, and elevation) corresponding to the tie points as unknown parameters. Two error equations can be listed for each tie point coordinate. All error equations are listed and expressed in matrix form, and then solved using an optimization algorithm to obtain the azimuth flight time, range-slant distance correction for each image, and the geographic coordinates corresponding to each tie point.

[0188] Through regional block adjustment based on the range-Doppler model, the geographic coordinates of each connection point are jointly solved and optimized to form consistent three-dimensional ground coordinates. The SAR image pairs are spatially aligned through common connection points, eliminating the systematic geometric deviation between images.

[0189] At this point, the SAR image pair that has completed geometric preprocessing is obtained.

[0190] The SAR image pairs are densely matched using object space elevation search to obtain a SAR point cloud.

[0191] That is, in this scheme, in view of the characteristics of the SAR image side-view imaging mode and the weak intersection angle characteristics of the stereo image pair, as well as the need for three-dimensional information extraction from SAR image data at different phases and angles, the SAR stereo image dense matching adopts a matching strategy based on direct elevation search in the object space to accelerate the matching efficiency of SAR image data at different phases and angles, such as Figure 12 shown.

[0192] Specifically, a plurality of candidate elevation values ​​are initialized according to the terrain characteristics of the target area, and a plurality of grid points representing the candidate positions are established in the object space of the target area;

[0193] Based on each grid point representing a candidate location, traverse the plurality of candidate elevation values ​​and project the grid point onto the SAR image pair to obtain corresponding image point coordinates; based on the image point coordinates, calculate the similarity of the grid point in the image pair at each candidate elevation value, and use the candidate elevation value corresponding to the highest similarity as the true elevation of the grid point;

[0194] The SAR point cloud is obtained based on the object space coordinates of each grid point, the image point coordinates in the SAR image pair and the true elevation.

[0195] Optionally, in the matching process described above, to address the issue of SAR images having large side-view tilt angles and being sensitive to geometric deformation between images, and to avoid grayscale inconsistencies in matching SAR images with large tilt angles, an adaptively corrected matching window (grid point) is employed. Specifically, based on the basic principle of correlation matching, an adaptive window is used to match the grayscale information within the matching windows of the two images, and the optimal adaptive adjustment strategy is selected to ensure consistent grayscale information within the matching windows of the same size in the two images.

[0196] In step S4, a multi-source digital surface model is generated by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. Point cloud conversion is performed on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0197] The digital surface model is a "three-dimensional point cloud" stored in a regular manner. For the ground surface, the difference between the digital surface model and the three-dimensional point cloud is very small. In engineering practice, the two are often converted to each other. The fusion of multiple point clouds with the help of the regular grid of the digital surface model has been verified by some studies. Considering that it is difficult to directly perform multi-source fusion in the form of a three-dimensional point cloud, and a large number of open source elevation products are digital ground models / digital elevation models that store elevations in a regular manner, which is not conducive to importing information obtained by other means, it is considered to construct a multi-source digital surface model based on the multi-source point cloud, and combine the information obtained by other means to perform elevation fusion in a regular grid unit. Finally, the digital surface model is used as an intermediate fusion medium, and the digital surface model and the original image are jointly fused to construct a color three-dimensional point cloud that fuses multi-source remote sensing three-dimensional information to serve the subsequent engineering facility target and structure identification. It is specifically achieved through two stages of technical content: the construction and fusion of multi-source digital surface models, and the construction of fused three-dimensional point clouds based on fused digital surface models, such as Figure 13 shown.

[0198] S4-1. Generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unify the coordinates and elevations of the multi-source digital surface model, and then perform multi-source elevation fusion to obtain a fused digital surface model.

[0199] In summary, for three-dimensional point clouds from different phases, different spatial positions, and different imaging sources (visible light / SAR), according to the point cloud fusion technology route based on multi-source fusion digital surface model, the advantages of multiple three-dimensional point clouds are integrated, and the three-dimensional point clouds from each source are fully utilized. Digital surface models are constructed one by one for the three-dimensional point clouds of each phase, spatial position, and imaging source (visible light / SAR). Then, the multiple digital surface models constructed are aligned and fused to obtain an accurate fused digital surface model, which provides support for the subsequent construction of fused three-dimensional point clouds. The steps include: multi-source digital surface model construction, multi-source digital surface model alignment, and multi-source digital surface model fusion.

[0200] 1a) Using the digital surface model to construct an interpolation method, the models generated after the visible light point cloud and the SAR point cloud are rasterized are interpolated to obtain the visible light digital surface model and the SAR digital surface model.

[0201] Specifically, the visible light point cloud is rasterized to generate an initial digital surface model, and the vacant areas in the initial digital surface model are marked;

[0202] Select multiple non-vacant areas, and use nearest neighbor interpolation, bilinear interpolation, and inverse distance interpolation in combination with the visible light point cloud to interpolate and fill each non-vacant area to obtain an interpolation area;

[0203] Calculating the elevation deviation between the interpolation area and the visible light point cloud under each interpolation method, and selecting the interpolation method corresponding to the minimum elevation deviation;

[0204] The interpolation method corresponding to the minimum elevation deviation is used in combination with the visible light point cloud to interpolate the missing area to obtain a visible light digital surface model.

[0205] Similarly, a SAR digital surface model corresponding to the SAR point cloud is obtained, and the visible light digital surface model and the SAR digital surface model together constitute a multi-source digital surface model.

[0206] Optionally, before constructing the digital surface model, refer to the geographical location distribution of each point cloud and select an appropriate geographical coordinate range as the range of the digital surface model so that the three-dimensional information of the point cloud is not missed; comprehensively consider the resolution of the three-dimensional point cloud source image, the size of the target to be observed later, etc., and unify the resolution of the digital surface model in the digital surface model construction stage to ensure that the subsequent digital surface model fusion can be carried out more effectively, providing a basis for three-dimensional information fusion.

[0207] For each point cloud irregularly distributed on the surface in the form of three-dimensional coordinates of longitude, latitude and elevation, full consideration is given to multiple requirements such as the scope of the digital surface model, the resolution of the digital surface model, and the construction method of the digital surface model, and the three-dimensional information of the point cloud is regularized into a grid form to provide a suitable fusion basis for three-dimensional information fusion.

[0208] 1b) selecting control points with the same name in the multi-source digital surface model, solving coordinate transformation parameters of the control points with the same name to a target geographic coordinate system by least squares fitting, and performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model in the same target geographic coordinate system;

[0209] The multi-source digital surface model under the same target geographic coordinate system is converted from the original elevation to a preset target elevation reference to obtain a final multi-source digital surface model.

[0210] In this solution, to eliminate geographic location deviations among multiple digital surface models due to errors in satellite image positioning models and point cloud reconstruction, and to achieve positioning accuracy as close to true geographic coordinates as possible, this solution fully considers the elevation similarities between multiple digital surface models, references other intelligence such as control data, and utilizes image matching and coordinate correction techniques to unify all digital surface models into the same geographic coordinate system. This ensures that the elevations of multi-source point clouds are corrected to the same grid pixel location / geographic location as much as possible, ensuring the fusion of elevation information. By unifying the coordinate framework and elevation benchmark, systematic deviations from multi-source data are eliminated, and adaptive elevation fusion is achieved based on reliability weights, improving the geometric accuracy of the resulting fused digital surface model while effectively preserving the advantages of each data source.

[0211] In order to solve the problem that the three-dimensional point clouds of multi-source satellite images have different elevation datums, we fully consider the starting position of the elevation values ​​of the multi-source three-dimensional point clouds, refer to engineering practice experience, select a widely accepted and easy-to-use elevation datum as the elevation datum of the digital surface model, and convert the elevations of all digital surface models to the same elevation datum to avoid fusion errors caused by inconsistent elevation datums.

[0212] 1c) Using the reliable elevation value weights of each point in the visible light point cloud and the SAR point cloud determined by the image matching cost and the coherence coefficient, combined with the pre-calibrated multi-source weight ratio, the final multi-source digital surface model is subjected to elevation fusion to generate a fused digital surface model. Specifically:

[0213] Using image matching cost and coherence coefficient, outliers in the visible light 3D point cloud and the SAR 3D point cloud are detected respectively, and a reliable elevation value weight of each point other than the outlier is determined, where the elevation value of the outlier point is an unreliable elevation value;

[0214] Based on the reliable elevation value weight of each point outside the outlier, the weight ratio of the visible light three-dimensional point cloud to the SAR three-dimensional point cloud calibrated in advance, and the elevation values ​​in the visible light digital surface model and the SAR digital surface model, the visible light digital surface model and the SAR digital surface model are fused in elevation to obtain a fused digital surface model.

[0215] Before the fusion of multi-source digital surface models, the multi-source digital surface models are pre-processed. Since the registered multi-source digital surface model corresponds to one or more elevation values ​​provided by different source point clouds at each grid position (i.e., geographic location), the multi-source three-dimensional information fusion is converted into pixel-by-pixel elevation value fusion under a regular grid. For grid positions with only a single elevation value or an invalid elevation value, no processing is performed to minimize the destruction of the geometric structure of the three-dimensional information; for grid positions with multiple redundant elevation values, the principle of point cloud construction is comprehensively considered, and the probability of each source elevation being close to the true elevation is estimated through point matching reliability, elevation statistical properties, etc., and multiple elevation values ​​are fused to obtain the final unique elevation value, completing the pre-processing before the fusion of the multi-source digital surface model.

[0216] S4-2. Perform point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0217] Based on the fused digital surface model, the 3D point cloud is constructed, taking into account the specific characteristics of the ground objects to ensure the integrity of the 3D point cloud's geometric structure. Subsequently, considering the need for grayscale / color information in the point cloud, and taking into account the differences in grayscale / color information caused by differences in the radiation environment and imaging angles of multi-temporal, multi-source, and multi-view satellite imagery, the 3D point cloud is endowed with image texture information by leveraging the relationship between the fused reference geographic / elevation coordinate system and the image's original geographic / elevation coordinate system, ensuring rich texture information in the 3D point cloud.

[0218] (1) Construction of three-dimensional point cloud. The fused digital surface model is converted into a three-dimensional point cloud expressed in three-dimensional spatial coordinates, and the raster elevation is converted to x, y, and z spatial coordinates pixel by pixel. Since the fused digital surface model is fused from multiple source digital surface models that have been geo-referenced, the converted three-dimensional point cloud does not require other registration or fusion operations. However, considering that a small number of side point clouds may be missed due to the regularization of the original point cloud during the construction of the digital surface model, it is necessary to combine appropriate strategies such as image segmentation or expert knowledge assistance to supplement the missing part of the point cloud in the conversion of the fused digital surface model into the three-dimensional point cloud to obtain a more complete three-dimensional spatial point cloud.

[0219] Specifically, the fused digital surface model is converted pixel by pixel from raster elevation to three-dimensional space coordinates to generate an initial fused three-dimensional point cloud;

[0220] Interpolating and supplementing the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud, including:

[0221] Performing point cloud density analysis and detection on the initial fused three-dimensional point cloud to obtain point cloud void areas;

[0222] Performing semantic segmentation and connected domain analysis on the visible light image pair and the SAR image pair using image segmentation technology to obtain multiple connected domains, and screening the connected domains based on preset screening rules to obtain candidate regions;

[0223] Associating the candidate regions of the visible light image pair and the SAR image pair with the point cloud hole regions of the initial fused three-dimensional point cloud through spatial indexing, screening the point cloud hole regions, and obtaining the point cloud regions to be supplemented and the key regions in the visible light image pair and the SAR image pair;

[0224] In the key area, feature matching is performed using the visible light image pair and the SAR image pair to generate sparse matching points, and the sparse matching points are densely reconstructed to obtain a new point cloud;

[0225] The newly added point cloud is used to interpolate the point cloud area to be supplemented to obtain a fused three-dimensional point cloud.

[0226] The detection of missing areas through point cloud density analysis mainly locates the range of missing geometric data, but it cannot directly determine whether these areas belong to targets such as building facades that need to be completed (which may be the ground, vegetation, etc.); image segmentation assistance uses semantic information to extract key areas (such as building edges, etc.) through semantic segmentation, and further screens valid candidate areas in the missing areas. That is, if there are point cloud holes in the key areas, they are determined to be valid candidate areas, avoiding unnecessary completion of non-candidate areas, and improving the accuracy and pertinence of the completion.

[0227] (2) Considering the demand for grayscale / color information of point clouds in actual usage scenarios of fused point clouds, the texture information of visible light / SAR images is comprehensively considered to obtain the projection relationship between point clouds and images, and grayscale / color information is provided for each spatial point cloud in the fused three-dimensional point cloud to ensure that more significant features can be extracted based on the three-dimensional point cloud.

[0228] The project should integrate the geometric registration transformation relationship during the digital surface model fusion process, comprehensively consider the relationship between the fused reference geographic / elevation coordinate system and the original geographic / elevation coordinate system of the image, and work backwards from the 3D point cloud to the image pixel position of the image to be extracted. Considering that each 3D point cloud position may correspond to multi-temporal, multi-source, and multi-view satellite imagery, and that different satellite images may have different grayscale / color information due to differences in imaging radiation environment and imaging angle, common grayscale unification algorithms such as image radiometric correction and filtering are used to fuse the multi-view imagery to obtain unique color values ​​for each band.

[0229] Specifically, the fused 3D point cloud is reversed based on the coordinate system conversion process in the digital surface model construction process to obtain the correspondence between the points in the fused 3D point cloud and the corresponding pixel points in the plurality of original images;

[0230] Performing radiation correction and walls filtering on the plurality of original images to obtain grayscale normalized original images, and determining a global weight of each original image based on the resolution and texture complexity of each original image;

[0231] Based on the global weight and pixel value of each original image and the correspondence between the points of the fused three-dimensional point cloud and the pixel points in each original image, respectively calculating the pixel value of each band of each point in the fused three-dimensional point cloud, each original image including the visible light image pair and the SAR image pair;

[0232] The fused three-dimensional point cloud is colored based on the pixel values ​​of each point in each band to obtain a fused three-dimensional point cloud.

[0233] At the same time, considering that the texture information of visible light / SAR images may have their own uses, the project retains the information of visible light / SAR images when extracting the grayscale / color information of visible light / SAR images to avoid information loss due to fusion.

[0234] Example 2

[0235] Based on the same inventive concept, the present invention also provides a multi-source remote sensing fusion system based on visible light images and SAR images, such as Figure 14 Shown, including:

[0236] The data acquisition module is used to obtain visible light image pairs and SAR image pairs of the target area at the same time and space;

[0237] a visible light point cloud extraction module, configured to perform feature matching and orientation on the visible light image pair to obtain an oriented image pair, perform dense matching on the oriented image pair using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extract a visible light point cloud from the disparity map;

[0238] A SAR point cloud extraction module is used to perform dense matching on the SAR image pairs using object space elevation search to obtain a SAR point cloud;

[0239] The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unify the coordinates and elevations of the multi-source digital surface model, and then perform multi-source elevation fusion to obtain a fused digital surface model; and perform point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area.

[0240] In one possible implementation, the visible light point cloud extraction module is specifically configured to:

[0241] Performing epipolar correction on the directional image pair to obtain an epipolar image pair, dividing the epipolar image pair into multiple resolution levels using a pyramid matching strategy, and determining a matching disparity range of an initial low-resolution level;

[0242] The matching cost combination is used to calculate the pixel similarity of candidate pixels in the matching disparity range and generate the initial cost cube;

[0243] Optimizing the initial cost cube using a semi-global matching algorithm to obtain the optimal disparity of the candidate pixel point, determining the matching disparity range of the next level based on the optimal disparity, and determining the optimal disparity of the candidate pixel point in the matching disparity range of the next level using a matching cost combination combined with the semi-global matching algorithm, and transferring the results step by step to determine the optimal disparity of the candidate pixel point in the matching disparity range of each level;

[0244] A disparity map of the epipolar image pair is obtained based on the optimal disparity of the candidate pixel points in the matching disparity range of each level.

[0245] In one possible implementation, the visible light point cloud extraction module is specifically configured to:

[0246] Obtain multiple pairs of image points with the same name in the directional image pairs corresponding to the disparity map;

[0247] Based on each pair of homonymous image points, a nonlinear error equation is established using the rational function model parameters of the two directional images and the collinearity equation corresponding to the image-space coordinates of the homonymous image points; the average value of the regularized translation parameters of the two directional images is used as the initial value of the object-space coordinates, and the nonlinear error equation is iteratively calculated in combination with a nonlinear least squares algorithm until a convergence condition is reached, thereby obtaining the object-space coordinates of the homonymous image points;

[0248] A visible light point cloud of the visible light image pair is determined based on the object space coordinates of each pair of image points with the same name.

[0249] In one possible implementation, the visible light point cloud extraction module is specifically configured to:

[0250] Using the SIFT algorithm combined with the distance metric function, based on the grayscale change characteristics and structural information of the visible light image pair, the feature points of the visible light image pair are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs;

[0251] Initialize the model parameters of the rational function model that introduces image space compensation parameters;

[0252] Establishing an error equation based on the image space compensation parameters and the model parameters for each pair of connection points, and combining the error equations of multiple pairs of connection points to obtain an overdetermined equation group;

[0253] Solving the overdetermined set of equations using a block adjustment technique to obtain image square compensation parameters and model parameters, and determining a final rational function model based on the image square compensation parameters and model parameters;

[0254] The object space coordinates of the visible light image pair are determined using the final rational function model, and a pair of oriented images with completed orientation is obtained based on the same object space coordinates.

[0255] In one possible implementation, the SAR point cloud extraction module is further configured to:

[0256] Based on each SAR image in a SAR image pair:

[0257] The atmospheric propagation delay correction model is used to generate the atmospheric propagation delay correction based on the neutral atmosphere and ionosphere parameters of the target area.

[0258] Based on each SAR image in the SAR image pair, calibration parameters in a pre-constructed calibration model are solved by least squares using the difference between the geographic coordinates of a corner reflector arranged in a calibration field in the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction; and the SAR image is corrected based on the calibration parameters to obtain a SAR image that has completed geometric calibration.

[0259] In a possible implementation, the SAR point cloud extraction module is specifically configured to:

[0260] a plurality of elevation candidate values are initialized according to the terrain features of the target region, and a plurality of grid points representing candidate positions are established in the object space of the target region;

[0261] based on each grid point representing a candidate position, a plurality of elevation candidate values are traversed, the grid point is projected into the SAR image pair to obtain corresponding image point coordinates, and based on the image point coordinates, the similarity of the grid point in the image pair under each elevation candidate value is calculated, and the elevation candidate value corresponding to the highest similarity is taken as the true elevation of the grid point;

[0262] based on the image point coordinates and the true elevation of each grid point in the object space in the SAR image pair, the SAR point cloud is obtained.

[0263] In a possible implementation, the point cloud fusion module is specifically configured to:

[0264] the same name control points are selected in the multi-source digital surface model, coordinate conversion parameters of the same name control points to a target geographic coordinate system are solved by least square fitting, and the multi-source digital surface model under the same target geographic coordinate system is obtained based on the coordinate conversion parameters;

[0265] the multi-source digital surface model under the same target geographic coordinate system is converted from an original elevation to a preset target elevation datum, and a final multi-source digital surface model is obtained;

[0266] the reliable elevation value weight of each point in the visible light point cloud and the SAR point cloud determined by the image matching cost and the coherence coefficient is combined with a pre-calibrated multi-source weight ratio to perform elevation fusion on the final multi-source digital surface model, and a fused digital surface model is generated.

[0267] In a possible implementation, the point cloud fusion module is specifically configured to:

[0268] the fused digital surface model is converted from a raster elevation to a three-dimensional space coordinate pixel by pixel to generate an initial fused three-dimensional point cloud, and the initial fused three-dimensional point cloud is interpolated to obtain a fused three-dimensional point cloud;

[0269] the original images are radiometrically corrected and wall filtered to obtain gray scale normalized original images, and a global weight of each original image is determined based on the resolution and texture complexity of each original image;

[0270] Based on the global weight and pixel value of each original image and the correspondence between the points of the fused three-dimensional point cloud and the pixel points in each original image, respectively calculating the pixel value of each band of each point in the fused three-dimensional point cloud, each original image including the visible light image pair and the SAR image pair;

[0271] The fused three-dimensional point cloud is colored based on the pixel values ​​of each point in each band to obtain a fused three-dimensional point cloud.

[0272] In one possible implementation, the point cloud fusion module is specifically used to:

[0273] Performing point cloud density analysis and detection on the initial fused three-dimensional point cloud to obtain point cloud void areas;

[0274] Performing semantic segmentation and connected domain analysis on the visible light image pair and the SAR image pair using image segmentation technology, and obtaining candidate regions in combination with screening rules;

[0275] Associating the candidate area with the point cloud hole area through spatial indexing to obtain the point cloud area to be supplemented and the key area in the visible light image pair and the SAR image pair;

[0276] In the key area, feature matching is performed using the visible light image pair and the SAR image pair to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a newly added point cloud;

[0277] The newly added point cloud is used to interpolate the point cloud area to be supplemented to obtain a fused three-dimensional point cloud.

[0278] Example 3

[0279] like Figure 15 As shown, the present invention also provides an electronic device, which may be a computer, a single-chip microcomputer, a smart mobile device, or the like. The electronic device in this embodiment may include a processor, a memory, a transceiver component, and the like. The memory, processor, and transceiver component are connected via a bus; the memory may be used to store an execution program, which may include instructions; and the processor may be used to execute the instructions stored in the memory. The memory may also be used to store data, which may be accessed and / or modified during the execution of the instructions.

[0280] The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions in a storage medium to implement the corresponding method flow or corresponding function, so as to implement the steps of a multi-source remote sensing fusion method based on visible light images and SAR images in the above-mentioned embodiment.

[0281] Example 4

[0282] Based on the same inventive concept, the present invention also provides a readable storage medium, specifically an electronic device readable storage medium (Memory), which is a memory device in the electronic device for storing programs and data. It can be understood that the storage medium here can include both the built-in storage medium in the electronic device and the extended storage medium supported by the electronic device. The storage medium provides a storage space, which stores the operating system of the terminal. In addition, one or more instructions suitable for being loaded and executed by the processor are also stored in the storage space. These instructions can be one or more execution programs (including program codes). It should be noted that the storage medium here can be a high-speed RAM memory or a non-volatile memory, such as at least one disk storage. The processor loads and executes one or more instructions stored in the storage medium, which can implement the steps of a multi-source remote sensing fusion method based on visible light images and SAR images in the above embodiment.

[0283] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0284] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0285] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device that implements the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0286] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks. Figure 1 one or more functions specified in the flowchart or multiple flows and / or blocks.

[0287] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application and not to limit the scope of protection, although the above embodiments of the present application are described in detail, those skilled in the art should understand that: the skilled person reading the present application can make various changes, modifications or equivalent replacements to the specific embodiments of the application, but these changes, modifications or equivalent replacements are all within the scope of protection of the claims.

Claims

1. A multi-source remote sensing fusion method based on visible light images and SAR images, characterized in that: include: Obtain visible light image pairs and SAR image pairs of the target area at the same time and space; performing feature matching and orientation on the visible light image pair to obtain an oriented image pair, performing dense matching on the oriented image pair using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extracting a visible light point cloud from the disparity map; densely matching the SAR image pairs using object space elevation search to obtain a SAR point cloud; A multi-source digital surface model is generated by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud. After unifying the coordinates and elevations of the multi-source digital surface model, multi-source elevation fusion is performed to obtain a fused digital surface model. Point cloud conversion is performed on the fused surface model to obtain a three-dimensional point cloud of the target area.

2. The method according to claim 1, wherein The method of using a pyramid matching strategy in combination with a semi-global algorithm to densely match the directional image pairs to obtain a disparity map includes: Performing epipolar correction on the directional image pair to obtain an epipolar image pair, dividing the epipolar image pair into multiple resolution levels using a pyramid matching strategy, and determining a matching disparity range of an initial low-resolution level; The matching cost combination is used to calculate the pixel similarity of candidate pixels in the matching disparity range and generate the initial cost cube; Optimizing the initial cost cube using a semi-global matching algorithm to obtain the optimal disparity of the candidate pixel point, determining the matching disparity range of the next level based on the optimal disparity, and determining the optimal disparity of the candidate pixel point in the matching disparity range of the next level using a matching cost combination combined with the semi-global matching algorithm, and transferring the results step by step to determine the optimal disparity of the candidate pixel point in the matching disparity range of each level; A disparity map of the epipolar image pair is obtained based on the optimal disparity of the candidate pixel points in the matching disparity range of each level.

3. The method according to claim 1 or 2, wherein: The extracting of a visible light point cloud from the disparity map includes: Obtain multiple pairs of image points with the same name in the directional image pairs corresponding to the disparity map; Based on each pair of homonymous image points, a nonlinear error equation is established using the rational function model parameters of the two directional images and the collinearity equation corresponding to the image-space coordinates of the homonymous image points; the average value of the regularized translation parameters of the two directional images is used as the initial value of the object-space coordinates, and the nonlinear error equation is iteratively calculated in combination with a nonlinear least squares algorithm until a convergence condition is reached, thereby obtaining the object-space coordinates of the homonymous image points; A visible light point cloud of the visible light image pair is determined based on the object space coordinates of each pair of image points with the same name.

4. The method according to claim 1, wherein The performing feature matching and orientation on the visible light image pair to obtain an oriented image pair includes: Using the SIFT algorithm combined with the distance metric function, based on the grayscale change characteristics and structural information of the visible light image pair, the feature points of the visible light image pair are extracted and matched to obtain multiple pairs of connection points between the visible light image pairs; Initialize the model parameters of the rational function model that introduces image space compensation parameters; Establishing an error equation based on the image space compensation parameters and the model parameters for each pair of connection points, and combining the error equations of multiple pairs of connection points to obtain an overdetermined equation group; Solving the overdetermined set of equations using a block adjustment technique to obtain image square compensation parameters and model parameters, and determining a final rational function model based on the image square compensation parameters and model parameters; The object space coordinates of the visible light image pair are determined using the final rational function model, and a pair of oriented images with completed orientation is obtained based on the same object space coordinates.

5. The method according to claim 1, wherein Before densely matching the SAR image pairs using object-space elevation search, the method further includes: The atmospheric propagation delay correction model is used to generate the atmospheric propagation delay correction based on the neutral atmosphere and ionosphere parameters of the target area. Based on each SAR image in the SAR image pair, calibration parameters in a pre-constructed calibration model are solved by least squares using the difference between the geographic coordinates of a corner reflector arranged in a calibration field in the target area and the corresponding image coordinates in the SAR image, as well as the atmospheric propagation delay correction; and the SAR image is corrected based on the calibration parameters to obtain a SAR image that has completed geometric calibration.

6. The method according to claim 1 or 5, wherein: The dense matching of the SAR image pairs using object-space elevation search to obtain a SAR point cloud includes: Initialize multiple candidate elevation values ​​according to the terrain characteristics of the target area, and establish multiple grid points representing the candidate positions in the object space of the target area; Based on each grid point representing a candidate location, traverse the plurality of candidate elevation values ​​and project the grid point onto the SAR image pair to obtain corresponding image point coordinates; based on the image point coordinates, calculate the similarity of the grid point in the image pair at each candidate elevation value, and use the candidate elevation value corresponding to the highest similarity as the true elevation of the grid point; The SAR point cloud is obtained based on the object space coordinates of each grid point, the image point coordinates in the SAR image pair and the true elevation.

7. The method according to claim 1, wherein After unifying the coordinates and elevations of the multi-source digital surface models, performing multi-source elevation fusion to obtain a fused digital surface model includes: Selecting control points with the same name in the multi-source digital surface model, solving coordinate transformation parameters of the control points with the same name to a target geographic coordinate system by least squares fitting, and performing coordinate transformation based on the coordinate transformation parameters to obtain a multi-source digital surface model in the same target geographic coordinate system; Converting the multi-source digital surface model under the same target geographic coordinate system from the original elevation to a preset target elevation benchmark to obtain a final multi-source digital surface model; The weight of the reliable elevation value of each point in the visible light point cloud and the SAR point cloud determined by the image matching cost and the coherence coefficient is combined with the pre-calibrated multi-source weight ratio to perform elevation fusion on the final multi-source digital surface model to generate a fused digital surface model.

8. The method according to claim 1 or 7, wherein: The step of performing point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area includes: Converting the fused digital surface model from grid elevation to three-dimensional space coordinates pixel by pixel to generate an initial fused three-dimensional point cloud, and interpolating and supplementing the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud; Performing radiation correction and walls filtering on the plurality of original images to obtain grayscale normalized original images, and determining a global weight of each original image based on the resolution and texture complexity of each original image; Based on the global weight and pixel value of each original image and the correspondence between the points of the fused three-dimensional point cloud and the pixel points in each original image, respectively calculating the pixel value of each band of each point in the fused three-dimensional point cloud, each original image including the visible light image pair and the SAR image pair; The fused three-dimensional point cloud is colored based on the pixel values ​​of each point in each band to obtain a fused three-dimensional point cloud.

9. The method according to claim 8, wherein The interpolation and supplementation of the initial fused three-dimensional point cloud to obtain a fused three-dimensional point cloud includes: Performing point cloud density analysis and detection on the initial fused three-dimensional point cloud to obtain point cloud void areas; Performing semantic segmentation and connected domain analysis on the visible light image pair and the SAR image pair using image segmentation technology, and obtaining candidate regions in combination with screening rules; Associating the candidate area with the point cloud hole area through spatial indexing to obtain the point cloud area to be supplemented and the key area in the visible light image pair and the SAR image pair; In the key area, feature matching is performed using the visible light image pair and the SAR image pair to generate sparse matching points, and dense reconstruction is performed on the sparse matching points to obtain a newly added point cloud; The newly added point cloud is used to interpolate the point cloud area to be supplemented to obtain a fused three-dimensional point cloud.

10. A multi-source remote sensing fusion system based on visible light images and SAR images, characterized by: include: The data acquisition module is used to obtain visible light image pairs and SAR image pairs of the target area at the same time and space; a visible light point cloud extraction module, configured to perform feature matching and orientation on the visible light image pair to obtain an oriented image pair, perform dense matching on the oriented image pair using a pyramid matching strategy combined with a semi-global algorithm to obtain a disparity map, and extract a visible light point cloud from the disparity map; A SAR point cloud extraction module is used to perform dense matching on the SAR image pairs using object space elevation search to obtain a SAR point cloud; The point cloud fusion module is used to generate a multi-source digital surface model by performing point cloud rasterization interpolation on the visible light point cloud and the SAR point cloud, unify the coordinates and elevations of the multi-source digital surface model, and then perform multi-source elevation fusion to obtain a fused digital surface model; and perform point cloud conversion on the fused surface model to obtain a three-dimensional point cloud of the target area.

Images