Satellite image disparity map and point cloud generation method based on SGM guided by physical simulation and deep learning, computer equipment and storage medium

Abstract

The invention relates to a satellite image disparity map and point cloud generation method based on physical simulation and deep learning guided SGM, computer equipment and a storage medium, and the method comprises the steps: S1, generating a simulation stereo image pair and an ideal disparity map truth value through employing DOM, DSM and satellite RFM parameters of a target region, and taking the truth values as training data; s2, using the training data to supervise and train a stereo matching network, and using the trained network to output an initial prediction disparity map for a real satellite image; s3, using the initial prediction disparity map as prior information, defining a dynamic local disparity search window for an SGM algorithm to carry out constrained cost aggregation and energy minimization, and obtaining a refined final disparity map; and S4, based on the refined final disparity map and the RFM parameters, generating a three-dimensional point cloud through forward intersection. According to the invention, the satellite image three-dimensional reconstruction precision and reliability can be improved.

Description

Technical Field

[0001] This application relates to the field of satellite image processing technology, and in particular to a method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM, as well as computer equipment and storage media. Background Technology

[0002] In recent years, with the rapid development of my country's aerospace industry, a series of high-resolution stereo mapping satellites, represented by ZY-3 and GF-7, have been successfully launched and put into operational use. These satellites can efficiently acquire stereo image pairs globally, providing valuable data sources for geographic information acquisition, topographic mapping, and 3D modeling in my country and even globally. Consequently, how to quickly and automatically produce high-precision, high-reliability 3D geographic information products from these massive amounts of stereo images has become one of the core problems that urgently need to be solved in the field of photogrammetry and remote sensing.

[0003] In this 3D reconstruction process, generating an accurate disparity map is a crucial and interconnected step, and its quality directly determines the accuracy of the final digital surface model or point cloud. Currently, the mainstream technical approaches are mainly divided into traditional algorithm-based and deep learning-based methods, but both face significant challenges when processing domestically produced satellite imagery.

[0004] In traditional algorithms, semi-global matching (SGM) and its variants are widely used due to their good balance between accuracy and efficiency. However, SGM is essentially a dense matching algorithm that optimizes locally. Given the large swath width and complex terrain features of satellite imagery, SGM needs to search over a vast range of parallax, which is not only computationally intensive but also inevitably leads to numerous mismatches in weakly textured areas (such as calm water surfaces and asphalt roads), repetitive textured areas (such as regular farmland and building complexes), and occluded areas due to similarity of matching primitives or missing information. This results in significant noise, holes, and geometric inconsistencies in the generated parallax map. These defects directly impact the 3D point cloud, severely affecting the usability of subsequent products.

[0005] With the advancement of artificial intelligence, deep learning-based stereo matching methods have demonstrated immense potential. These data-driven methods can extract high-level features from massive datasets and implicitly construct complex matching constraints, thus surpassing traditional methods on numerous public datasets. However, deep learning is a supervised learning method, and its performance heavily relies on training with large-scale, high-precision disparity map-stereo pair datasets. In the field of satellite remote sensing, obtaining such real-world datasets is a significant challenge: field measurements are extremely costly and impractical on a large scale, while disparity maps produced using existing traditional processing methods inevitably contain substantial errors. Currently, some researchers have attempted to use disparity maps generated by SGM (Specialized Gaussian Mutual Aspect Ratio) as "pseudo-ground values" to train models, but this essentially leads to a paradox: the model's learning object is itself a flawed and inaccurate intermediate product. This makes it difficult for deep learning models to learn correct and robust matching relationships, limiting their generalization ability and ultimate performance ceiling from the outset, preventing them from fully realizing their theoretical advantages.

[0006] In summary, the bottlenecks of existing technologies can be attributed to the following: with the increasing abundance of domestic satellite data sources, there is a lack of an effective method to provide high-quality, high-precision supervisory data for deep learning models, which hinders the deep application and performance breakthrough of deep learning technology in satellite 3D reconstruction; at the same time, the inherent error problem of traditional SGM methods has not been fundamentally solved. Summary of the Invention

[0007] Based on this, the purpose of this invention is to address the above-mentioned technical problems by providing a method, computer equipment, and storage medium for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM. This method can solve the problems in the prior art where the lack of high-precision real disparity maps as supervision signals makes it difficult for deep learning models to learn correct stereo matching relationships, and the high mismatch rate of traditional SGM algorithms in complex scenes, thereby improving the accuracy and reliability of satellite image 3D reconstruction.

[0008] To achieve the aforementioned objectives, the first aspect of this application provides a method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM, comprising: S1: Obtain high-precision digital orthophoto image DOM and digital surface model DSM of the target area, and generate simulated stereo image pairs by forward projection based on the rational function model RFM parameters of the target satellite, and calculate the true value of the corresponding ideal disparity map. S2: Using the simulated stereo image pairs and their corresponding ideal disparity map ground values ​​as training data, a stereo matching deep learning network is trained in a supervised learning manner to obtain a trained network model; real satellite stereo image pairs are input into the trained network model to output an initial predicted disparity map. S3: Input the initial predicted disparity map as prior information into the semi-global matching (SGM) algorithm, define a dynamic local disparity search window centered on each pixel value in the initial predicted disparity map for the semi-global matching (SGM) algorithm, perform constrained cost aggregation and energy minimization within the window, and obtain the refined final disparity map. S4: Based on the final disparity map and the rational function model (RFM) parameters of the satellite, generate a three-dimensional point cloud through forward intersection.

[0009] Preferably, step S1 specifically includes: S11. Acquire and register the high-precision digital orthophoto image DOM and digital surface model DSM of the target area; S12. Load the rational function model (RFM) parameters of the target satellite's dual-line array camera; S13. Traverse each grid point on the digital surface model (DSM) to obtain its three-dimensional coordinates and color values ​​on the digital orthophoto image (DOM); using the rational function model (RFM) parameters, project the three-dimensional coordinates of each grid point onto the image coordinates of the simulated left and right images respectively, and assign the corresponding color values ​​to the corresponding image coordinates. After resampling, generate the simulated left and right images. S14. Based on the geometric relationship between the simulated left image and the simulated right image generated in step S13, directly calculate the position of the corresponding point of each pixel on the simulated left image on the simulated right image to obtain the true value of the ideal disparity map that corresponds one-to-one with the simulated stereo image pair.

[0010] Preferably, in step S13, the specific process of forward projection using RFM is as follows: after normalizing the three-dimensional coordinates of the grid points on the digital surface model (DSM), the coordinates are substituted into the polynomial of the rational function model (RFM) to calculate the normalized image point coordinates on the simulated image, and then the actual pixel coordinates are obtained by inverse normalization.

[0011] Preferably, in step S14, the calculation of the true value of the ideal disparity map follows the following: for the simulated left image pixels on Its corresponding ideal parallax Defined as:

[0012] in, It is in the simulation of right image The coordinates of the image points with the same name on the image.

[0013] Preferably, in step S2, the stereo matching deep learning network includes GWC-Net.

[0014] Preferably, in step S2, the loss function used in the supervised learning is the smoothed L1 loss function. This loss function L combines the advantages of both L1 and L2 losses, is insensitive to outliers, and is conducive to training convergence. The definition of the smoothed L1 loss function is as follows:

[0015]

[0016] Where N is the sample size. and represents the predicted disparity and the ground truth disparity of the i-th pixel, respectively.

[0017] Preferably, in step S3, the dynamic local disparity search window is defined as: for each pixel p in the image, its candidate disparity set Limited to:

[0018] in, The value of the initial predicted disparity map at pixel p. This is the preset search radius.

[0019] Preferably, in step S3, in the candidate disparity set Internally, by minimizing the energy function of the SGM To solve for the final disparity, the energy function is:

[0020] in, This is a data item used to measure the parallax of pixel P. The matching cost is calculated using the Census transform, which is insensitive to radiation differences; and These are the first and second parts of the smoothing term, which are used to constrain the disparity relationship between adjacent pixels p and q. and The penalty coefficient and < The first part of the smoothing term penalizes the change in disparity between adjacent pixels p and q when it is 1. Smooth slopes are allowed; the second part of the smoothing term penalizes when the disparity change between adjacent pixels p and q is greater than 1. This allows for parallax abrupt changes at the edges of objects; This is an indicator function; its value is 1 when the condition inside the parentheses is true, and 0 otherwise. This represents the neighborhood set of pixel p.

[0021] To achieve the purpose of the invention, the second aspect of this application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM as described in the first aspect of the above technical solution.

[0022] To achieve the purpose of the invention, the third aspect of this application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, it implements the steps of the method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM as described in the first aspect of the above technical solution.

[0023] Compared with the prior art, the beneficial effects of this invention are: This invention constructs a simulation dataset with strict geometric consistency to fully train the model, and combines it with SGM-guided refinement optimization to build a serial process consisting of three key stages: "physical simulation dataset generation", "deep learning model training and prediction", and "SGM-guided refinement". This process can provide high-quality and high-precision supervision data for deep learning models, break through the performance bottleneck caused by "pseudo-true values", and effectively integrate the advantages of deep learning and traditional methods, thereby improving the accuracy and reliability of satellite image 3D reconstruction. Attached Figure Description

[0024] Figure 1 This is a schematic diagram illustrating the steps of the method described in this application; Figure 2 This is a schematic diagram illustrating the specific process of the method described in this application in the embodiments; Figure 3 This is a schematic diagram of the GWC-Net network structure in the embodiment; Figure 4 This is a comparison diagram of the left epipolar image, the right epipolar image, and the disparity map of the traditional SGM matching in the embodiment; Figure 5 This is a comparison of the left epipolar image, the right epipolar image, and the disparity map obtained by deep learning prediction of the simulation data using the method of this application in the embodiment. Figure 6 This is a comparison image of the left epipolar image, the right epipolar image, and the disparity map obtained by SGM refinement using the method of this application in the embodiment; Figure 7 This is a comparison of the disparity maps predicted by existing deep learning methods for the left and right epipolar images and the disparity maps obtained by the traditional SGM global search method in the embodiment. Figure 8This is a comparison chart showing the disparity map obtained by existing deep learning methods from the left nuclear line image, traditional SGM matching, and disparity map obtained by traditional SGM global search, the disparity map obtained by deep learning model based on simulation data of this application, and the disparity map obtained by SGM refinement of the disparity map obtained by the method of this application from the disparity map obtained by deep learning model based on simulation data of this application. Figure 9 The point cloud shown in this example is generated by the traditional SGM global search. Figure 10 The initial values ​​of the disparity map generated by deep learning from the simulation data in this embodiment are used to refine the point cloud using SGM. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of the invention. The following embodiments are used to illustrate the invention but are not intended to limit its scope.

[0026] Example 1 Embodiment 1 of this application provides a method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM, such as... Figure 1 As shown, it includes: S1: Obtain high-precision digital orthophoto image DOM and digital surface model DSM of the target area, and generate simulated stereo image pairs by forward projection based on the rational function model RFM parameters of the target satellite, and calculate the true value of the corresponding ideal disparity map. S2: Using the simulated stereo image pairs and their corresponding ideal disparity map ground values ​​as training data, a stereo matching deep learning network is trained in a supervised learning manner to obtain a trained network model; real satellite stereo image pairs are input into the trained network model to output an initial predicted disparity map. S3: Input the initial predicted disparity map as prior information into the semi-global matching (SGM) algorithm, define a dynamic local disparity search window centered on each pixel value in the initial predicted disparity map for the semi-global matching (SGM) algorithm, perform constrained cost aggregation and energy minimization within the window, and obtain the refined final disparity map. S4: Based on the final disparity map and the rational function model (RFM) parameters of the satellite, generate a three-dimensional point cloud through forward intersection.

[0027] like Figure 2As shown, the specific process of steps S1-S4 includes three stages, where S1 belongs to stage one: generation of high-quality simulation dataset based on the physical imaging model; S2 belongs to stage two: training and preliminary prediction of the deep learning model; S3 belongs to stage three: guided refinement based on SGM; and step S4 is summarized at the end of stage three. Steps S1-S4 are detailed below: Step S1 is the cornerstone and key innovation for achieving high-precision stereo matching. It creatively solves the biggest bottleneck faced by deep learning in satellite remote sensing—the lack of large-scale, high-precision, and truth-reliable supervisory data. Existing technologies are forced to use erroneous "pseudo-truth values" due to the inability to obtain true parallax values, thus limiting the upper limit of model performance. This application takes a different approach, fundamentally ensuring the quality and reliability of training data through physics-driven simulation technology.

[0028] Step S1 specifically includes the following steps: S11: Collaborative preprocessing of multi-source geospatial data.

[0029] Acquire high-precision digital orthophotos (DOM) and digital surface models (DSM) of the target area that have been accurately registered. The DOM provides accurate spectral information of the ground features, while the DSM provides their three-dimensional spatial structure. The core of preprocessing is to ensure the consistency of the DOM and DSM in terms of spatial reference frame and geometric resolution, and to eliminate systematic errors introduced by inconsistent data sources. This is the primary prerequisite for generating geometrically non-destructive simulation data.

[0030] S12: Geometric modeling and parameter preparation for rational function model (RFM).

[0031] Load the rational function model (RFM) parameters (RPC parameters) of the dual-line array camera (forward-looking, backward-looking) of the target satellite platform (such as GF-7, ZY-3). RFM accurately describes the transformation relationship from object space coordinates to image space coordinates through rational fractions, and its inverse solution form is shown in formula (1):

[0032] in,( , () represents the normalized coordinates of the image plane coordinate system. These are the normalized coordinates of the object coordinate system. The purpose of the normalization parameter is to ensure that the values ​​of the image-space coordinates and the object-space coordinates are within the range of -1 to 1, avoiding calculation errors caused by differences in coordinate magnitudes. The normalization of the image-space coordinates and the object-space coordinates follows formulas (2)-(6) respectively:

[0033]

[0034]

[0035]

[0036]

[0037] In the formula, Represents the coordinates of the image point in the image plane coordinate system; Represents the object-space coordinates of a point on the ground; , , , The normalization parameter for the image-side coordinates; , , , , , The normalization parameter for the object coordinates.

[0038] In this embodiment, the original, unmodified RPC parameters published by the satellite data provider are directly applied. To eliminate the inherent scale uncertainty of the model, industry standard practice uses the polynomial in the denominator of formula (1). , The constant term is fixed at 1. This invention follows this standard to ensure the mathematical rigor and uniqueness of the geometric model used. The RFM parameters are known inputs and are not involved in subsequent adjustment optimization.

[0039] S13: Simulated image generation based on RPC forward projection.

[0040] This step is the core of generating the simulation dataset. Its core idea is to use the RPC parameters obtained in step S12 to perform a deterministic, forward geometric projection, achieving a deterministic geometric transformation from the object-side 3D coordinates to the image-side 2D coordinates. The specific implementation process is as follows: 1. Preparation and normalization of object coordinates: Traverse every grid point on the Digital Surface Model (DSM), which contains an object-space 3D coordinate. Transform the coordinates to the geodetic coordinate system. And using the normalization parameters in the RPC parameters, the parameters are normalized to the parameter space required by the RPC model through formulas (4), (5), and (6). .

[0041] 2. Image point coordinate calculation: Normalized object points Substituting these values ​​into the rational function model (RFM) formula (1) for the forward-looking and rear-looking cameras respectively, the normalized image point coordinates (x, y) on the simulated left and right images are calculated. Wherein, , , , There are four normalized coordinates Structure-independent cubic polynomials. Taking a polynomial as an example, its complete form is shown in formula (7). (Other polynomials have similar structures, but each has its own independent set of coefficients.)

[0042] In this polynomial, These are specific coefficients provided in the RPC parameter file.

[0043] 3. Image-side coordinate inverse normalization: The calculated normalized image point coordinates By performing inverse normalization using formulas (2) and (3), the actual pixel coordinates on the simulated image are restored. .

[0044] 4. Image rendering: The color values ​​(e.g., R, G, B) of the object point P obtained on the DOM are assigned to the calculated pixel positions (X, Y) in the simulated image. Due to the discrete nature of the object point, the projected image points may not cover all pixels of the image or may overlap. To solve this discrete sampling problem, a resampling operation is performed to generate a visually continuous and complete simulated image. This resampling operation only affects the spectral (color) information of the simulated image, aiming to fill in pixel gaps caused by discrete projection, improve the visual continuity of the simulated image, and thus ensure the training effectiveness of the deep learning model. Independently, the ground truth of the disparity map... It is generated directly and independently based on strict imaging geometry (Equation (8)). Its process is not affected by the aforementioned resampling strategy, thus ensuring the absolute accuracy of the disparity map in geometry and the strict, one-to-one geometric relationship between it and the stereo image pair.

[0045] S14: Calculation of the true value of the ideal disparity map.

[0046] Based on the known and deterministic geometric relationships between the simulated left and right images, the corresponding point position of each pixel in the left image on the right image is directly calculated, thus obtaining a high-precision, noise-free disparity map ground truth that corresponds one-to-one with the simulated stereo image pair (i.e., the simulated stereo image pair). For the simulated left image... pixels on Its corresponding ideal parallax Defined as:

[0047] in, It is in the simulation of right image The coordinates of the corresponding image points on the left image are used. By traversing all pixels of the left image, a high-precision disparity map ground truth that precisely corresponds to the simulated image pair and is free of matching noise and holes can be generated. .

[0048] Step S2 utilizes the geometrically rigorous simulation dataset generated in the first stage to train the advanced stereo matching network GWC-Net using supervised learning. Its core objective is to enable the model to learn highly discriminative image features and robust matching rules from "perfect" data, thereby gaining powerful inference capabilities that can be directly generalized to real satellite imagery.

[0049] Step S2 includes the following steps: S21: Model building based on GWC-Net.

[0050] It should be noted that other deep learning stereo matching networks, such as ACV-Net, can also be used. In this embodiment, GWC-Net is preferred as the core deep learning architecture for stereo matching.

[0051] A schematic diagram of the GWC-Net network structure is shown below. Figure 3 As shown, the GWC-Net network includes a feature extraction network, a cascaded cost body, a group correlation cost body, a joint cost body, and a 3D aggregation network; the feature extraction network extracts the left and right images respectively (I... L I R The system constructs a cascaded cost body by grouping feature maps and sharing weights. It then convolves and compresses the channels of the feature maps from both images, concatenating them along the channel dimension. A group correlation cost body groups the feature channels and calculates the matching cost independently. The cascaded and group correlation cost bodies form a joint cost body, which is input into a 3D convolutional network for depth optimization and filtering, outputting a predicted disparity image. GWC-Net achieves high-precision and high-efficiency stereo matching through its group correlation cost body and 3D convolutional convolutional network. The group correlation mechanism constructs an information-rich and computationally efficient cost body by grouping feature channels and calculating the matching cost independently. The 3D convolutional convolutional network then performs depth optimization and filtering on this cost body, effectively improving the accuracy of disparity estimation. Furthermore, its structure is particularly suitable for processing wide-swath satellite imagery.

[0052] S22: Supervised training based on geometric truth values.

[0053] The simulated left image generated in stage one and simulated right image As input to the GWC-Net network, the corresponding high-precision disparity map ground truth is used. As a supervisory signal, the network is trained end-to-end. During this process, GWC-Net learns the accurate mapping from stereo image pairs to disparity maps. The optimization objective of the training process is to minimize the network's predicted disparity. with truth value The differences between them. This invention uses smoothed L1 loss as the loss function, which combines the advantages of both L1 and L2 losses, is insensitive to outliers, and facilitates training convergence. Its definition is as follows:

[0054]

[0055] Where N is the sample size. and These represent the predicted disparity and ground truth disparity of the i-th pixel, respectively. Through this optimization process, GWC-Net ultimately learns accurate image features and disparity mapping rules based on physical geometric priors.

[0056] S23: Generalized prediction for real satellite imagery.

[0057] Real ZY-3 or GF7 satellite stereo images The input is directly fed into the already trained GWC-Net model. Through forward propagation, the network utilizes its learned powerful feature discrimination and matching capabilities to output an initial predicted disparity map. Since GWC-Net is trained on geometrically perfect simulation data and its inherent attention mechanism can effectively improve the robustness of features, the initial predicted disparity map not only inherits the geometrically consistent prior of the simulation data, but also has the ability to generalize to complex scenes in real images, providing a high-quality starting point for subsequent fine processing.

[0058] Step S3 is guided refinement based on SGM, aiming to solve the problems of high computational complexity and high false matching rate caused by the blind search within the global disparity range of the traditional SGM algorithm. Its basic idea is to use the disparity map predicted by deep learning as a strong prior to strictly constrain the search space of SGM, thereby realizing a paradigm shift from "global coarse search" to "local refinement", which combines the semantic robustness of deep learning with the local accuracy advantage of SGM.

[0059] Step S3 includes the following steps: S31: Prior information injection and problem reconstruction.

[0060] The initial predicted disparity map obtained in stage two This serves as input to the SGM algorithm. This step is not simply data transfer, but a fundamental restructuring of the optimization problem that SGM aims to solve. It provides a crucial initial solution for SGM, transforming what was originally an unconstrained or weakly constrained global optimization problem into a highly constrained local optimization problem centered on deep learning prediction.

[0061] S32: Constrained cost aggregation and dynamic search range definition.

[0062] Based on prior information, a dynamic, pixel-level local disparity search window is defined for the SGM. For each pixel p in the image, its candidate disparity set... Strictly limited to:

[0063] in, For the preset search radius (e.g.) =3), this definition means that the cost calculation and aggregation process of SGM is only within this narrow window. This is performed internally. Unlike traditional SGM, which requires traversing the entire parallax range... In comparison, the computational complexity of this invention is reduced from Significantly reduced to Where W and H are the height and width of the image size, this is not only a leap in efficiency, but also fundamentally eliminates a large number of erroneous candidate parallaxes that lead to mismatches.

[0064] S33: Energy minimization and fine-grained matching.

[0065] The purpose of this step is to use SGM's powerful local optimization capabilities to refine and regularize the high-confidence parallax skeleton provided by deep learning at the sub-pixel level, effectively filling in any small gaps that may exist in the deep learning predictions and ensuring that its edges are precisely aligned with the object's outline.

[0066] In the constraint window Internally, SGM minimizes its standard energy function. To solve for the final disparity map : (12) in, This is a data item used to measure the parallax of pixel P. The matching cost is calculated using the Census transform, which is insensitive to radiation differences; and These are the first and second parts of the smoothing term, which are used to constrain the disparity relationship between adjacent pixels p and q. and The penalty coefficient and < The first part of the smoothing term applies a smaller penalty when the disparity change between adjacent pixels p and q is 1. Smooth slopes are allowed; the second part of the smoothing term imposes a larger penalty when the disparity change between adjacent pixels p and q is greater than 1. This allows for parallax abrupt changes at the edges of objects; This is an indicator function; its value is 1 when the condition inside the parentheses is true, and 0 otherwise. This represents the neighborhood set of pixel p. This energy function ensures that the disparity map remains smooth and has sharp edges when optimized within a local window.

[0067] Step S4 includes: S41: 3D product generation.

[0068] Utilizing refined high-precision disparity maps By combining the rational function model of satellites and using the forward intersection principle, parallax information is converted into three-dimensional coordinates in the real world. This process ultimately generates high-density, high-precision 3D point clouds, providing a reliable data foundation for applications such as digital surface model production and 3D reality modeling.

[0069] Example 2 Embodiment 2 of this application is based on Embodiment 1, and experiments are conducted to verify the effectiveness of the method of this application and to compare it with traditional methods.

[0070] This experiment uses the three-dimensional reconstruction of a city area based on the imagery of the domestic Gaofen-7 (GF-7) satellite as an example, and the method of this application is explained in detail with reference to the accompanying figures.

[0071] Experimental environment and data preparation: Hardware: Equipped with an Intel(R) Core(TM) Ultra7 265K processor, 32GB of RAM and an NVIDIA RTX5060Ti graphics card with 16GB of video memory; Software: Python 3.13, PyTorch 2.3.1.

[0072] Basic data: 1. The region has a DOM with a resolution of 0.3 meters and a DSM with a grid spacing of 0.5 meters.

[0073] 2. Forward and backward panchromatic stereo image pairs from the GF-7 satellite (imaging time is close to that of DOM / DSM).

[0074] 3. Satellite Geometric Model Parameters: The key input used in this invention is the Rational Function Model (RFM) parameters (i.e., RPC parameters) provided with the GF-7 satellite imagery. This parameter file is provided by the satellite data distribution agency (such as the China Center for Resources Satellite Data and Application), and its standard format includes normalized parameters for the image and object sides, as well as 80 polynomial coefficients for coordinate calculation. These parameters fully define the imaging geometric relationships described by formulas (1) to (6), and are the basis for high-precision simulation in Stage 1. The correspondence between the key normalized parameters and the formulas described in this invention is as follows: 1) The offset and scaling parameters of the object's latitude, longitude, and elevation (LAT_OFF, LONG_OFF, HEIGHT_OFF, LAT_SCALE, LONG_SCALE, HEIGHT_SCALE) correspond to the parameters in formulas (4), (5), and (6) of this invention, respectively. , , , , , Used for object point coordinates Normalized calculation.

[0075] 2) The offset and scaling parameters (LINE_OFF, SAMP_OFF, LINE_SCALE, SAMP_SCALE) of the image row and column coordinates correspond to the values ​​in formulas (2) and (3) of this invention, respectively. , , , Used for image point coordinates Normalization and denormalization calculations.

[0076] 3) The polynomial coefficients (LINE_NUM_COEFF_n, LINE_DEN_COEFF_n, SAMP_NUM_COEFF_n, SAMP_DEN_COEFF_n, where n=1~20) constitute the polynomials in the examples of formula (1) and formula (5). , , , Specific coefficients .

[0077] The above steps yield realistic physical simulation data for the left and right images. Using the traditional SGM (Global Search) algorithm, the global disparity search range is set to -144 to 144. The Census transform is used as the matching cost result. Figure 4 As shown, Figure 4 The left epipolar image (referred to as the left image) in the embodiment ( Figure 4Left and right epipolar images (referred to as right images) Figure 4 (Chinese) disparity map of traditional SGM matching ( Figure 4 The comparison image on the right.

[0078] Depend on Figure 4 As can be seen, the traditional algorithm SGM (Global Search) performs reasonably well in areas with rich texture, reflecting the basic outline. However, in areas with large areas of repetitive texture (such as fields, steps, etc.), large areas of holes appear. In areas lacking texture, large areas of mismatches occur. These errors directly lead to high noise levels in the subsequently generated 3D point cloud and distortion of building facades (such as...). Figure 9 As shown in the figure, it is difficult to meet the requirements of high-precision applications.

[0079] The method of this invention includes: simulation data generation, deep learning prediction, and SGM-guided refinement. Using the RPC parameters of DOM, DSM, and GF-7, simulated stereo image pairs and corresponding ideal disparity maps are generated through back projection. Model training: The GWC-Net network is selected and trained using the aforementioned simulation dataset, with a smoothed L1 loss function. Initial prediction: Real GF-7 stereo images are input into the trained model to obtain an initial predicted disparity map. After the model is trained until the loss convergence, it is compared with the three sets of real satellite images mentioned above. Figure 5 As shown, Figure 5 Left nuclear line image ( Figure 5 Left and right epipolar images ( Figure 5 (in the middle), disparity maps predicted by deep learning using the method described in this application ( Figure 5 The comparison image on the right shows how the deep learning model predicts the disparity map.

[0080] Depend on Figure 5 As can be seen from the results, compared with the traditional SGM method (global search), the deep learning model of this invention exhibits extremely strong robustness in weakly textured regions. The disparity maps of areas such as roads and squares are very smooth, coherent, and almost noise-free. Thanks to the high-level semantic information and robust matching rules learned by the model from high-quality simulation data, the outlines of core features such as buildings are complete and continuous, effectively avoiding problems such as striped breaks, holes, and mismatches that are prone to occur in traditional SGM. It should be noted that the initial predicted disparity map output by this deep learning model still has certain limitations: at extremely small structures or object edges, there may be slight blurring or local sub-pixel inaccuracies, and the edge sharpness is not as good as that of traditional global search SGM. This is the core reason why further optimization through SGM-guided refinement is needed. Therefore, the disparity map predicted by the deep learning method of this application ( Figure 5 (Right) is used as the initial value for the SGM search in the method of Example 1, guiding the SGM to be refined. The result of the refinement is as follows: Figure 6 As shown, Figure 6 Left nuclear line image ( Figure 6 Left and right epipolar images ( Figure 6 (middle), disparity map obtained by SGM refinement using the method of this application ( Figure 6 The comparison image on the right shows that... Figure 6 It can be seen from the initial predicted disparity map ( Figure 5 (Right) is the prior, and a local search window is set for each pixel (in this embodiment, it is set as follows). =3), under this constraint, the SGM algorithm is run for fine-tuning matching to obtain the final disparity map. ( Figure 6 (Right). A comparison shows... Perfectly inherited All the advantages: smooth textured areas and complete building outlines. At the same time, it effectively compensates for... The drawback is that parallax becomes sharper and more precise at object edges and details through local optimization using SGM. Final result. Visually optimal, it has neither ( Figure 4 The noise and voids on the right are even more pronounced than those on the right. It has more refined details.

[0081] To illustrate the effectiveness of this invention, a network trained using existing deep learning methods was constructed by combining real images with a disparity map obtained through the traditional SGM global search method to predict the aforementioned three sets of images. The results are as follows: Figure 7 As shown, Figure 7 Left nuclear line image ( Figure 7 Left and right epipolar images ( Figure 7 The disparity map obtained by the traditional SGM global search method is used to predict the disparity map using existing deep learning methods. Figure 7 A comparison image (right). Disparity maps obtained using existing deep learning methods ( Figure 7 (Right) Local noise is present in the weak texture area and the building edge outline is slightly blurred.

[0082] To clearly compare the advantages of this invention, the following three sets of images are presented in sequence: the left image, the traditional SGM global search method, the disparity map predicted by the model using deep learning based on real image data, the disparity map predicted by the model using deep learning based on simulated data, and the disparity map obtained from simulated data after SGM refinement. Figure 8 As shown, Figure 8 From left to right, the images are of the left epipolar line ( Figure 8 Left 1), Traditional SGM Matching ( Figure 8(Left 2) Disparity map predicted by existing deep learning methods using real images combined with the disparity map obtained by the traditional SGM global search method. Figure 8 (in Chinese), the disparity map predicted by the deep learning model of the simulation data in this application (in Chinese). Figure 8 (Right 2) The disparity map obtained by refining the disparity map predicted by the deep learning model using the SGM method of this application is obtained from the simulation data of this application. Figure 8 (Right 1) shows the effect more intuitively. From Figure 8 It is clearly evident that the disparity map predicted by the deep learning model using simulation data for real GF7 satellite imagery exhibits better texture and edge detail than the model obtained by directly using real satellite imagery data and the traditional SGM global search method. The disparity map predicted by this invention has clearer sharpness and edge information, and the SGM refinement, based on the predicted disparity map as the initial value, achieves even higher accuracy.

[0083] By comparing the disparity map obtained using the traditional SGM global search method with the disparity map obtained by this invention to generate a point cloud, the significant advantages of this invention are clearly evident. Figure 9 The point cloud generated by the traditional SGM method shows the presence of outliers. Figure 10 The point cloud generated by the method in this application is significantly better than the traditional global search and matching method.

[0084] To further demonstrate the advantages of this invention, the results were compared using EPE (mean disparity error) and D (error disparity ratio), as shown in Table 1. The method of this invention shows a certain degree of reduction in both EPE and D. This reduction is mainly due to the invention's use of a simulation dataset with a completely correct mapping relationship between the left and right images and the disparity map, allowing the model to fully learn this correct matching relationship.

[0085] Table 1. Accuracy comparison between the method of this invention and the traditional SGM global search method

[0086] Example 3 This embodiment 3, based on embodiments 1 and 2, provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the satellite image disparity map and point cloud generation method based on physical simulation and deep learning guided SGM described in embodiment 1.

[0087] The memory, used to store programs, can include volatile memory, such as random-access memory (RAM), including static random-access memory (SRAM) and double-data-rate synchronous dynamic random-access memory (DDR SDRAM); it can also include non-volatile memory, such as flash memory. The memory is used to store computer programs (such as application programs and functional modules that implement the above methods), computer instructions, etc.

[0088] Computer programs, computer instructions, and other data can be stored in partitions within one or more memory locations. Furthermore, these computer programs, computer instructions, and data can be accessed by a processor.

[0089] The processor and memory can be separate structures or integrated structures. When the processor and memory are separate structures, they can be coupled together via a bus.

[0090] This embodiment 3 also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the satellite image disparity map and point cloud generation method based on physical simulation and deep learning-guided SGM described in embodiment 1 above.

[0091] The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or device or apparatus) or a propagation medium. Computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), hard disks, and optical discs. Optical discs include optical disc-read-only memory (CD-ROM), optical disc-read / write (CD-R / W), and DVDs.

[0092] In summary, this invention provides a method, computer equipment, and storage medium for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM, which can achieve the following beneficial effects: 1. It fundamentally solves the pain point of scarce training data for deep learning in the field of satellite remote sensing. By simulating and generating ground truth datasets of disparity maps with strict geometric consistency through DOM and DSM, it is low-cost, infinitely scalable, and lays a solid foundation for training high-performance deep learning models.

[0093] 2. Significantly improves the performance ceiling and robustness of deep learning models. The model learns accurate matching rules based on deterministic geometric relationships, rather than erroneous "pseudo-true values," enabling it to learn more essential image matching features. This results in stronger generalization ability and higher prediction accuracy for difficult regions such as weak textures and repetitive textures.

[0094] 3. This invention achieves a deep integration and complementary advantages between deep learning and traditional SGM. It is not a simple replacement, but rather a creative combination of the two. Deep learning acts as "global navigation," providing robust parallax priors; SGM acts as "local refinement," performing pixel-level optimization within certain regions. This collaborative model overcomes the shortcomings of their respective pure methods, achieving a "1+1>2" effect.

[0095] 4. The final product has high accuracy and reliability. The final disparity map generated through the above process has significantly fewer noise, holes, and mismatched points than the results produced by a single SGM or a deep learning model trained based on "pseudo-ground values," thus ensuring the high quality of subsequent 3D point cloud products.

[0096] 5. Significantly improves the efficiency of 3D reconstruction. This invention greatly reduces the search space of SGM through deep learning priors, decreasing the computational complexity of SGM from O(W*H*D) to O(W*H*2r), where 2r <

Claims

1. A method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM, characterized in that, Includes the following steps: S1: Obtain high-precision digital orthophoto image DOM and digital surface model DSM of the target area, and generate simulated stereo image pairs by forward projection based on the rational function model RFM parameters of the target satellite, and calculate the true value of the corresponding ideal disparity map. S2: Using the simulated stereo image pairs and their corresponding ideal disparity map ground values ​​as training data, a stereo matching deep learning network is trained in a supervised learning manner to obtain a trained network model; real satellite stereo image pairs are input into the trained network model to output an initial predicted disparity map. S3: Input the initial predicted disparity map as prior information into the semi-global matching (SGM) algorithm, define a dynamic local disparity search window centered on each pixel value in the initial predicted disparity map for the semi-global matching (SGM) algorithm, perform constrained cost aggregation and energy minimization within the window, and obtain the refined final disparity map. S4: Based on the final disparity map and the rational function model RFM parameters of the satellite, a three-dimensional point cloud is generated using the forward intersection principle.

2. The method according to claim 1, characterized in that, Step S1 specifically includes: S11. Acquire and register the high-precision digital orthophoto image DOM and digital surface model DSM of the target area; S12. Load the rational function model (RFM) parameters of the target satellite's dual-line array camera; S13. Traverse each grid point on the digital surface model (DSM) to obtain its three-dimensional coordinates and color values ​​on the digital orthophoto image (DOM); using the rational function model (RFM) parameters, project the three-dimensional coordinates of each grid point onto the image coordinates of the simulated left and right images respectively, and assign the corresponding color values ​​to the corresponding image coordinates. After resampling, generate the simulated left and right images. S14. Based on the geometric relationship between the simulated left image and the simulated right image generated in step S13, directly calculate the position of the corresponding point of each pixel on the simulated left image on the simulated right image to obtain the true value of the ideal disparity map that corresponds one-to-one with the simulated stereo image pair.

3. The method according to claim 2, characterized in that, In step S13, the specific process of forward projection using RFM is as follows: after normalizing the three-dimensional coordinates of the grid points on the digital surface model DSM, the coordinates are substituted into the polynomial of the rational function model RFM to calculate the normalized image point coordinates on the simulated image, and then the actual pixel coordinates are obtained by inverse normalization.

4. The method according to claim 2, characterized in that, In step S14, the calculation of the true value of the ideal disparity map follows the following: for the simulated left image pixels on Its corresponding ideal parallax Defined as: in, It is in the simulation of right image The coordinates of the image points with the same name on the image.

5. The method according to claim 1, characterized in that, In step S2, the stereo matching deep learning network includes GWC-Net.

6. The method according to claim 5, characterized in that, In step S2, the loss function used in the supervised learning is the smoothed L1 loss function L. This loss function L combines the advantages of L1 and L2 losses, is insensitive to outliers, and is conducive to training convergence. The definition of the smoothed L1 loss function is as follows: Where N is the sample size. and represents the predicted disparity and the ground truth disparity of the i-th pixel, respectively.

7. The method according to claim 1, characterized in that, In step S3, the dynamic local disparity search window is defined as: for each pixel p in the image, its candidate disparity set... Limited to: in, The value of the initial predicted disparity map at pixel p. This is the preset search radius.

8. The method according to claim 7, characterized in that, In step S3, in the candidate disparity set Internally, by minimizing the energy function of the SGM To solve for the final disparity, the energy function is: in, This is a data item used to measure the parallax of pixel P. The matching cost is calculated using the Census transform, which is insensitive to radiation differences; and These are the first and second parts of the smoothing term, which are used to constrain the disparity relationship between adjacent pixels p and q. and The penalty coefficient and < The first part of the smoothing term penalizes the change in disparity between adjacent pixels p and q when it is 1. Smooth slopes are allowed; the second part of the smoothing term penalizes when the disparity change between adjacent pixels p and q is greater than 1. This allows for parallax abrupt changes at the edges of objects; This is an indicator function; its value is 1 when the condition inside the parentheses is true, and 0 otherwise. This represents the neighborhood set of pixel p.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the satellite image disparity map and point cloud generation method based on physical simulation and deep learning-guided SGM as described in any one of claims 1 to 8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for generating satellite image disparity maps and point clouds based on physical simulation and deep learning-guided SGM as described in any one of claims 1 to 8.

Images