A stereo matching method for high-resolution stereo satellite panchromatic image pairs

Abstract

The present application relates to the field of remote sensing image processing technology, and in particular to a stereo matching method for high-resolution stereo satellite panchromatic image pairs. By comprehensively applying combined geometric coding convolution and gated iterative optimization mechanisms, the model's ability to capture complex geometric shapes is significantly enhanced; by aggregating multi-scale cost volumes, the model can effectively match ground objects at different scales, while effectively handling occlusion and shadow areas, thereby improving the accuracy and robustness of matching; the introduction of a loss function with gradient consistency constraints further optimizes the disparity estimation process, ensuring clear distinction of disparity maps in edge areas and smooth transitions in weak texture areas, thereby improving the overall quality of the disparity map. This method aims to solve the problem of how to improve the ability to capture complex geometric shapes in remote sensing image stereo matching while reducing dependence on dataset accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of remote sensing image processing, and in particular to a stereo matching method for high-resolution stereo satellite panchromatic image pairs. Background Art

[0002] In the field of computer vision, stereo matching technology refers to inferring the three-dimensional structure of a scene by comparing the position differences (parallax) of corresponding pixels in a stereo image pair. The development of high-resolution stereo satellite sensors and the improvement of computing power have made large-scale stereo image data processing efficient. Satellite image stereo matching technology, which combines stereo matching technology with high-resolution stereo satellites, plays an important role in global change monitoring, urban planning, disaster response and other fields.

[0003] Traditional stereo matching methods have certain limitations in processing densely built areas, weak texture areas, and shadow areas in stereo satellite panchromatic images, and have high requirements for dataset quality. For imperfect datasets, the effect of capturing the geometric shapes of key areas will be significantly reduced.

[0004] In view of this, this application proposes a stereo matching method for high-resolution stereo satellite panchromatic image pairs, aiming to improve the ability to capture complex geometric shapes in remote sensing image stereo matching while reducing dependence on dataset accuracy, thereby improving the accuracy of the stereo matching algorithm for satellite panchromatic image pairs. Summary of the Invention

[0005] The main purpose of this application is to provide a stereo matching method for high-resolution stereo satellite panchromatic image pairs, aiming to solve the problem of how to improve the ability to capture complex geometric shapes in remote sensing image stereo matching while reducing dependence on dataset accuracy.

[0006] To achieve the above objectives, the present application provides a stereo matching method for a pair of high-resolution stereo satellite panchromatic images, the method comprising:

[0007] S10, normalizes the left / right panchromatic image pairs collected by high-resolution stereo satellites, and uses a shared pyramid feature extraction layer to obtain multiple left / right feature maps including multi-scale pyramid features of different receptive field maps;

[0008] S20, calculating the correlation between the left and right feature maps, and constructing a multi-scale geometric cost volume according to the correlations;

[0009] S30, calculating the full pairwise correlation of each of the left / right feature maps, and constructing a combined cost volume by combining the full pairwise correlation with the multi-scale geometric cost volume, and inputting the combined cost volume into a convolutional gated recurrent unit to iteratively update the disparity, wherein a loss function with gradient consistency constraint is used to guide the training process of the stereo matching network during the iterative process;

[0010] S40, after the iteration is completed, obtain the trained stereo matching network and the disparity estimation result of the high-resolution satellite panchromatic image pair.

[0011] Optionally, the S10 specifically includes:

[0012] S11, normalize the left / right panchromatic image pair:

[0013]

[0014] in, image p is the value of each pixel of the input image, is the average value of all pixel values ​​of the left / right panchromatic image pair, is the standard deviation of all pixel values ​​in the left / right panchromatic image pair;

[0015] S12: Input the normalized left / right full-color image pair into a shared feature extraction network, so that the shared feature extraction network extracts left / right feature maps of different scales:

[0016]

[0017]

[0018] Where, Indicates Left feature map at resolution, Indicates Right feature map at resolution, Indicates the horizontal and vertical coordinates of the corresponding pixel point, H is the height of the input image, W is the width, is the characteristic dimension.

[0019] Optionally, the S20 specifically includes:

[0020] S21, group the left / right feature maps along the channel dimension and calculate the correlation group by group using element-by-element subtraction:

[0021]

[0022] Where, For correlation, is the index of the group, is the disparity index value, Indicates the horizontal and vertical coordinates of the corresponding pixel point, is the number of feature channels, is the number of groups;

[0023] S22: Cost aggregation is performed on the calculated correlations of each scale and the left / right feature maps of the corresponding scale to obtain a multi-scale geometric cost volume.

[0024] Optionally, the S22 specifically includes:

[0025] S221, selecting the current minimum scale correlation;

[0026] S222, generates a multi-scale geometric cost volume by upsampling and fusing higher-scale correlations.

[0027] Optionally, the S30 specifically includes:

[0028] S31, calculate the full pairwise correlation of the left / right feature maps of the corresponding scale;

[0029] S32, determine the initial coarse disparity value of the multi-scale geometric cost volume:

[0030]

[0031] Where, is the initial coarse disparity value, is the disparity index value, is the preset minimum parallax value, is the preset maximum parallax value, is a function of probability distribution;

[0032] S33, storing the full pairwise correlation structure as a correlation volume, and synthesizing the correlation volume and the multi-scale geometric cost volume into a geometric coding cost volume ;

[0033] S34, iteratively update the disparity using multi-layer convolutional gated recurrent units:

[0034]

[0035] Where, It is The disparity value after iterations, It is The disparity value after iterations, To decode the hidden state through two convolutional layers The obtained residual time difference.

[0036] Optionally, in S34, the residual time difference The calculation process specifically includes:

[0037] S341, the geometric coding cost volume Input , input to the multi-layer convolutional gated recurrent unit:

[0038]

[0039] Where, is the input of the multi-layer convolutional gated recurrent unit, Encode the convolutional layer, is the current disparity after k iterations, for Synthetic geometric cost volume of the same scale. The “:” symbol indicates the concatenation of feature dimensions.

[0040] S342, substitute the input into the update gate parameter function:

[0041]

[0042] Where, is the update gate parameter, yes Activation function, the output range is between (0, 1), is the convolution operation, is the hidden state at the previous moment, is the weight of the update gate, is the bias of the update gate, provided by the context network, used to initialize the update parameters;

[0043] And, substituting the input into the function that resets the gate parameters:

[0044]

[0045] Where, is to reset the gate parameters, is the weight of the reset gate, is the bias of the reset gate;

[0046] S343, substituting the update gate parameter and the reset gate parameter into the candidate hidden state:

[0047]

[0048] Where, is a candidate hidden state, is the hyperbolic tangent activation function, the output range is between (-1, 1), W h is the weight of the candidate value, is the bias of the candidate value, " represents the Hadamard product, that is, the element-by-element product;

[0049] S344, substitute the candidate hidden state into the final hidden state function:

[0050]

[0051] Where, is the final hidden state, representing the previous hidden state and candidate hidden states The weighted sum of Make updates;

[0052] S345, decoding hidden states through two convolutional layers , and get the residual time difference .

[0053] Optionally, in S30, the training process of the stereo matching network is guided by a loss function with gradient consistency constraint, which specifically includes:

[0054] S31, introduce the disparity estimation loss function and gradient consistency loss function to adjust the disparity value:

[0055]

[0056] Where, is the disparity estimation loss, is the loss of the coarse disparity map," " is the sum of n items, " ” is the L1 norm, that is, the absolute value loss, is the current disparity value, is the true disparity value, is the attenuation factor;

[0057]

[0058] Where, is the gradient consistency loss, " means that for each coordinate The sum of pixels, represents the gradient value of the disparity map, is the predicted final disparity;

[0059] S32, determining a loss function for gradient consistency constraint according to the disparity estimation loss function and the gradient consistency loss function:

[0060]

[0061] Where, is the loss function with gradient consistency constraint, , is the loss weight.

[0062] In addition, to achieve the above-mentioned purpose, the present application also provides an application of the stereo matching method using the above-mentioned high-resolution stereo satellite panchromatic image pair in disparity estimation.

[0063] In addition, to achieve the above-mentioned purpose, the present application also provides a computer system, which includes: a memory, a processor, and a computer program stored on the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the stereo matching method of the high-resolution stereo satellite panchromatic image pair as described above are implemented.

[0064] In addition, to achieve the above-mentioned purpose, the present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the stereo matching method of the high-resolution stereo satellite panchromatic image pair as described in any one of the above items are implemented.

[0065] This application has at least the following beneficial effects:

[0066] 1. By comprehensively applying combined geometric coding convolution and gated iterative optimization mechanisms, the model's ability to capture complex geometric shapes is significantly enhanced, especially in accurately matching ground objects in remote sensing images;

[0067] 2. The application of multi-scale cost volumes enables the model to effectively match ground objects at different scales while effectively handling occlusion and shadow areas, improving the accuracy and robustness of matching;

[0068] 3. The gradient-constrained loss function further optimizes the disparity estimation process, ensuring clear distinction of the disparity map in edge areas and smooth transition of weak texture areas, thereby improving the overall quality of the disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 This is a flowchart of a first embodiment of the stereo matching method for a pair of high-resolution stereo satellite panchromatic images of the present application;

[0070] Figure 2 This is a visualization effect diagram of the disparity estimation of dense buildings and occluded areas (clouds) involved in the embodiment of the present application;

[0071] Figure 3 This is a visualization effect diagram of the disparity estimation of the weak texture area (water surface) involved in the embodiment of the present application;

[0072] Figure 4This is a visualization effect diagram of the disparity estimation of dense buildings and shadow areas involved in the embodiment of the present application;

[0073] Figure 5 This is a visualization effect diagram of the disparity estimation of the geometric shape of the non-dense area involved in the embodiment of the present application;

[0074] Figure 6 A schematic diagram of the architecture of the hardware operating environment of the computer system involved in the embodiments of the present application;

[0075] The realization of the objectives, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0076] To better understand the above technical solutions, exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] First embodiment

[0078] Reference Figure 1 In this embodiment, a stereo matching method for a high-resolution stereo satellite panchromatic image pair is provided, which specifically includes the following steps:

[0079] S10, normalizes the left / right panchromatic image pairs collected by high-resolution stereo satellites, and uses a shared pyramid feature extraction layer to obtain multiple left / right feature maps including multi-scale pyramid features of different receptive field maps;

[0080] In this step, specifically, prepare the open source dataset WHU-Stereo, read the corresponding high-resolution satellite stereo image pairs according to the index, the data is type and convert them into PyTorch tensors before performing necessary preprocessing.

[0081] In some optional implementations, the normalization process of the left / right panchromatic image pair acquired by the high-resolution stereo satellite is specifically as follows:

[0082]

[0083] in, image p is the value of each pixel of the input image, is the average value of all pixel values ​​of the left / right panchromatic image pair, is the standard deviation of all pixel values ​​in the left / right panchromatic image pair;

[0084] Furthermore, the normalized left / right full-color image pair is input into a shared feature extraction network, so that the shared feature extraction network extracts left / right feature maps of different scales:

[0085]

[0086]

[0087] Where, Indicates Left feature map at resolution, Indicates Right feature map at resolution, Indicates the horizontal and vertical coordinates of the corresponding pixel point, H is the height of the input image, W is the width, is the characteristic dimension.

[0088] It should be noted that these feature maps are used to construct a multi-scale cost volume. Features are then further extracted and concatenated with the original feature maps along the channel dimension, increasing the feature dimensionality and information content. Finally, a descriptor network and convolution operations are used to generate feature descriptors for the concatenated feature maps. These descriptors are then used in the subsequent full-pair correlation construction process.

[0089] Further, calculating the correlation between each of the left / right feature maps, and constructing a multi-scale geometric cost volume according to each of the correlations;

[0090] In this step, for each scale, a grouped correlation volume is constructed that combines the features Divided along the channel dimension The correlation is calculated group by group using element-by-element subtraction.

[0091] In some optional implementations, the dependencies are expressed in terms of cost volumes:

[0092]

[0093] in That is, the correlation cost body represents, is the index of the group, is the disparity index value, Indicates the horizontal and vertical coordinates of the corresponding pixel point, is the number of feature channels, is the number of groups.

[0094] Then, the correlation of each scale is aggregated to obtain the geometric cost volume at multiple scales:

[0095] Starting from a lower-scale geometric coding cost volume, a multi-scale geometric cost volume encoding rich information is generated by upsampling and fusing it with a higher-scale cost volume ( ).

[0096] S30, calculating the full pairwise correlation of each of the left / right feature maps, and constructing a combined cost volume by combining the full pairwise correlation with the multi-scale geometric cost volume, and inputting the combined cost volume into a convolutional gated recurrent unit to iteratively update the disparity, wherein a loss function with gradient consistency constraint is used to guide the training process of the stereo matching network during the iterative process;

[0097] In this step, the above cost body is applied , get the probability distribution of each pixel at different disparity values, perform weighted summation of each disparity probability value within the defined disparity range (i.e. disparity regression), and finally get the most likely disparity value, i.e. the rough disparity value :

[0098]

[0099] Where, is the initial coarse disparity value, is the disparity index value, is the preset minimum parallax value, is the preset maximum parallax value, is a function of probability distribution;

[0100] Calculate the pairwise correlation between the corresponding left and right features. This can be achieved by calculating the similarities (or distances) between all pixels in the left image and all possible candidate pixels in the right image, constructing a complete correlation matrix, and obtaining the pairwise correlation. In some optional implementations, this can be achieved using methods such as the sum of absolute differences (SAD) and sum of squared differences (SSD), which are not described in detail in this example.

[0101] Then, the full pairwise correlation is structured and stored as a correlation volume, a correlation volume and a multi-scale fused geometric cost volume ( ) forms a synthetic geometric coding cost volume ;

[0102] Iteratively update the disparity using multi-layer convolutional gated recurrent units:

[0103]

[0104] Where, It is The disparity value after iterations, It is The disparity value after iterations, To decode the hidden state through two convolutional layers The obtained residual time difference.

[0105] In this step, from Initially, a multi-layer convolutional gated recurrent unit is used to iteratively update the disparity. The hidden state of each layer of gated unit is initialized with multi-scale context features. For each iteration, the current disparity is used and the combined geometric cost volume of the corresponding scale as a hint.

[0106] In some optional implementations, first the unit input is as follows:

[0107]

[0108] in, That is, using convolutional layer encoding, for The current disparity after iterations, For Synthetic geometric cost volumes of the same scale. The “:” symbol indicates the concatenation of feature dimensions. The input obtained. ,in is the update gate parameter, which determines the proportion of candidate hidden state information retained in the current time step. yes Activation function, the output range is between (0, 1). is the convolution operation, is the hidden state at the previous moment, is the weight of the update gate, is the bias of the update gate, provided by the context network, and used to initialize the update parameters.

[0109] Furthermore, the input is substituted into the update gate parameter function:

[0110]

[0111] Where, is the update gate parameter, yes Activation function, the output range is between (0, 1), is the convolution operation, is the hidden state at the previous moment, is the weight of the update gate, is the bias of the update gate, provided by the context network, and used to initialize the update parameters.

[0112] And, substituting the input into the function that resets the gate parameters:

[0113]

[0114] Where, is to reset the gate parameters, is the weight of the reset gate, is the bias of the reset gate;

[0115] Furthermore, the update gate parameters and the reset gate parameters are substituted into the candidate hidden state:

[0116]

[0117] Where, is a candidate hidden state, Is the hyperbolic tangent activation function, the output range is between (-1, 1). h is the weight of the candidate value, is the bias of the candidate value. " ” denotes the Hadamard product.

[0118] Furthermore, the candidate hidden state is substituted into the final hidden state function

[0119]

[0120] Where, is the final hidden state, representing the previous hidden state and candidate hidden states The weighted sum of Make updates;

[0121] Finally, a residual disparity is obtained by decoding through two convolutional layers

[0122]

[0123] Where, It is The disparity after the iteration is obtained by summing the disparity of the previous iteration and the update amount.

[0124] S40, after the iteration is completed, obtain the trained stereo matching network and the disparity estimation result of the high-resolution satellite panchromatic image pair.

[0125] In this step, steps S10-S30 can be understood as training operations for the deep learning network. After the iteration is completed, a trained stereo matching network is obtained. The high-resolution stereo satellite panchromatic image pair that needs to be stereo matched is input into the network, and the disparity estimation result of the high-resolution satellite panchromatic image pair output by the network is obtained.

[0126] In the technical solution provided in this embodiment, the comprehensive application of combined geometric coding convolution and gated iterative optimization mechanism significantly enhances the model's ability to capture complex geometric shapes, especially in the precise matching of ground objects in remote sensing images. The application of multi-scale cost volumes enables the model to effectively match ground objects at different scales, while effectively handling occlusion and shadow areas, thereby improving the accuracy and robustness of matching. In addition, the gradient-constrained loss function further optimizes the disparity estimation process, ensuring clear distinction of the disparity map in edge areas and smooth transition of weak texture areas, thereby improving the overall quality of the disparity map.

[0127] Second embodiment

[0128] In this embodiment, the effect achieved by the method proposed in the first embodiment is verified and explained as follows:

[0129] 1. Experimental data:

[0130] WHU-Stereo dataset: This dataset is produced based on the 2020 China GF-7 satellite image data. It contains 1757 (1024 × 1024 pixels) panchromatic epipolar-rectified stereo image pairs of Chinese cities and ground truth disparity maps produced from airborne LiDAR point clouds.

[0131] 2. Experimental setup:

[0132] The example network of this invention is implemented in the Pytorch 2.4.1 framework, using AdamW as the model training optimizer. The initial learning rate is 0.0002, and the cosine annealing strategy in OneCycleLR is used to dynamically adjust the learning rate. The total number of epochs is set to 300. 1220 pairs of images from four cities are used for training, 122 pairs are used for validation, and the remaining 415 pairs, covering six cities, are used for testing. The disparity range of the network input is set to [-128, 64] based on the disparity distribution of the dataset. The absolute error (EPE) and the proportion of mismatched pixels (D1) are used as quantitative evaluation metrics.

[0133] EPE measures the average error between the predicted disparity map and the true disparity map. Specifically, it is the average of the absolute values ​​of the differences between the predicted disparity values ​​and the true disparity values. It is calculated as follows:

[0134]

[0135] in, is the number of valid points of parallax within the preset range, is the predicted disparity value of the corresponding point, is the true disparity value of the corresponding point.

[0136] D1 represents the proportion of pixels whose predicted disparity value differs from the actual disparity value within a certain threshold (the threshold used in the experiment is 3) among all pixels. It is calculated as follows:

[0137]

[0138] in Indicates the number of pixels with an absolute disparity error greater than 3.

[0139] For comparison, this embodiment also selects the classic Google Stereo Matching Network (StereoNet) and Pyramid Stereo Matching Network (PSM-Net) as comparison models.

[0140] The method described in this embodiment is abbreviated as (PMGC-Net(ours)).

[0141] 3. Experimental results

[0142] Through the above steps, the WHU-Stereo dataset is experimentally verified. The predicted experimental results are shown in Table 1. The original image of the dataset, disparity labels and predicted disparity maps are visualized as shown in Table 1. Figure 2-5 The disparity estimation visualization effect diagrams shown are for dense buildings and occluded areas (clouds), the disparity estimation visualization effect diagrams for weak texture areas (water surface), the disparity estimation visualization effect diagrams for dense buildings and shadow areas, and the disparity estimation visualization effect diagrams for the geometric shapes of non-dense areas.

[0143] It should be noted that Figure 2-5 From left to right are reference image, ground truth label, StereoNet, PSM-Net and PMGC-Net (ours).

[0144] The results of the three methods are shown in Table 1 below:

[0145] Table 1. Prediction results of the WHU-Stereo dataset

[0146]

[0147] As shown in Table 1, the optimal values ​​for each metric are highlighted in bold (smaller absolute error and mismatch ratio values ​​indicate better results). The proposed method improves disparity estimation accuracy on the dataset to varying degrees, with absolute error estimation accuracy increasing by 23% and mismatch ratio decreasing by 37%. This demonstrates that the proposed method, compared to existing techniques, is able to better model the complex correlations in high-resolution satellite imagery, thereby improving matching accuracy. Figure 2-5 The visualization shows the comparison between the prediction graphs of our method and other methods, which more intuitively reflects the advantages of our method:

[0148] The proposed PMGC-Net method reduces edge blurring and adhesion in densely built-up areas and achieves superior prediction results in foggy built-up areas. It also achieves lower noise compared to other methods in weakly textured water areas. Other methods produce erroneous disparity predictions in shadowed areas. Finally, the proposed method more accurately predicts building geometry in areas without densely built-up areas. This demonstrates that the proposed method is more accurate than existing techniques in real-world scene observations.

[0149] In addition, as an implementation solution, this embodiment further provides an application of the stereo matching method of a high-resolution stereo satellite panchromatic image pair as described in any of the previous embodiments in disparity estimation.

[0150] In addition, as an implementation solution, Figure 6 This is a schematic diagram of the architecture of the hardware operating environment of the computer system involved in the embodiment of the present application.

[0151] like Figure 6 As shown, the computer system may include: a processor 1001, such as a CPU, a memory 1005, a user interface 1003, a network interface 1004, and a communication bus 1002. The communication bus 1002 is used to implement communication between these components. The user interface 1003 may include a display and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed RAM memory or a non-volatile memory, such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0152] Those skilled in the art will understand that Figure 6 The computer system architecture shown in the figure does not constitute a limitation of the computer system, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0153] like Figure 6 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module and a computer program. Among them, the operating system is a program that manages and controls the hardware and software resources of the computer system, the operation of the computer program and other software or programs.

[0154] exist Figure 6In the computer system shown, the user interface 1003 is mainly used to connect to the terminal and communicate data with the terminal; the network interface 1004 is mainly used to communicate data with the background server; the processor 1001 can be used to call the computer program stored in the memory 1005.

[0155] In this embodiment, the computer system includes: a memory 1005, a processor 1001, and a computer program stored in the memory and executable on the processor, wherein:

[0156] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0157] S10, normalizes the left / right panchromatic image pairs collected by high-resolution stereo satellites, and uses a shared pyramid feature extraction layer to obtain multiple left / right feature maps including multi-scale pyramid features of different receptive field maps;

[0158] S20, calculating the correlation between the left and right feature maps, and constructing a multi-scale geometric cost volume according to the correlations;

[0159] S30, calculating the full pairwise correlation of each of the left / right feature maps, and constructing a combined cost volume by combining the full pairwise correlation with the multi-scale geometric cost volume, and inputting the combined cost volume into a convolutional gated recurrent unit to iteratively update the disparity, wherein a loss function with gradient consistency constraint is used to guide the training process of the stereo matching network during the iterative process;

[0160] S40, after the iteration is completed, obtain the trained stereo matching network and the disparity estimation result of the high-resolution satellite panchromatic image pair.

[0161] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0162] S11, normalize the left / right panchromatic image pair:

[0163]

[0164] in, image p is the value of each pixel of the input image, is the average value of all pixel values ​​of the left / right panchromatic image pair, is the standard deviation of all pixel values ​​in the left / right panchromatic image pair;

[0165] S12: Input the normalized left / right full-color image pair into a shared feature extraction network, so that the shared feature extraction network extracts left / right feature maps of different scales:

[0166]

[0167]

[0168] Where, Indicates Left feature map at resolution, Indicates Right feature map at resolution, Indicates the horizontal and vertical coordinates of the corresponding pixel point, H is the height of the input image, W is the width, is the characteristic dimension.

[0169] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0170] S21, group the left / right feature maps along the channel dimension and calculate the correlation group by group using element-by-element subtraction:

[0171]

[0172] Where, For correlation, is the index of the group, is the disparity index value, Indicates the horizontal and vertical coordinates of the corresponding pixel point, is the number of feature channels, is the number of groups;

[0173] S22, the calculated correlation of each scale is cost-aggregated with the left / right feature map of the corresponding scale to obtain a multi-scale geometric cost volume. When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0174] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0175] S31, calculate the full pairwise correlation of the left / right feature maps of the corresponding scale;

[0176] S32, determine the initial coarse disparity value of the multi-scale geometric cost volume:

[0177]

[0178] Where, is the initial coarse disparity value, is the disparity index value, is the preset minimum parallax value, is the preset maximum parallax value, is a function of probability distribution;

[0179] S33, storing the full pairwise correlation structure as a correlation volume, and synthesizing the correlation volume and the multi-scale geometric cost volume into a geometric coding cost volume ;

[0180] S34, iteratively update the disparity using multi-layer convolutional gated recurrent units:

[0181]

[0182] Where, It is The disparity value after iterations, It is The disparity value after iterations, To decode the hidden state through two convolutional layers The obtained residual time difference.

[0183] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0184] S341, the geometric coding cost volume Input , input to the multi-layer convolutional gated recurrent unit:

[0185]

[0186] Where, is the input of the multi-layer convolutional gated recurrent unit, Encode the convolutional layer, is the current disparity after k iterations, For Synthetic geometric cost volume of the same scale. The “:” symbol indicates the concatenation of feature dimensions.

[0187] S342, substitute the input into the update gate parameter function:

[0188]

[0189] Where, is the update gate parameter, yes Activation function, the output range is between (0, 1), is the convolution operation, is the hidden state at the previous moment, is the weight of the update gate, is the bias of the update gate, provided by the context network, used to initialize the update parameters;

[0190] And, substituting the input into the function that resets the gate parameters:

[0191]

[0192] Where, is to reset the gate parameters, is the weight of the reset gate, is the bias of the reset gate;

[0193] S343, substituting the update gate parameter and the reset gate parameter into the candidate hidden state:

[0194]

[0195] Where, is a candidate hidden state, is the hyperbolic tangent activation function, the output range is between (-1, 1), W h is the weight of the candidate value, is the bias of the candidate value, " represents the Hadamard product, that is, the element-by-element product;

[0196] S344, substitute the candidate hidden state into the final hidden state function:

[0197]

[0198] Where, is the final hidden state, representing the previous hidden state and candidate hidden states The weighted sum of Make updates;

[0199] S345, decoding hidden states through two convolutional layers , and get the residual time difference .

[0200] When the processor 1001 calls the computer program stored in the memory 1005, it performs the following operations:

[0201] S31, introduce the disparity estimation loss function and gradient consistency loss function to adjust the disparity value:

[0202]

[0203] Where, is the disparity estimation loss, is the loss of the coarse disparity map," " is the sum of n items, " ” is the L1 norm, that is, the absolute value loss, is the current disparity value, is the true disparity value, is the attenuation factor;

[0204]

[0205] Where, is the gradient consistency loss, " means that for each coordinate The sum of pixels, represents the gradient value of the disparity map, is the predicted final disparity;

[0206] S32, determining a loss function for gradient consistency constraint according to the disparity estimation loss function and the gradient consistency loss function:

[0207]

[0208] Where, is the loss function with gradient consistency constraint, , is the loss weight.

[0209] Furthermore, those skilled in the art will appreciate that all or part of the steps in the method of the above-described embodiment can be implemented by instructing the relevant hardware through a computer program. The computer program includes program instructions, which can be stored in a storage medium that is a computer-readable storage medium. The program instructions are executed by at least one processor in a computer system to implement the steps in the process of the above-described method embodiment.

[0210] Therefore, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the various steps of the stereo matching method of the high-resolution stereo satellite panchromatic image pair as described in the above embodiment.

[0211] The computer-readable storage medium may be any computer-readable storage medium that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a magnetic disk, or an optical disk.

[0212] It should be noted that since the storage medium provided in the embodiments of this application is the storage medium used to implement the method of the embodiments of this application, based on the method described in the embodiments of this application, those skilled in the art will be able to understand the specific structure and deformation of the storage medium, and therefore will not be described in detail here. All storage media used in the method of the embodiments of this application fall within the scope of protection to be provided by this application.

[0213] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0214] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0215] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0216] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0217] It should be noted that in the claims, any reference signs placed between parentheses shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claim. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The present application may be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by one and the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names.

[0218] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0219] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A stereo matching method for high-resolution stereo satellite panchromatic image pairs, characterized in that: The method comprises the following steps: S10, normalizes the left / right panchromatic image pairs collected by high-resolution stereo satellites, and uses a shared pyramid feature extraction layer to obtain multiple left / right feature maps including multi-scale pyramid features of different receptive field maps; S20, calculating the correlation between the left and right feature maps, and constructing a multi-scale geometric cost volume according to the correlations; S30, calculating the full pairwise correlation of each of the left / right feature maps, and constructing a combined cost volume by combining the full pairwise correlation with the multi-scale geometric cost volume, and inputting the combined cost volume into a convolutional gated recurrent unit to iteratively update the disparity, wherein a loss function with gradient consistency constraint is used to guide the training process of the stereo matching network during the iterative process; S40, after the iteration is completed, obtain the trained stereo matching network and the disparity estimation result of the high-resolution satellite panchromatic image pair; The S30 specifically includes: S31, calculate the full pairwise correlation of the left / right feature maps of the corresponding scale; S32, determine the initial coarse disparity value of the multi-scale geometric cost volume: ； Where, is the initial coarse disparity value, is the disparity index value, is the preset minimum parallax value, is the preset maximum parallax value, is a function of probability distribution; S33, storing the full pairwise correlation structure as a correlation volume, and synthesizing the correlation volume and the multi-scale geometric cost volume into a geometric coding cost volume ; S34, iteratively update the disparity using multi-layer convolutional gated recurrent units: ； Where, It is The disparity value after iterations, It is The disparity value after iterations, To decode the hidden state through two convolutional layers The obtained residual time difference; In the S34, the residual time difference The calculation process specifically includes: S341, the geometric coding cost volume Input , input to the multi-layer convolutional gated recurrent unit: ； Where, is the input of the multi-layer convolutional gated recurrent unit, Encode the convolutional layer, is the current disparity after k iterations, For Synthetic geometric cost volume of the same scale, the ":" symbol represents the concatenation of feature dimensions; S342, substitute the input into the update gate parameter function: ； Where, is the update gate parameter, yes Activation function, the output range is between (0, 1), is the convolution operation, is the hidden state at the previous moment, is the weight of the update gate, is the bias of the update gate, provided by the context network, used to initialize the update parameters; And, substituting the input into the function that resets the gate parameters: ； Where, is to reset the gate parameters, is the weight of the reset gate, is the bias of the reset gate; S343, substituting the update gate parameter and the reset gate parameter into the candidate hidden state: ； Where, is a candidate hidden state, is the hyperbolic tangent activation function, the output range is between (-1, 1), W h is the weight of the candidate value, is the bias of the candidate value, " represents the Hadamard product, that is, the element-by-element product; S344, substitute the candidate hidden state into the final hidden state function: ； Where, is the final hidden state, representing the previous hidden state and candidate hidden states The weighted sum of Make updates; S345, decoding hidden states through two convolutional layers , and get the residual time difference .

2. The method according to claim 1, wherein The S10 specifically includes: S11, normalize the left / right panchromatic image pair: ； in, image p is the value of each pixel of the input image, is the average value of all pixel values ​​of the left / right panchromatic image pair, is the standard deviation of all pixel values ​​in the left / right panchromatic image pair; S12: Input the normalized left / right full-color image pair into a shared feature extraction network, so that the shared feature extraction network extracts left / right feature maps of different scales: ； ； Where, Indicates The left feature map at resolution, Indicates Right feature map at resolution, Indicates the horizontal and vertical coordinates of the corresponding pixel point, H is the height of the input image, W is the width, is the characteristic dimension.

3. The method according to claim 1, wherein The S20 specifically includes: S21, group the left / right feature maps along the channel dimension and calculate the correlation group by group using element-by-element subtraction: ； Where, For correlation, is the index of the group, is the disparity index value, Indicates the horizontal and vertical coordinates of the corresponding pixel point, is the number of feature channels, is the number of groups; S22: Cost aggregation is performed on the calculated correlations of each scale and the left / right feature maps of the corresponding scale to obtain a multi-scale geometric cost volume.

4. The method according to claim 3, wherein Said S22 specifically includes: S221, selecting the current minimum scale correlation; S222, generates a multi-scale geometric cost volume by upsampling and fusing higher-scale correlations.

5. The method according to claim 1, wherein In S30, the training process of the stereo matching network is guided by a loss function constrained by gradient consistency, which specifically includes: S31, introduce the disparity estimation loss function and gradient consistency loss function to adjust the disparity value: ； Where, is the disparity estimation loss, is the loss of the coarse disparity map," " is the sum of n terms, " ” is the L1 norm, that is, the absolute value loss, is the current disparity value, is the true disparity value, is the attenuation factor; ； Where, is the gradient consistency loss, " means that for each coordinate The sum of pixels, represents the gradient value of the disparity map, is the predicted final disparity; S32, determining a loss function for gradient consistency constraint according to the disparity estimation loss function and the gradient consistency loss function: ； Where, is the loss function with gradient consistency constraint, , is the loss weight.

6. Application of the stereo matching method of the high-resolution stereo satellite panchromatic image pair according to any one of claims 1 to 5 in disparity estimation.

7. A computer system, characterized in that: The computer system includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps of the stereo matching method for a high-resolution stereo satellite panchromatic image pair are implemented as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the stereo matching method for a high-resolution stereo satellite panchromatic image pair according to any one of claims 1 to 5.

Images