Stereo matching method and system based on feature enhancement and multi-scale cost fusion

Abstract

The invention discloses a stereo matching method and system based on feature enhancement and multi-scale cost fusion, and relates to the technical field of optical satellite remote sensing image information extraction, and the method comprises the steps: obtaining a pair of stereo images of an optical satellite with the same size after epipolar line correction, carrying out the feature extraction, obtaining the multi-scale features of the two images, and carrying out the feature enhancement; carrying out cost body construction on the multi-scale features after feature enhancement according to a set parallax range, and then carrying out fusion and aggregation processing to obtain a cost body under the highest resolution; regularizing the cost body with the highest resolution to obtain an initial disparity map; and inputting the initial disparity map, the cost body under the highest resolution and the context information of the left view of the image into the GRU for iterative updating to obtain a refined disparity map, and performing up-sampling on the updated disparity map to a full size according to the up-sampling mask. The invention provides an efficient and reliable stereo matching new technology for the field of photogrammetry and remote sensing.

Description

Technical Field

[0001] The present invention relates to the technical field of optical satellite remote sensing image information extraction, and in particular to a stereo matching method and system based on feature enhancement and multi-scale cost fusion. Background Art

[0002] Remote sensing image stereo matching technology is widely used for generating high-precision DSM / DEMs and 3D reconstructions. However, existing methods still suffer from limited matching accuracy, low computational efficiency, insufficient adaptability to multi-source matching, and difficulty coping with complex terrain. Matching accuracy is unstable in low-texture areas and occluded scenes, and mismatches and noise are more serious. Furthermore, traditional global optimization methods have high computational complexity. Furthermore, issues such as building occlusion and image distortion in complex terrain and urban environments can also affect matching stability.

[0003] The goal of stereo matching is to recover the three-dimensional structure of a scene from multi-view images (typically binocular or multi-view remote sensing images). This is primarily accomplished by calculating the disparity or depth between image pairs to obtain surface elevation information. However, previous research has generally failed to effectively address issues such as occlusion and disparity discontinuities in the images. These issues have resulted in suboptimal disparity prediction accuracy in these areas. Summary of the Invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a stereo matching technology based on feature enhancement and multi-scale cost fusion for the disparity estimation task of high-resolution optical satellite stereo images.

[0005] To achieve the above technical objectives, the present application provides a stereo matching method based on feature enhancement and multi-scale cost fusion for disparity estimation of optical satellite stereo images, comprising the following steps: Obtain a pair of epipolar-corrected optical satellite stereo images of the same size, perform feature extraction, obtain multi-scale features of the two images, and perform feature enhancement; The multi-scale features after feature enhancement are used to construct a cost volume according to the set disparity range, and then fused and aggregated to obtain the cost volume at the highest resolution; Regularize the highest resolution cost volume to obtain the initial disparity map; The initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image are input into the GRU for iterative update to obtain the refined disparity map, and the updated disparity map is upsampled to the full size according to the upsampling mask.

[0006] Preferably, in the process of performing feature extraction, multi-scale features are extracted from a pair of images through convolution operations and downsampling operations.

[0007] Preferably, during the feature enhancement process, the features at each scale are enhanced using a Transformer method.

[0008] Preferably, when performing feature enhancement, the features of the two images are spliced ​​at the same scale; the spliced ​​features at the same scale are associated using the cross attention of the Transformer to obtain enhanced features; the spliced ​​enhanced features are separated to obtain enhanced features of the two images respectively.

[0009] Preferably, in the process of obtaining the cost volume at the highest resolution, the enhanced features of the same scale of the two images are subtracted along the set disparity range to construct a multi-scale cost volume; the cost volume of each scale is aggregated using 3D convolution; the aggregated multi-scale cost volume is upsampled or downsampled to the same scale and fused according to the channel attention and spatial attention methods; the fused cost volume is aggregated again to obtain the cost volume at the highest resolution.

[0010] Preferably, in the process of obtaining the initial disparity map, the highest resolution cost body is regularized using 3D convolution to output the number of channels of the cost body as 1; the disparity probability distribution of the regularized cost body is calculated using the Softmax function; and the set disparity range is multiplied by the probability distribution to obtain the initial disparity map.

[0011] Preferably, in the process of obtaining the refined disparity map, the disparity residual and the hidden state of the context information are updated by GRU, and the disparity residual and the initial disparity image are added; the update is continued until the set number of updates is reached; and the updated disparity map and upsampling mask are output.

[0012] The present invention discloses a stereo matching system based on feature enhancement and multi-scale cost fusion for disparity estimation of optical satellite stereo images, comprising: The feature extraction and enhancement module is used to obtain a pair of epipolar-corrected optical satellite stereo images of the same size, perform feature extraction, obtain multi-scale features of the two images, and perform feature enhancement; The fusion and aggregation module is used to construct a cost volume based on the multi-scale features after feature enhancement according to the set disparity range, and then fuse and aggregate them to obtain the cost volume at the highest resolution; Regularization module, used to regularize the highest resolution cost volume to obtain the initial disparity map; The stereo matching module is used to input the initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image into the GRU for iterative update to obtain the refined disparity map, and then upsample the updated disparity map to the full size according to the upsampling mask.

[0013] The present invention discloses the following technical effects: The present invention constructs cost volumes at multiple scales, and then undergoes fusion and aggregation processing at multiple scales, thereby improving the information dissemination capability in occluded areas, greatly improving the disparity prediction capability of optical satellite stereo images in occluded areas, and effectively improving the accuracy of disparity prediction. The stereo matching model formed by the Transformer feature enhancement and multi-scale cost fusion method in the present invention constructs cost volumes of multiple scales and fuses and aggregates them at multiple scales, which effectively addresses the complex occlusion problem of real optical satellite stereo images and provides an efficient and reliable solution for the field of photogrammetry and remote sensing stereo matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0015] Figure 1 This is a flowchart of a stereo matching method based on Transformer feature enhancement and multi-scale cost fusion in an embodiment of the present invention.

[0016] Figure 2 It is a specific schematic module diagram of a stereo matching method based on Transformer feature enhancement and multi-scale cost fusion in an embodiment of the present invention.

[0017] Figure 3 Schematic diagram of a Transformer feature enhancement module of a stereo matching method based on Transformer feature enhancement and multi-scale cost fusion according to an embodiment of the present invention.

[0018] Figure 4 It is an overall schematic diagram of the cost fusion aggregation module of the stereo matching method based on Transformer feature enhancement and multi-scale cost fusion in an embodiment of the present invention.

[0019] Figure 5 It is a detailed schematic diagram of the cost fusion module of the stereo matching method based on Transformer feature enhancement and multi-scale cost fusion in an embodiment of the present invention.

[0020] Figure 6 It is a detailed schematic diagram of the cost aggregation module of the stereo matching method based on Transformer feature enhancement and multi-scale cost fusion in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0022] Example 1: See Figure 1 The present invention provides a stereo matching method based on Transformer feature enhancement and multi-scale cost fusion, the specific steps are as follows: S1: Acquire a pair of epipolar-corrected optical satellite stereo images of the same size; S2: Feature extraction is performed on the two images to obtain multi-scale features of the two images; S3: Use Transformer to enhance the features of the same scale in the two images to obtain enhanced multi-scale features; S4: Construct a multi-scale cost volume for the enhanced multi-scale features, fuse and aggregate the multi-scale cost volumes to obtain the cost volume at the highest resolution; S5: Regularize the highest resolution cost volume to obtain the initial disparity map; S6: Input the initial disparity map, the highest resolution cost volume, and the context information of the left view of the image into the GRU and iteratively update it to obtain the refined disparity map; S7: Upsample the refined disparity map to the original image size to obtain a full-size disparity map.

[0023] The specific steps of step S2 are: S21 extracts features of multiple scales from a pair of images through convolution and downsampling operations; In step S3, the specific steps are: S31: Concatenate the features of the two images at the same scale.

[0024] S32: Use the cross attention of Transformer to associate the splicing features at the same scale to obtain enhanced features.

[0025] S33: Separate the spliced ​​enhanced features to obtain enhanced features of the two images respectively.

[0026] In step S4, the specific steps are: S41: Subtract the enhanced features of the same scale of the two images along the set disparity range to construct a multi-scale cost volume.

[0027] S42: Aggregate the cost volume of each scale using 3D convolution.

[0028] S43: Upsample or downsample the aggregated multi-scale cost volume to the same scale and fuse them according to the channel attention and spatial attention methods.

[0029] S44: Aggregate the fused cost body again.

[0030] S45: Get the highest resolution cost volume.

[0031] In step S5, the specific steps are: S51: Regularize the highest resolution cost volume using 3D convolution and output the number of channels of the cost volume as 1.

[0032] S52: Use the Softmax function to calculate the disparity probability distribution of the regularized cost volume.

[0033] S53: Multiplying the set disparity range by the probability distribution to obtain an initial disparity map.

[0034] In step S6, the specific steps are: S61: Input the initial disparity map, cost volume and context information extracted from the left image into the GRU.

[0035] S62: GRU updates the hidden state of the disparity residual and context information.

[0036] S63: Add the disparity residual and the initial disparity image.

[0037] S64: Continue updating until the set number of updates is reached.

[0038] S65: Output the updated disparity map and upsampling mask.

[0039] S71: Upsampling the updated disparity map to the full size according to the upsampling mask.

[0040] In this embodiment 1, multi-scale enhancement features of a stereo image pair are extracted to construct a multi-scale cost volume, and then the multi-scale cost volume is fused and aggregated, and an initial disparity map is obtained after regularization. The disparity map is iteratively refined and upsampled to the full size to achieve the disparity map generation function for the stereo image.

[0041] Example 2: The steps of this example are the same as those of Example 1, except that each step is applied to a specific example, specifically including the following steps: S1: Acquire a pair of epipolar-corrected optical satellite stereo images of the same size H×W , ; S21: Extracting multi-scale features from a pair of stereo images through convolution operations: ; , Where, 、 、 are the number of channels, height, and width of the image, respectively.

[0042] S31: Concatenate the features of the two images at the same scale: ; S32: Use the Transformer’s cross attention to associate the spliced ​​features at the same scale and then separate them to obtain enhanced multi-scale features: .

[0043] S41: Subtract the enhanced features of the same scale in the two images along the set parallax range: formula, is the parallax range, Represent pixel row and column coordinates respectively, represents the difference between eigenvectors.

[0044] S42: Get the constructed multi-scale cost volume: formula, Indicates the parallax range; S43: Fuse and aggregate the multi-scale cost volumes to obtain the cost volume at the highest resolution: ; S51: Regularize the highest resolution cost volume to obtain: .

[0045] S52: Use softmax to calculate the probability distribution and then multiply it by the disparity range to get the initial disparity map: ; S61: Initial disparity map , the highest resolution cost body , contextual information of the left view of the image , Input to GRU and iteratively update S62: The calculation process of a single GRU is: S63: Initial disparity is updated: is the disparity residual for each update.

[0046] S71: Upsample the updated disparity map to full size according to the upsampling mask: The parallax of the last update.

[0047] In this second embodiment, multi-scale features are extracted and enhanced to construct a multi-scale cost volume. This is then fused and aggregated from coarse to fine to obtain the highest-resolution cost volume, improving information dissemination and further enhancing prediction capabilities in occluded areas. The disparity residual is then updated using a GRU to obtain a more refined disparity map.

[0048] The operating environment of this embodiment 2 is based on the open source machine learning library Pytorch and is implemented under the Nvidia RTXA6000 graphics processor, which can provide efficient and stable computing support for the operation of optical satellite stereo image stereo matching.

[0049] Example 3: This example 3 implements the principles of the above-mentioned method embodiments by constructing a stereo matching module based on Transformer feature enhancement and multi-scale cost fusion, including: a multi-scale feature extraction module, a feature enhancement module, a cost volume fusion aggregation module, and a disparity update module. The multi-scale feature extraction module is used to extract multi-scale features of stereo images; the feature enhancement module is used to enhance multi-scale features and improve feature discrimination; the cost volume fusion module is used to fuse information from cost volumes of different scales; and the cost volume aggregation module is used to correct erroneous information in the cost volumes and improve information consistency. The disparity update module is used to refine the initially predicted disparity information to achieve more accurate disparity estimation.

[0050] The feature extraction module is used to extract multi-scale features of stereo image pairs. It uses convolutional neural networks to perform feature extraction and downsampling operations to obtain features at multiple scales.

[0051] The feature enhancement module is used to enhance the relevance of features and improve the recognition of features. Figure 3As shown in Figure 2, the feature enhancement module is designed based on Transformer, which achieves the purpose of feature enhancement by cross-focusing on the features of the same scale of two images.

[0052] like Figure 4 As shown in Figure 1, the overall schematic diagram of the cost volume fusion aggregation module is shown in Figure 2. Figure 5 The cost volume fusion module details shown in Figure 2 integrates cost information of different scales through channel attention and spatial attention. Figure 6 The cost volume aggregation module details shown in the figure. The cost aggregation module aggregates the information in the cost volume by serializing 3D convolutions, effectively maintaining the smoothness and effectiveness of the cost volume information.

[0053] The disparity update module is based on the GRU design. The GRU updates the hidden state of the context information through a gating method, and updates the disparity by indexing from the cost volume based on the context information and the initial disparity.

[0054] Example 4: This example extracts and enhances features at multiple scales to construct a multi-scale cost volume. This multi-scale cost volume is then fused and aggregated from coarse to fine resolution to obtain the highest-resolution cost volume. This multi-scale fusion method improves information dissemination and further enhances prediction capabilities in occluded areas. A GRU is then used to update the disparity residual to obtain a more refined disparity prediction result.

[0055] This Example 4 uses stereo imagery from the Gaofen-7 satellite for training and testing. During the training phase, the input images are randomly cropped to 640×960, and during the testing phase, the original images are used. End-to-end training is performed using the AdamW optimizer. For all datasets, a single-cycle learning rate is used for training. The number of Transformer enhancement blocks is set to 6, and the number of disparity refinement iterations is set to 16.

[0056] The present invention extracts multi-scale features of a stereo image; enhances the multiple scale features using a Transformer to obtain multi-scale enhanced features; constructs a multi-scale cost volume based on the enhanced features, aggregates the multi-scale cost volumes, and fuses them sequentially from the lowest resolution to the highest resolution after aggregation to output a cost volume with the highest resolution; inputs the output cost volume and disparity map into a GRU for update to obtain a refined disparity map; and upsamples the refined disparity map to the full size, thereby achieving high-quality disparity map generation for stereo images.

[0057] The present invention performs feature extraction and cost volume construction at multiple scales, improving the accuracy of disparity prediction. The end-to-end stereo matching network constructed based on deep learning methods effectively addresses the disparity estimation problem of stereo matching of real optical satellite stereo images.

[0058] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowcharts and / or block diagrams, as well as combinations of processes and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts and / or block diagrams. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0059] In the description of the present invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of the technical features indicated. Therefore, a feature specified as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the present invention, "plurality" means two or more, unless otherwise specifically defined.

[0060] Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if such changes and modifications fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. A stereo matching method based on feature enhancement and multi-scale cost fusion, characterized in that: Disparity estimation for optical satellite stereo images includes the following steps: Obtain a pair of epipolar-corrected optical satellite stereo images of the same size, perform feature extraction, obtain multi-scale features of the two images, and perform feature enhancement; The multi-scale features after feature enhancement are used to construct a cost volume according to the set disparity range, and then fused and aggregated to obtain the cost volume at the highest resolution; Regularize the highest resolution cost volume to obtain the initial disparity map; The initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image are input into the GRU for iterative update to obtain the refined disparity map, and the updated disparity map is upsampled to the full size according to the upsampling mask.

2. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 1, characterized in that: In the process of performing feature extraction, the multi-scale features are extracted from a pair of images through convolution operations and downsampling operations.

3. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 2, characterized in that: During the feature enhancement process, the features of each scale are enhanced using the Transformer method.

4. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 3, characterized in that: When performing feature enhancement, the features of the two images are spliced ​​at the same scale; the cross-attention of Transformer is used to associate the spliced ​​features at the same scale to obtain the enhanced features; the spliced ​​enhanced features are separated to obtain the enhanced features of the two images respectively.

5. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 4, characterized in that: In the process of obtaining the cost volume at the highest resolution, the enhanced features of the same scale of the two images are subtracted along the set disparity range to construct a multi-scale cost volume; the cost volume of each scale is aggregated using 3D convolution; the aggregated multi-scale cost volume is upsampled or downsampled to the same scale and fused using channel attention and spatial attention; the fused cost volume is aggregated again; Get the cost volume at the highest resolution.

6. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 5, characterized in that: In the process of obtaining the initial disparity map, the highest resolution cost volume is regularized using 3D convolution to output the number of channels of the cost volume as 1; the Softmax function is used to calculate the disparity probability distribution of the regularized cost volume; and the set disparity range is multiplied by the probability distribution to obtain the initial disparity map.

7. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 6, characterized in that: In the process of obtaining the refined disparity map, the GRU updates the hidden state of the disparity residual and context information, adds the disparity residual and the initial disparity image; continues updating until the set number of updates is reached; and outputs the updated disparity map and upsampling mask.

8. A stereo matching system based on feature enhancement and multi-scale cost fusion, characterized in that: Disparity estimation for optical satellite stereo images, including: The feature extraction and enhancement module is used to obtain a pair of epipolar-corrected optical satellite stereo images of the same size, perform feature extraction, obtain multi-scale features of the two images, and perform feature enhancement; The fusion and aggregation module is used to construct a cost volume based on the multi-scale features after feature enhancement according to the set disparity range, and then fuse and aggregate them to obtain the cost volume at the highest resolution; Regularization module, used to regularize the highest resolution cost volume to obtain the initial disparity map; The stereo matching module is used to input the initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image into the GRU for iterative update to obtain the refined disparity map, and then upsample the updated disparity map to the full size according to the upsampling mask.

Images