Stereo matching method and system based on feature enhancement and multi-scale cost fusion

Abstract

The application discloses a stereo matching method and system based on feature enhancement and multi-scale cost fusion, relates to the technical field of optical satellite remote sensing image information extraction, and comprises the following steps: acquiring a pair of optical satellite stereo images of the same size after epipolar rectification, performing feature extraction, acquiring multi-scale features of the two images and performing feature enhancement; performing fusion and aggregation processing on the multi-scale features after feature enhancement after cost volume construction according to a set disparity range to obtain a cost volume under the highest resolution; normalizing the cost volume under the highest resolution to obtain an initial disparity map; inputting the initial disparity map, the cost volume under the highest resolution and context information of a left view of an image into a GRU for iterative updating to obtain a refined disparity map, and up-sampling the updated disparity map to a full-size size according to an up-sampling mask. The application provides a new technology of efficient and reliable stereo matching for the fields of photogrammetry and remote sensing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of optical satellite remote sensing image information extraction, in particular to a stereo matching method and system based on feature enhancement and multi-scale cost fusion. BACKGROUND

[0002] The stereo matching technology of remote sensing images has a wide range of needs in generating high-precision DSM / DEM and 3D reconstruction. However, the existing methods still have problems such as limited matching accuracy, low computational efficiency, insufficient adaptability of multi-source matching, and difficulty in dealing with complex terrain. In low-texture areas and occluded scenes, the matching accuracy is unstable, and the problems of false matching and noise are more serious. At the same time, the traditional global optimization method has high computational complexity. In addition, the problems of building occlusion, image distortion, etc. in complex terrain and urban environment also affect the stability of matching.

[0003] The goal of stereo matching is to recover the three-dimensional structure of the scene from multi-view images (usually binocular or multi-view remote sensing images), mainly by calculating the disparity or depth between image pairs to obtain the elevation information of the ground surface. However, previous research has not achieved good results in dealing with occluded areas and discontinuous disparity in images in most cases. These situations make the disparity prediction accuracy in these areas not meet the expected requirements. SUMMARY

[0004] In order to solve the above problems, the purpose of the present application is to provide a stereo matching technology based on feature enhancement and multi-scale cost fusion for the disparity estimation task of high-resolution optical satellite stereo images.

[0005] In order to achieve the above technical purpose, the present application provides a stereo matching method based on feature enhancement and multi-scale cost fusion for disparity estimation of optical satellite stereo images, including the following steps:

[0006] A pair of optical satellite stereo images of the same size after epipolar rectification is obtained, feature extraction is performed, and multi-scale features of the two images are obtained and enhanced;

[0007] The multi-scale features after feature enhancement are fused and aggregated after cost volume construction according to the set disparity range to obtain the cost volume at the highest resolution;

[0008] The cost volume at the highest resolution is regularized to obtain an initial disparity map;

[0009] The initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image are input into the GRU for iterative updating to obtain a refined disparity map, and the updated disparity map is upsampled to full size according to the upsample mask.

[0010] Preferably, in the process of performing feature extraction, multi-scale features are extracted from the pair of images through convolution operation and down-sampling operation.

[0011] Preferably, in the process of performing feature enhancement, the features of each scale are enhanced using the Transformer method.

[0012] Preferably, in the process of performing feature enhancement, the features of the two images are spliced at the same scale; the spliced features at the same scale are associated using the cross-attention of the Transformer to obtain enhanced features; and the spliced enhanced features are separated to obtain enhanced features of the two images respectively.

[0013] Preferably, in the process of obtaining the cost volume at the highest resolution, the enhanced features of the same scale of the two images are subtracted along the set disparity range to construct a multi-scale cost volume; the cost volume of each scale is aggregated using 3D convolution; the aggregated multi-scale cost volume is up-sampled or down-sampled to the same scale and fused in the manner of channel attention and spatial attention; the fused cost volume is aggregated again to obtain the cost volume at the highest resolution.

[0014] Preferably, in the process of obtaining the initial disparity map, the cost volume at the highest resolution is regularized using 3D convolution to output the channel number of the cost volume to 1; the regularized cost volume is used to calculate the disparity probability distribution using the Softmax function; and the set disparity range is multiplied by the probability distribution to obtain the initial disparity map.

[0015] Preferably, in the process of obtaining the refined disparity map, the disparity residual and the hidden state of the context information are updated by GRU, and the disparity residual and the initial disparity image are added; the updating is continued until the set number of updates is reached; and the updated disparity map and the up-sampling mask are output.

[0016] The application discloses a stereo matching system based on feature enhancement and multi-scale cost fusion, which is used for disparity estimation of optical satellite stereo images and comprises:

[0017] The feature extraction and enhancement module is used for obtaining a pair of core line corrected stereo images of optical satellites of the same size, performing feature extraction, obtaining multi-scale features of the two images, and performing feature enhancement.

[0018] The fusion and aggregation module is used for performing fusion and aggregation processing on the multi-scale features after feature enhancement according to the set disparity range to obtain the cost volume at the highest resolution.

[0019] The regularization module is used for regularizing the cost volume at the highest resolution to obtain the initial disparity map.

[0020] The stereomatching module is used for inputting the initial disparity map, the cost volume under the highest resolution and the context information of the left view of the image into the GRU for iterative updating, obtaining the refined disparity map, and up-sampling the updated disparity map to the full size according to the up-sampling mask.

[0021] The present application discloses the following technical effects:

[0022] The present application constructs the cost volume under multiple scales, and then performs fusion and aggregation processing, thereby improving the information propagation capability in the occluded area, greatly improving the disparity prediction capability of the optical satellite stereo image in the occluded area, and effectively improving the accuracy of disparity prediction.

[0023] The stereomatching model formed based on the Transformer feature enhancement and the multi-scale cost fusion method constructs multiple scale cost volumes, performs fusion and aggregation under multiple scales, well deals with the complex occlusion problem of real optical satellite stereo images, and provides an efficient and reliable solution for the photogrammetry and remote sensing stereomatching field. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0025] Figure 1 is the flowchart of the stereomatching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiments of the present application.

[0026] Figure 2 is the specific schematic module diagram of the stereomatching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiments of the present application.

[0027] Figure 3 is the Transformer feature enhancement module schematic diagram of the stereomatching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiments of the present application.

[0028] Figure 4 is the overall schematic diagram of the cost fusion and aggregation module of the stereomatching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiments of the present application.

[0029] Figure 5is a cost aggregation module detail schematic diagram of the stereo matching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiment of the application.

[0030] Figure 6 is a cost aggregation module detail schematic diagram of the stereo matching method based on the Transformer feature enhancement and the multi-scale cost fusion of the embodiment of the application. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work belong to the scope of protection of the present application.

[0032] Embodiment 1: see Figure 1 The present application provides a stereo matching method based on Transformer feature enhancement and multi-scale cost fusion, and the specific steps are as follows:

[0033] S1: Obtain a pair of epipolar rectified stereo images of optical satellites of the same size;

[0034] S2: Feature extraction is performed on the two images to obtain multi-scale features of the two images;

[0035] S3: The Transformer is used to enhance the features of the same scale of the two images to obtain enhanced multi-scale features;

[0036] S4: A multi-scale cost volume is constructed for the enhanced multi-scale features, and the multi-scale cost volume is fused and aggregated to obtain a cost volume at the highest resolution;

[0037] S5: The cost volume at the highest resolution is regularized to obtain an initial disparity map;

[0038] S6: The initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image are input into the GRU for iterative updating to obtain a refined disparity map;

[0039] S7: The refined disparity map is up-sampled to the original image size to obtain a full-size disparity map.

[0040] The specific steps of step S2 are as follows:

[0041] S21: Extracting features of multiple scales from a pair of images through convolution operation and down-sampling operation.

[0042] The specific steps of step S3 are as follows:

[0043] S31: Splicing the features of the two images at the same scale.

[0044] S32: Using cross-attention of the Transformer to associate the spliced features at the same scale to obtain enhanced features.

[0045] S33: Separating the spliced enhanced features to obtain enhanced features of the two images respectively.

[0046] The specific steps of step S4 are as follows:

[0047] S41: Constructing a multi-scale cost volume by differencing the enhanced features of the two images at the same scale within a set disparity range.

[0048] S42: Using 3D convolution to aggregate the cost of each scale.

[0049] S43: Up-sampling or down-sampling the aggregated multi-scale cost volume to the same scale, and fusing according to the channel attention and spatial attention.

[0050] S44: Aggregating the fused cost volume again.

[0051] S45: Obtaining the cost volume with the highest resolution.

[0052] The specific steps of step S5 are as follows:

[0053] S51: Using 3D convolution to regularize the cost volume with the highest resolution to output the channel number of the cost volume to 1.

[0054] S52: Using the Softmax function to calculate the disparity probability distribution of the regularized cost volume.

[0055] S53: Multiplying the set disparity range by the probability distribution to obtain the initial disparity map.

[0056] The specific steps of step S6 are as follows:

[0057] S61: Inputting the initial disparity map, the cost volume, and the context information extracted from the left image into the GRU.

[0058] S62: The GRU updates the disparity residual and the hidden state of the context information.

[0059] S63: add the disparity residual and the initial disparity image.

[0060] S64: continue updating until a set number of updates is reached.

[0061] S65: output the updated disparity map and the upsampled mask.

[0062] S71: upsample the updated disparity map to full size according to the upsampled mask.

[0063] Embodiment 1 extracts multi-scale enhanced features of a stereo image pair, constructs a multi-scale cost volume, fuses and aggregates the multi-scale cost volume, obtains an initial disparity map after regularization, iteratively refines the disparity map, and upsample the disparity map to full size, thereby realizing the generation of a disparity map for a stereo image.

[0064] Embodiment 2: The steps of this embodiment are the same as those of Embodiment 1, except that each step is applied to a specific example, which includes the following steps:

[0065] S1: obtain a pair of core line corrected optical satellite stereo images of the same size HxW , ;

[0066] S21: extract multi-scale features from a pair of stereo images through convolution operation:

[0067] ;

[0068] ,

[0069] wherein, , , are the number of channels, height and width of the image, respectively.

[0070] S31: stitch the features of the two images at the same scale:

[0071] ;

[0072] S32: use the cross-attention of the Transformer to associate the stitched features at the same scale, and then separate them to obtain enhanced multi-scale features:

[0073] .

[0074] S41: subtract the enhanced features of the two images at the same scale within a set disparity range:

[0075]

[0076] equation, is the disparity range, respectively denote the pixel row, column coordinates, denotes the difference of feature vectors.

[0077] S42: Obtain the constructed multi-scale cost volume:

[0078]

[0079] the formula, denotes the disparity range;

[0080] S43: Fuse and aggregate the multi-scale cost volume. Obtain the cost volume under the highest resolution:

[0081] ;

[0082] S51: Regularize the cost volume under the highest resolution to obtain:

[0083] .

[0084] S52: Calculate the probability distribution using softmax, and then multiply it by the disparity range to obtain the initial disparity map:

[0085] ;

[0086] S61: Input the initial disparity map , the cost volume under the highest resolution , and the context information of the left view of the image , into the GRU for iterative updating

[0087] S62: The calculation process of a single GRU is:

[0088]

[0089] S63: The initial disparity is updated:

[0090]

[0091] is the disparity residual for each update.

[0092] S71: According to the up-sampling mask, up-sample the updated disparity map to the full size:

[0093]

[0094] is the disparity after the last update.

[0095] The embodiment 2 extracts and enhances multi-scale features to construct a multi-scale cost volume; then, the multi-scale cost volumes are fused and aggregated from coarse to fine to obtain a cost volume at the highest resolution, which improves the information propagation capability and further improves the prediction capability in the occluded area. Then, the disparity residual is updated by GRU to obtain a more fine disparity map.

[0096] The running environment of the embodiment 2 is based on the open source machine learning library Pytorch and is implemented under the Nvidia RTX A6000 graphics processor, which can provide efficient and stable computing support for the running of stereo matching of optical satellite stereo images.

[0097] Embodiment 3: The embodiment 3 is used to implement the principle of the above method embodiment to construct a stereo matching module based on Transformer feature enhancement and multi-scale cost fusion, which includes a multi-scale feature extraction module, a feature enhancement module, a cost volume fusion and aggregation module and a disparity updating module. The multi-scale feature extraction module is used to extract multi-scale features of stereo images. The feature enhancement module is used to enhance the multi-scale features to improve the discrimination ability of the features. The cost volume fusion module is used to fuse the cost volume information of different scales. The cost volume aggregation module is used to correct the error information in the cost volume and improve the consistency of the information. The disparity updating module is used to refine the initial predicted disparity information to achieve more accurate disparity estimation.

[0098] The feature extraction module is used to extract multi-scale features of a stereo image pair. The features are extracted and down-sampled by a convolutional neural network to obtain features at multiple scales.

[0099] The feature enhancement module is used to enhance the correlation of the features and improve the recognition of the features. As shown in Figure 3 , the feature enhancement module is designed based on Transformer, which cross-attends the features of the same scale of the two images to achieve feature enhancement.

[0100] As shown in Figure 4 , the cost volume fusion and aggregation module is a whole schematic diagram. As shown in Figure 5 , the cost volume fusion module details, the cost volume fusion module integrates the cost information of different scales through channel attention and spatial attention. As shown in Figure 6 , the cost volume aggregation module details, the cost aggregation module aggregates the information in the cost volume through a series of 3D convolution, which effectively maintains the smoothness and effectiveness of the cost volume information.

[0101] The disparity updating module is designed based on GRU. GRU updates the hidden state of the context information through gating method, indexes the disparity from the cost volume according to the context information and the initial disparity, and updates the disparity.

[0102] Embodiment 4: This embodiment 4 extracts multiple scale features and enhances, constructs a multi-scale cost volume; then, from coarse to fine, the multi-scale cost volume is fused and aggregated to obtain the cost volume under the highest resolution, and this multi-scale fusion method improves the information propagation capability and further improves the prediction capability in the occluded area. Then, the GRU is used to update the disparity residual to obtain a more fine disparity prediction result.

[0103] This embodiment 4 uses high-resolution satellite stereo images to train and test. In the training stage, the input image size is randomly cropped to 640x960, and in the test stage, the original image is used, and the AdamW optimizer is used for end-to-end training. For all data sets, we use a single cycle learning rate for training. The Transformer enhancement block is set to 6, and the number of iterations of the disparity refinement is set to 16.

[0104] The application extracts multi-scale features of stereo images; respectively, the multiple scale features are enhanced using the Transformer to obtain multi-scale enhanced features; a multi-scale cost volume is constructed through the enhanced features, the multi-scale cost volume is aggregated, and after aggregation, from the lowest resolution to high, the fusion is carried out in turn, and the cost volume with the highest resolution is output; the output cost volume and the disparity map are input into the GRU to update, and the refined disparity map is obtained; the refined disparity map is up-sampled to full size, and high-quality disparity map generation for stereo images is realized.

[0105] The application improves the accuracy of disparity prediction by performing feature extraction and cost volume construction at multiple scales. The end-to-end stereo matching network constructed based on the deep learning method of the application well solves the problem of disparity estimation in real optical satellite stereo matching.

[0106] The application is described with reference to the flowcharts and / or block diagrams according to the embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices produce a machine that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The functions specified in one flow or multiple flows and / or blocks Figure 1 The device that realizes the functions specified in one block or multiple blocks.

[0107] In the description of the application, it needs to be understood that the terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0108] Obviously, those skilled in the art can make various modifications and variations to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalent technologies, the present application also intends to include these modifications and variations.

Claims

1. A stereo matching method based on feature enhancement and multi-scale cost fusion, characterized in that, Parallax estimation for optical satellite stereo imagery includes the following steps: Acquire a pair of stereo images of an epipolar-corrected optical satellite of the same size, extract features, obtain multi-scale features from the two images, and enhance the features. After feature enhancement, the multi-scale features are constructed into a cost volume according to the set disparity range, and then fused and aggregated to obtain the cost volume at the highest resolution. The highest resolution cost volume is regularized to obtain the initial disparity map; The initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image are input into the GRU for iterative updates to obtain a refined disparity map. The updated disparity map is then upsampled to full size based on the upsampling mask. When performing feature enhancement, the features of the two images are stitched together at the same scale; the cross attention of the Transformer is used to associate the stitched features at the same scale to obtain the enhanced features; the stitched enhanced features are then separated to obtain the enhanced features of the two images respectively.

2. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 1, characterized in that: During feature extraction, the multi-scale features are extracted from a pair of images through convolution and downsampling operations.

3. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 2, characterized in that: In the process of acquiring the cost volume at the highest resolution, the enhancement features of the same scale in two images are subtracted along a set disparity range to construct a multi-scale cost volume; the cost volume at each scale is aggregated using 3D convolution; the aggregated multi-scale cost volumes are upsampled or downsampled to the same scale and fused according to channel attention and spatial attention; the fused cost volumes are then aggregated again. Obtain the cost volume at the highest resolution.

4. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 3, characterized in that: In the process of obtaining the initial disparity map, the highest resolution cost volume is regularized using 3D convolution to output the number of channels of the cost volume as 1; the disparity probability distribution is calculated on the regularized cost volume using the Softmax function; the disparity range is set and multiplied by the probability distribution to obtain the initial disparity map.

5. The stereo matching method based on feature enhancement and multi-scale cost fusion according to claim 4, characterized in that: During the process of acquiring the refined disparity map, the hidden state of the disparity residual and context information is updated by GRU, and the disparity residual is added to the initial disparity image; the update continues until the set number of updates is reached; and the updated disparity map and upsampling mask are output.

6. A stereo matching system based on feature enhancement and multi-scale cost fusion, characterized in that, Parallax estimation for optical satellite stereo imagery includes: The feature extraction and enhancement module is used to acquire stereo images of a pair of epipolar-corrected optical satellites of the same size, extract features, obtain multi-scale features of the two images, and enhance the features. The fusion and aggregation module is used to construct a cost volume from the enhanced multi-scale features according to the set disparity range, and then perform fusion and aggregation processing to obtain the cost volume at the highest resolution. The regularization module is used to regularize the cost volume at the highest resolution to obtain the initial disparity map; The stereo matching module is used to input the initial disparity map, the cost volume at the highest resolution, and the context information of the left view of the image into the GRU for iterative updates to obtain a refined disparity map, and then upsample the updated disparity map to the full size according to the upsampling mask. When performing feature enhancement, the features of the two images are stitched together at the same scale; the cross attention of the Transformer is used to associate the stitched features at the same scale to obtain the enhanced features; the stitched enhanced features are then separated to obtain the enhanced features of the two images respectively.

Images