Context-aware geometry-based progressive disparity optimization method

Abstract

This invention relates to a progressive stereo disparity optimization method based on contextual geometry awareness, belonging to the fields of computer vision and photogrammetry. The method includes: acquiring initial images of left and right views, preprocessing the images, and extracting image features; generating an initial cost volume based on the stitching cost volume and grouped correlation cost volume; aggregating costs using several hourglasses, and convolving the cost volumes of the last two hourglasses at different scales; performing disparity regression prediction based on the output content; and performing iterative optimization of spatial disparity using SAWG and SPON modules to complete the matching. This invention continuously incorporates multi-scale cost volume information during stereo matching, iteratively optimizing the initial disparity map, ultimately solving the problems of edge dilation and texture loss.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and photogrammetry, and relates to a progressive parallax optimization method based on contextual geometry awareness. Background Technology

[0002] With the continuous development of computer technology, two-dimensional spatial data, represented by maps and images, can no longer meet people's needs for understanding complex geographical spaces. Therefore, accurately and efficiently acquiring three-dimensional information has become a hot research topic in the fields of photogrammetry and computer vision. Currently, the main methods for acquiring three-dimensional data can be divided into two categories: active sensing and passive sensing. Active sensing methods acquire three-dimensional data through LiDAR, which, although highly accurate, suffers from high costs and data sparsity. In contrast, passive sensing methods, such as stereo matching, can acquire high-precision three-dimensional data through image matching calculations and have the significant advantage of lower costs. Stereo matching, as the core of key technologies such as 3D reconstruction, stereo navigation, and non-contact ranging, is widely used in many fields such as smart cities, autonomous driving, digital twins, natural disaster monitoring, cultural heritage protection, lunar exploration, and intelligent robot control.

[0003] Currently, stereo matching algorithms are mainly divided into traditional methods and deep learning methods. Traditional stereo matching methods mainly include steps such as matching cost calculation, cost aggregation, disparity calculation, and disparity post-processing. Representative methods include SGM and PatchMatch. SGM combines local information with global constraints and uses path accumulation to aggregate costs, gradually finding the optimal disparity estimate; PatchMatch uses random initialization and propagation strategies to iteratively optimize and quickly search for similar local regions in the image, gradually approaching the optimal matching result. Although these methods have low computational resource requirements and strong logic, they mainly rely on manually designed cost functions and have poor robustness in complex scenarios such as changes in illumination and missing textures.

[0004] Deep learning-based methods inherit the theoretical foundation of traditional methods, including steps such as feature extraction, cost volume construction, cost aggregation, and disparity regression. GC-Net is the first end-to-end deep learning stereo matching network, achieving cost aggregation through 3D convolution; IGEV-Stereo constructs a combined geometric encoding volume, which encodes geometric and contextual information as well as local matching details, and iteratively indexes it to update the disparity map. These methods significantly improve the accuracy of disparity estimation, but some shortcomings remain: models typically perform cost aggregation at a 1 / 4 scale and then interpolate to the full scale for disparity regression, which easily leads to edge inflation and loss of texture information in the prediction results. Furthermore, many methods use stacked hourglass modules to aggregate cost volume information and fuse the encoded and decoded cost volumes through splicing or residual connections, failing to fully utilize the fused cost volume information. This limits the model's matching accuracy in weakly textured regions and reduces its robustness in scenes with large disparity ranges. Therefore, effectively addressing disparity map edge inflation and texture loss while improving model robustness has become a key challenge in acquiring 3D geometric information for urban scenes. Summary of the Invention

[0005] To address the issues of edge inflation and texture loss in existing methods for predicting disparity maps, this invention proposes a context-aware, geometry-aware progressive disparity optimization method to improve the model's prediction accuracy in edge and texture regions, as well as its overall robustness.

[0006] To achieve the above objectives, the present invention provides the following technical solution:

[0007] A context-aware geometry-based progressive disparity optimization method, comprising the following steps:

[0008] S1. Obtain the initial images of the left and right views, preprocess the images, and extract image features;

[0009] S2. Construct the splicing cost volume and grouping correlation cost volume based on the splicing features and correlation features, which are used to generate the initial cost volume;

[0010] S3. Cost aggregation is performed using three hourglasses, and the cost volumes of the last two hourglasses at different scales are convolved into a single channel and then interpolated to the full scale as input for the disparity optimization stage.

[0011] S4. Based on the cost volume output by the last hourglass, interpolate to the full scale, and then perform disparity regression to predict the initial disparity map.

[0012] S5. Perform disparity iterative optimization through the SAWG and SPON modules. In the SAWG module, spatial attention weights are used to guide the cost volume at each scale. In the SPON module, the disparity map is updated using cost volume information and the spatial consistency between the RGB image and the initial disparity map to complete the matching.

[0013] Furthermore, in step S1, the preprocessing includes: randomly cropping the input images of the left and right views into RGB format of a preset size and performing standardization processing;

[0014] Then, features are extracted using the ResNet network. The feature extraction process includes:

[0015] The input image is first downsampled to extract 32 channels of feature information at a 1 / 2 scale. Then, after further feature extraction through two convolutional layers and three residual blocks, it is downsampled again, passing through 16 residual blocks to obtain 64 channels of feature information l1 at a 1 / 4 scale. Next, l1 is processed through three residual blocks to extract 128 channels of feature information l2, and further processed through another three residual blocks to obtain 128 channels of feature information l3. Finally, l1, l2, and l3 are concatenated along the channel dimension to obtain 320 channels of feature information l4, which is used to construct the correlation cost body. Simultaneously, l4 is convolved to extract 12 channels of feature information, which are used to construct the concatenated cost body.

[0016] Furthermore, in step S2, the features are convolved to 12 channels to construct the stitching cost volume, which is expressed as:

[0017] C concat (d,x,y,·)=Concat{f l (x,y),f r (xd,y)}

[0018] In the formula, C concat (d,x,y,·) represents the stitching cost volume, where d,x,y are the disparity, pixel x-coordinate, and pixel y-coordinate, respectively, and "·" indicates a placeholder. l (·) represents the pixel feature vector of the left image, f r (·) represents the pixel feature vector of the right image, and Concat{·} represents the concatenation method;

[0019] The channels are divided into several groups, and the correlation cost volume is constructed for each group, as follows:

[0020]

[0021] In the formula, C gwc (d,x,y,g) represents the relevant cost volume, g is the index of the feature channel group, and N... cN g These represent the total number of feature channels and the number of groups, respectively. Let be the eigenvectors of the left image of the g-th group and the eigenvectors of the right image of the g-th group, respectively, and <·> denote the dot product of the vectors;

[0022] The concatenated cost volume and the grouped related cost volume are concatenated and fused together as the input for cost aggregation.

[0023] Furthermore, in step S3, multiple 3D hourglasses are used for cost volume aggregation, and each 3D hourglass includes:

[0024] Encoding stage: First, the 32-channel 1 / 4 scale cost volume V4 is downsampled to obtain a 64-channel 1 / 8 scale cost volume. Then, it is passed through a 3D convolution to obtain V8, which is then downsampled again and passed through a 3D convolution to obtain a 128-channel 1 / 16 scale cost volume V. 16 ;

[0025] Decoding phase: Cost volume V 16 Perform 3D deconvolution to obtain V′8, add V′8 and V8 after passing through a 1*1*1 convolution kernel, and then activate the result with the ReLU function to obtain V″8; V″8 is obtained in the same way to obtain V″4; V″4 is used as the input for the subsequent hourglass;

[0026] Take 1 / 16, 1 / 8, and 1 / 4 of the cost volume of the last two hourglasses, convolve them to a single channel, and interpolate them to the full scale. Use this as input for disparity optimization.

[0027]

[0028] In the formula, res represents the scale, and V res Cost volume V representing the res scale res ∈R (D×res)×(H×res)×(W×res) f conv (.) represents 3D convolution, and Interp(.) represents trilinear interpolation. The full-scale cost volume obtained by interpolating the res-scale cost volume.

[0029] Further, in step S4, for each output module, a 1-channel cost volume is generated using two 3D convolutions and then upsampled. Subsequently, a softmax function is applied along the disparity dimension to convert the cost volume into a probability volume. Finally, the predicted disparity for each pixel is calculated by weighted summation, as shown in the following formula:

[0030]

[0031] In the formula, D is the predicted disparity value. maxFor maximum disparity, k is a discrete candidate value for disparity, and p k This represents the probability value for k.

[0032] Furthermore, in the SAWG module of step S5, a lightweight U-net network is used to extract fine features from the RGB image. Then, spatial attention weights are generated using the Sigmoid activation function to activate the cost volume, generating weights that are applied to the cost volume. The process is represented as follows:

[0033] V'(d,x,y)=V(d,x,y)⊙(σ(f U-Net (I)))

[0034] In the formula, V(.) represents the cost volume V(.)∈R D×H×W I is an RGB image I∈R 3×H×W f U-Net (.) represents a lightweight U-Net network, ⊙ denotes element-wise multiplication, and σ(.) is the sigmoid activation function.

[0035] Furthermore, the SPON module in step S5 includes the following operations:

[0036] S511. After expanding the RGB image and the initial disparity map to the C channel, they are stitched and fused. Then, the cost volume interpolated to the full scale is processed by convolution to the C channel, and the two are stitched and fused again.

[0037] S512. Subsequently, a U-Net network is used to learn the spatial consistency of disparity map, features and cost volume, generate weights of B*9*H*W and biases (B*16*H*W) in the x and y directions of the surrounding 8 pixels.

[0038] S513. Then, the center point is inserted into the bias. The generated bias is used to locate the corresponding point through the sampler. Finally, the residual is obtained by matrix multiplication with the decentralized weights, and the disparity map is updated.

[0039] The beneficial effects of this invention are as follows:

[0040] This invention utilizes cost volume information, spatial consistency between disparity maps and RGB images to generate correlation weights for each pixel with surrounding pixels, and performs spatial optimization on the full-scale disparity map. This can greatly enhance the geometric consistency of the disparity map to solve the problems of disparity information discontinuity and edge inflation.

[0041] This invention makes full use of the multi-scale cost volume information of two hourglasses, and uses cost volumes of different scales to iteratively update the disparity map, thereby enhancing the prediction effect of the predicted disparity map in weak texture regions and the robustness of large disparity range scenes.

[0042] This invention uses a lightweight U-Net to extract fine feature information to generate spatial attention weights to guide cost volume information, effectively alleviating the problems of excessive smoothing of cost volume and insufficient texture information.

[0043] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0044] To make the objectives, technical solutions, and advantages of the present invention clearer, the preferred embodiments of the present invention will be described in detail below with reference to the accompanying drawings, wherein:

[0045] Figure 1 This is a schematic diagram of the overall architecture of the context-aware, geometry-aware progressive disparity optimization method of the present invention.

[0046] Figure 2 This is a schematic diagram of the feature extraction process of the present invention;

[0047] Figure 3 This is a schematic diagram of the cost-volume polymerization process of the present invention;

[0048] Figure 4 This is a schematic diagram of the SAWG module of the present invention;

[0049] Figure 5 This is a schematic diagram of the SPON module of the present invention. Detailed Implementation

[0050] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0051] The accompanying drawings are for illustrative purposes only and are schematic diagrams, not actual pictures. They should not be construed as limiting the invention. To better illustrate the embodiments of the invention, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions. It is understandable to those skilled in the art that some well-known structures and their descriptions may be omitted in the drawings.

[0052] In the accompanying drawings of the embodiments of the present invention, the same or similar reference numerals correspond to the same or similar components. In the description of the present invention, it should be understood that if terms such as "upper," "lower," "left," "right," "front," and "rear" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, the terms used to describe positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present invention. For those skilled in the art, the specific meaning of the above terms can be understood according to the specific circumstances.

[0053] Please see Figures 1-5 This is a progressive parallax optimization method based on context-aware geometry.

[0054] Example

[0055] This embodiment provides a detailed implementation process of a context-aware, geometry-aware, progressive disparity optimization method, which specifically employs a multi-scale spatial optimization network model architecture, such as... Figure 1 As shown, this model enhances the prediction performance of the disparity map in weakly textured regions and its robustness in scenes with large disparity ranges through a multi-scale cost volume optimization strategy. In this model, based on the GWC-net framework, the last two hourglass modules of GWC-Net are used, and cost volumes at scales of 1 / 16, 1 / 8, and 1 / 4 of each hourglass are sequentially passed through the SAWG (Spatial Attention Weight Generation) module to supplement spatial attention feature information. The obtained cost volumes and the disparity map to be updated are input into the SPON (Spatial Optimization Network) module to generate residuals for updating the disparity map. The model of this invention continuously incorporates multi-scale cost volume information, updating the disparity from coarse to fine, ultimately solving the problems of edge dilation and disparity inconsistency. Specifically, the following steps are included:

[0056] S1. Obtain the initial images of the left and right views, preprocess the images, and extract image features;

[0057] S2. Generate the initial cost volume based on the splicing cost volume and the grouping-related cost volume;

[0058] S3. Cost aggregation is performed using three hourglasses, and the cost volumes of the last two hourglasses at different scales are convolved into a single channel and then interpolated to the full scale as input for the disparity optimization stage.

[0059] S4. Based on the cost volume output by the last hourglass, interpolate to the full scale, and then perform disparity regression to predict the initial disparity map.

[0060] S5. Spatial disparity iterative optimization is performed through the SAWG and SPON modules. In the SAWG module, spatial attention weights are used to guide the cost volume at each scale. In the SPON module, the disparity map is updated using cost volume information and the spatial consistency between the RGB image and the initial disparity map to complete the matching.

[0061] In step S1 of this embodiment, the left and right view images are first preprocessed by randomly cropping the input images of the left and right views into RGB format with a size of 512×256 and standardizing the data. Then, features are extracted using the ResNet network, such as... Figure 2 As shown, the feature extraction process includes:

[0062] The input image is first downsampled to extract 32 channels of feature information at a 1 / 2 scale. Then, after further feature extraction through two convolutional layers and three residual blocks, it is downsampled again, passing through 16 residual blocks to obtain 64 channels of feature information l1 at a 1 / 4 scale. Next, l1 is processed through three residual blocks to extract 128 channels of feature information l2, and further processed through another three residual blocks to obtain 128 channels of feature information l3. Finally, l1, l2, and l3 are concatenated along the channel dimension to obtain 320 channels of feature information l4, which is used to construct the correlation cost body. Simultaneously, l4 is convolved to extract 12 channels of feature information, which are used to construct the concatenated cost body.

[0063] In step S2 of this embodiment, the cost volume consists of two parts: the concatenated cost volume and the group-related cost volume. Concatenated volume: Features are convolved into 12 channels for constructing the concatenated cost volume. Group-related cost volume: Channels are divided into 40 groups, and a related cost volume is constructed for each group to obtain the group-related cost volume. Finally, the concatenated cost volume and the group-related cost volume are concatenated and fused as the input for cost aggregation. The formula for the concatenated cost volume is:

[0064] C concat (d,x,y,·)=Concat{f l (x,y),f r (xd,y)}

[0065] In the formula, C concat (d,x,y,·) represents the stitching cost volume, where d,x,y are the disparity, pixel x-coordinate, and pixel y-coordinate, respectively, and "·" indicates a placeholder. l (·) represents the feature vector of the pixel in the left image, f r(·) represents the feature vector of the pixel in the right image, and Concat{·} represents the concatenation method;

[0066] The relevant cost volume formula is:

[0067]

[0068] In the formula, C gwc (d,x,y,g) represents the relevant cost volume, g is the index of the feature channel group, and N... c N g These represent the total number of feature channels and the number of groups, respectively. Let be the eigenvectors of the left image of the g-th group and the eigenvectors of the right image of the g-th group, respectively, and <·> denote the dot product of the vectors;

[0069] In step S3 of this embodiment, multiple 3D hourglasses are used for cost body aggregation, wherein, as... Figure 3 As shown, this embodiment uses three 3D hourglasses to aggregate the cost volume. Each 3D hourglass includes an encoding stage and a decoding stage.

[0070] Encoding stage: First, the 32-channel 1 / 4 scale cost volume V4 is downsampled to obtain a 64-channel 1 / 8 scale cost volume. Then, it is passed through a 3D convolution to obtain V8. After downsampling and passing through a 3D convolution, a 128-channel 1 / 16 scale cost volume V16 is obtained.

[0071] Decoding stage: The cost volume V16 is deconvolved in 3D to obtain V8_1. V8_1 is added to V8 after passing through a 1*1*1 convolution kernel, and the result is activated by the ReLU function to obtain V8_2. V8_2 is processed in the same way to obtain V4_2. V4_2 is used as the input for the subsequent hourglass.

[0072] Take 1 / 16, 1 / 8, and 1 / 4 of the scale cost volume of the last two hourglasses, convolve them to a single channel, interpolate them to the full scale, and use them as input for disparity optimization.

[0073]

[0074] In the formula, res represents the scale, and V res Cost volume V representing the res scale res ∈R (D×res)×(H×res)×(W×res) f conv (.) represents 3D convolution, and Interp(.) represents trilinear interpolation. The full-scale cost volume obtained by interpolating the res-scale cost volume.

[0075] In step S4 of this embodiment, the disparity regression prediction process is as follows: For each output module, a cost volume with 1 channel is generated using two 3D convolutions and then upsampled. Subsequently, a softmax function is applied along the disparity dimension to convert the cost volume into a probability volume. Finally, the predicted disparity for each pixel is calculated by weighted summation (i.e., multiplying the probability value of each disparity by the disparity value and summing them), as shown in the following formula:

[0076]

[0077] In the formula, D is the predicted disparity value. max For maximum disparity, k is a discrete candidate value for disparity, and p k This represents the probability value for k;

[0078] In step S5 of this embodiment, the SAWG module is used to process each cost volume using multi-scale cost volume information to alleviate the problems of over-smoothing of cost volumes and insufficient texture information. By inputting the optimized cost volume and the RGB image, the SPON module is used to learn the spatial consistency between pixels and generate residuals to update the full-scale disparity map.

[0079] To alleviate the problems of excessive smoothing of cost volumes and insufficient texture information, this invention proposes a SAWG module, which utilizes RGB images to generate subtle spatial attention weights to guide cost volumes at various scales. Figure 4 As shown, this embodiment mainly uses a lightweight U-net to extract fine features from the RGB image, and then uses the Sigmoid activation function to generate spatial attention weights to activate the cost volume. The generated weights are applied to the cost volume, and the process is represented as follows:

[0080] V'(d,x,y)=V(d,x,y)⊙(σ(f U-Net (I)))

[0081] In the formula, V(.) represents the cost volume V(.)∈R D×H×W I is an RGB image I∈R 3×H×W f U-Net (.) represents a lightweight U-Net, ⊙ denotes element-wise multiplication, and σ(.) is the sigmoid activation function.

[0082] To address the issues of edge dilation and disparity discontinuity caused by interpolation, this invention proposes a SPON module to optimize the disparity map across all scales. It primarily uses cost volume information, the RGB image, and the initial disparity map to obtain weights and biases through deformable convolution. These weights and biases are then used to identify spatial consistency points between pixels and their surrounding pixels. Residuals generated from these consistency points are then used to update the disparity map.

[0083] like Figure 5 As shown, firstly, the RGB image and the initial disparity map are expanded to the C channel and then stitched together. Then, the cost volume interpolated to the full scale is convolved to the C channel, and the two are stitched together.

[0084] F in =Concat(f conv (I,C),f conv (D init ,C))

[0085] F out =f conv (Concat(f conv (F in ,C),f conv (V init ,C)),C)

[0086] f conv (.,C) represents convolution to channel C, and I represents the input RGB image I∈R. 3×H×W D init Disparity map D representing the input init ∈R 1×H×W V init V represents the full-scale cost volume of the input. init ∈R 192×H×W Concat(.) represents splicing along channel C.

[0087] Subsequently, a U-Net network is used to learn the spatial consistency of the disparity map, features, and cost volume. The obtained consistency information serves two purposes: first, it is used to generate weights (B*9*H*W), and second, it is used to generate the biases of surrounding pixels (B*16*H*W).

[0088] F corr =f U-Net (F out )

[0089] weight(C,H,W)=f conv (F corr ,9)

[0090]

[0091] offset(C,H,W)=f conv (F corr ,16)

[0092] f conv (.,C) represents convolution to channel C, f U-Net (.) represents a lightweight U-Net, C iThis represents the value of the i-th channel.

[0093] Then, the biases in the x and y directions of the center point are inserted into the bias. The generated biases are used to locate the corresponding points through a sampler, and finally, a matrix multiplication is performed with the weights to obtain the residual, which is used to update the disparity map. The process is represented as follows:

[0094]

[0095] coord=f gridgenerate (k)+offset(b*hw,r,r,2)

[0096] coord=(coord / k×2)-1

[0097] depth res =f sample (f unfold (depth,k),coord)

[0098] In the formula, f Pixelinsert (.) represents offset insertion at the pixel center point, f gridgenerate (.) represents a uniformly generated mesh, f unfold (.) indicates selecting pixel values ​​within a k*k range for each pixel, f sample (.) indicates that the parallax pixel is selected based on the grid coordinates.

[0099] The cost volumes obtained from the last two hourglasses are interpolated into full-scale regression using conventional methods to obtain the initial full-scale disparity map. Spatial attention weights are introduced to guide the interpolated cost volumes at each scale. Then, the disparity map, the RGB left image, and the guided cost volume are input into SPON to generate residuals, and the initial disparity map is iteratively updated. Using GWC-net as the baseline model, this embodiment updates the disparity map 6 times.

[0100] Output and Result Validation:

[0101] In this embodiment, the Sceneflow dataset is used to verify the performance of the present invention. The size of the generated final disparity map is the same as that of the original input. Figure 1 The final comparison results between this invention and other existing methods are shown in Table 1:

[0102] Table 1

[0103]

[0104] Experiments show that state-of-the-art results were achieved on the Sceneflow dataset. Compared to Gwc-net, the EPE was significantly improved from 0.61 to 0.43 under the same training conditions. The strategy of this invention achieves state-of-the-art results on the Sceneflow dataset.

[0105] It is worth mentioning that the SAWG module and SPON module of the present invention can be seamlessly embedded into other stereo matching networks that use hourglass modules for the optimization and updating of full-scale disparity maps. Models such as PSM-Net, GC-Net and CF-Net can all integrate the above modules.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A progressive disparity optimization method based on context-aware geometry, characterized in that: The method includes the following steps: S1. Obtain the initial images of the left and right views, preprocess the images, and extract image features; S2. Construct the splicing cost volume and grouping correlation cost volume based on the splicing features and correlation features, which are used to generate the initial cost volume; S3. Cost aggregation is performed using three hourglasses, and the cost volumes of the last two hourglasses at different scales are convolved into a single channel and then interpolated to the full scale as input for the disparity optimization stage. S4. Based on the cost volume output by the last hourglass, interpolate to the full scale, and then perform disparity regression to predict the initial disparity map. S5. Perform disparity iterative optimization through SAWG and SPON modules. In the SAWG module, spatial attention weights are used to guide the cost volume at each scale. In the SPON module, the disparity map is updated using cost volume information, spatial consistency between the RGB image and the initial disparity map, and matching is completed. In step S3, multiple 3D hourglasses are used for cost volume aggregation. Each 3D hourglass includes: Encoding stage: First, the 1 / 4 scale cost volume of the 32 channels is... Downsampling is performed to obtain a 64-channel, 1 / 8-scale cost volume, which is then processed by a 3D convolution to obtain... Then, downsampling is performed, followed by 3D convolution, to obtain a 128-channel, 1 / 16 scale cost volume. ; Decoding phase: Cost volume 3D deconvolution is performed to obtain , Will and after passing through a 1*1*1 convolution kernel Add them together, then activate the ReLU function on the result to obtain the final product. ; Obtained in the same way ; As input for subsequent hourglasses; Take 1 / 16, 1 / 8, and 1 / 4 of the cost volume of the last two hourglasses, convolve them to a single channel, and interpolate them to the full scale. Use this as input for disparity optimization. In the formula, Representative scale, represent The cost of scale , Represents 3D convolution. Represents trilinear interpolation. The full-scale cost volume obtained by interpolating the res-scale cost volume. ; In the SAWG module of step S5, a lightweight U-net network is used to extract fine features from the RGB image. Then, spatial attention weights are generated using the Sigmoid activation function to activate the cost volume. The generated weights are applied to the cost volume. The process is represented as follows: In the formula, For the sake of volume I represents an RGB image. , For lightweight U-Net networks, This indicates element-wise multiplication. It is the sigmoid activation function; The SPON module in step S5 includes the following operations: S511. After expanding the RGB image and the initial disparity map to the C channel, they are stitched and fused. Then, the cost volume interpolated to the full scale is processed by convolution to the C channel, and the two are stitched and fused again. S512. Subsequently, a U-Net network is used to learn the spatial consistency of disparity map, features and cost volume, generate weights of B*9*H*W and biases of B*16*H*W in the x and y directions of the surrounding 8 pixels. S513. Then, the center point is inserted into the bias. The generated bias is used to locate the corresponding point through the sampler. Finally, the residual is obtained by matrix multiplication with the decentralized weights, and the disparity map is updated.

2. The progressive disparity optimization method based on context-aware geometry as described in claim 1, characterized in that: In step S1, the preprocessing includes: randomly cropping the input images of the left and right views into RGB format of a preset size and performing normalization processing; Then, features are extracted using the ResNet network. The feature extraction process includes: The input image is first downsampled to extract 32 channels of feature information at a 1 / 2 scale. Then, after further feature extraction through two convolutional layers and three residual blocks, it is downsampled again, passing through 16 residual blocks to obtain 64 channels of feature information at a 1 / 4 scale. ;Then, The feature information was extracted into 128 channels after three residual blocks. Furthermore, feature information for 128 channels is obtained through three additional residual blocks. Finally, By performing concatenation along the channel dimension, we obtain feature information for 320 channels. This is used to construct the correlation cost body; simultaneously, for Convolution is performed to extract feature information from 12 channels, which is used to construct the splicing cost body.

3. The progressive disparity optimization method based on context-aware geometry according to claim 2, characterized in that: In step S2, the features are convolved to 12 channels to construct the stitching cost volume, which is expressed as: In the formula, To compensate for the volume loss during splicing, These represent parallax, pixel x-coordinate, pixel y-coordinate, and so on. " indicates a placeholder. This represents the pixel feature vector of the left image. This represents the pixel feature vector in the right image. Indicates the splicing method; The channels are divided into several groups, and the correlation cost volume is constructed for each group, as follows: In the formula, For the relevant cost volume, Index for grouping feature channels. These represent the total number of feature channels and the number of groups, respectively. These are the feature vectors of the left image and the right image of the g-th group, respectively. Represents the dot product of vectors; The concatenated cost volume and the grouped related cost volume are concatenated and fused together as the input for cost aggregation.

4. The progressive disparity optimization method based on context-aware geometry as described in claim 1, characterized in that: In step S4, for each output module, a 1-channel cost volume is generated using two 3D convolutions and then upsampled. Subsequently, a softmax function is applied along the disparity dimension to convert the cost volume into a probability volume. Finally, the predicted disparity for each pixel is calculated by weighted summation, as shown in the following formula: In the formula, For the predicted disparity value, For maximum parallax, These are discrete disparity candidate values. This represents the probability value for k.

Images