A binocular stereo matching method based on lightweight cost volume

Abstract

The application belongs to the technical field of computer vision, and proposes a binocular stereo matching method based on a light-weight cost volume. The method comprises the following steps: inputting left and right image data; inputting the left and right images into a feature extraction network, selecting features of at least three scales at the tail for splicing, and respectively forming upper branch features and lower branch features; respectively constructing an upper branch cost volume and a lower branch cost volume; inputting the upper branch cost volume and the lower branch cost volume into a multi-scale coupled aggregation network to obtain coupled upper branch fused geometric features and lower branch fused geometric features; performing disparity regression to obtain a low-resolution initial disparity map; and according to superpixel neighborhood weighted upsampling, restoring the full-resolution disparity map as the final matching result. The application improves the information expression ability of the cost volume, enhances the collaborative utilization effect of shallow and deep features, and improves the disparity estimation accuracy and robustness in complex regions while controlling the computational complexity and storage overhead.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology, and in particular relates to a binocular stereo matching method based on lightweight cost volume. Background Technology

[0002] The goal of binocular stereo matching is to find pixel correspondences between the left and right corrected images and recover scene depth from disparity. Existing deep learning stereo matching methods typically include four core stages: feature extraction, cost volume construction, cost aggregation, and disparity regression. Among these, the cost volume stores the matching costs of left and right image features under different disparity assumptions and is a key intermediate representation for the entire network to estimate the geometric structure.

[0003] While existing binocular stereo matching technology can achieve disparity estimation to a certain extent, it still has significant shortcomings in complex scenes, mainly in the following aspects: (1) Most existing methods typically employ only a single form of cost body construction, such as using only a correlation cost body, only a feature concatenation cost body, or only a certain grouping difference expression method. Although this type of single cost body can extract some matching information, the information dimension it represents is relatively limited, making it difficult to simultaneously take into account global similarity, local differences, and cross-channel contextual relationships. The technical reason for this is that different types of cost bodies have significantly different modeling capabilities for matching information: correlation cost bodies are better at describing the similarity between left and right features, but are insufficient in characterizing fine-grained structural differences and boundary changes; concatenation cost bodies retain more information, but have higher redundancy, resulting in a heavier burden on subsequent 3D convolution aggregation. Therefore, the single-path, single-expression cost body construction mechanism is prone to insufficient utilization of matching information, thereby affecting the final disparity estimation accuracy.

[0004] (2) Common cost volume construction and aggregation methods in existing technologies often face the problem of balancing accuracy and computational complexity. To improve matching accuracy, many methods tend to construct high-dimensional, dense cost volumes and perform aggregation processing through multi-layer 3D convolutional networks. While this approach can enhance context modeling capabilities, it significantly increases memory usage and computational load, resulting in large model parameter sizes and slow inference speeds, which is not conducive to practical deployment. The fundamental technical reason is that traditional stereo matching networks typically rely on full cost stacking within a large disparity search range. As the disparity dimension, spatial dimension, and channel dimension all expand, the cost volume rapidly increases, and the cost of 3D convolutional processing also increases accordingly. Therefore, existing solutions often require significant computational and storage costs when pursuing higher matching performance.

[0005] (3) Existing technologies do not adequately utilize the synergistic effect between shallow detail information and deep semantic information. In binocular matching tasks, shallow features typically contain rich texture, edge, and local structural information, which is beneficial for restoring object boundaries and detailed regions; while deep features have stronger semantic expressive power, which is beneficial for establishing stable matching in areas with weak texture, repetitive texture, and low contrast. However, many existing methods either focus on deep feature aggregation or simply perform multi-scale stitching, lacking an effective coupling mechanism for the differences between features at different levels, resulting in the inability to form complementary enhancements between information at different levels. The technical reason for this is that shallow features and deep features have natural differences in resolution, semantic intensity, and response patterns. Without a specially designed feature interaction and fusion structure, it is often difficult to achieve effective information alignment and complementary advantages by simply adding or stitching together features.

[0006] (4) Existing technologies often fail to provide satisfactory estimation results for disparity boundaries, occluded regions, and weakly textured regions, easily leading to problems such as blurred edges, matching errors, or excessively smoothed disparities. The technical reason for this is that these regions are inherently difficult areas in binocular matching: in occluded regions, there is a lack of information or mismatch between the left and right images; in weakly textured regions, the local feature discrimination is insufficient, easily resulting in multiple solutions; in regions with abrupt disparity boundary changes, if the cost representation capability is insufficient or the aggregation process is excessively smoothed, pixels from different depth layers are easily mixed, causing loss of edge details. Existing single cost representation methods and simple feature fusion methods often cannot simultaneously meet the matching requirements of these complex regions, thus leading to the long-standing existence of the above problems.

[0007] (5) Although some existing methods attempt to reduce complexity by introducing group correlation, feature compression, or lightweight structures, they still suffer from insufficient information representation or single branch function. The technical reason is that if only single-path compression or single-path correlation calculation is performed on the original features, although the amount of computation can be reduced, some discriminative information useful for matching may be lost. Especially in complex scenarios, the cost volume generated by relying on only a single path still has limited ability to jointly characterize feature similarity, difference, and contextual complementarity, making it difficult to further improve the robustness of disparity prediction.

[0008] In summary, existing technologies lack a network structure design that can simultaneously accommodate the representation of multiple types of matching information, the collaborative fusion of hierarchical features, and the balance between accuracy and efficiency. Summary of the Invention

[0009] To address the problems in existing binocular stereo matching techniques, such as the single cost volume representation, insufficient utilization of matching information, high computational complexity, insufficient coordination between shallow detail information and deep semantic information, and low disparity estimation accuracy in occluded regions, weak texture regions, and disparity abrupt boundary regions, this invention proposes a binocular stereo matching method based on a lightweight cost volume. This method constructs a dual cost volume branch structure and combines it with a multi-layer feature coupling mechanism to improve the information representation capability of the cost volume, enhance the synergistic utilization of shallow and deep features, and improve the disparity estimation accuracy and robustness in complex regions while controlling computational complexity and storage overhead.

[0010] The technical solution of the present invention is as follows: A stereo matching method based on lightweight cost volume includes the following steps: Step 1: Acquire the left and right images after camera calibration and epipolar correction, and perform preprocessing; Step 2: Input the left and right images into the feature extraction network, select features at least three scales at the end of the feature extraction network and splice them together to form upper branch features and lower branch features respectively; Step 3: Construct the upper branch cost body and the lower branch cost body based on the upper branch features and the lower branch features, respectively; Step 4: Input the upper branch cost body and the lower branch cost body into the multi-scale coupled aggregation network to obtain the coupled upper branch fused geometric features and lower branch fused geometric features; The multi-scale coupled aggregation network consists of two parallel three-dimensional aggregation networks and a coupling module located after at least one upsampling layer. Step 5: After summing or equivalently combining the upper branch fusion geometric features and the lower branch fusion geometric features, perform disparity regression to obtain a low-resolution initial disparity map; Step 6: Based on the superpixel neighborhood weighted upsampling strategy, restore the low-resolution initial disparity map to a full-resolution disparity map, which will be used as the final matching result.

[0011] Preferably, the left and right images are RGB color images, grayscale images, multispectral images, or special imaging techniques used in industrial vision.

[0012] Preferably, the preprocessing in step 1 includes normalization, scaling, random cropping, or color perturbation.

[0013] Preferably, the feature extraction network is a convolutional network with shared weights, including multi-layer two-dimensional convolution, batch normalization, non-linear activation, and downsampling units.

[0014] Preferably, the upper branch cost body is constructed using a group correlation matching mechanism, and the lower branch cost body is constructed using a norm correlation matching mechanism, a splicing mechanism, or a group difference mechanism.

[0015] Preferably, the upper branch cost body is constructed using a group-related matching mechanism, as follows:

[0016] in, Indicated in the disparity index Pixel position and the The group-related value of each group. Indicates the candidate disparity value. Represents the normalization coefficient. Indicates the left image features at pixel locations Place, No. Feature vectors in each group Indicates the right image features at pixel locations Place, No. Feature vectors in each group.

[0017] Preferably, the lower branch cost body adopts a norm-related matching mechanism, expressed as:

[0018] in, Indicated in the disparity index and pixel position The norm-related algebraic value at a given position, with the symbol ":" indicating the channel dimension output corresponding to that position. Indicates the left image features at pixel locations The feature vectors of all channels at that location. Indicates the right image features at pixel locations The feature vectors of all channels at that location. This represents the normalized correlation coefficient.

[0019] Preferably, the lower branch cost body adopts a splicing mechanism, represented as follows:

[0020] in, Indicated in the disparity index and pixel position The splicing-related cost features at the location, Indicates the left image features at pixel locations The feature vector extracted at that location, Indicates the right image features at pixel locations The feature vector extracted at that location, This indicates a concatenation operation.

[0021] Preferably, the three-dimensional aggregation network includes three-dimensional convolutional units, normalization units, activation units, and residual connection structures; the upper branch cost volume and the lower branch cost volume are respectively input into two three-dimensional aggregation networks, and a coupling module set after at least one upsampling layer enables information exchange between the intermediate cost representations of the two branches.

[0022] Preferably, the coupling module is represented as follows:

[0023] in, This represents the fused feature tensor output by the coupling module. This represents the input feature tensor from the lower branch or lower-level path, preserving local detail information, edge information, and texture information. This represents the input feature tensor from the upper branch or higher-level path, containing semantic and contextual information. This represents a convolution operation with a kernel size of 1×3×3. Indicates to Perform a 1×3×3 convolution operation.

[0024] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. The binocular stereo matching method based on lightweight cost bodies proposed in this invention can obtain more complete matching information than a single small cost body under limited computational budget by modeling two small cost bodies with different construction mechanisms in parallel.

[0025] 2. The binocular stereo matching method based on lightweight cost volume proposed in this invention does not simply splice the original cost volume, but performs coupling and fusion in the aggregated geometric feature space, which reduces the interference of redundant information on network learning.

[0026] 3. The binocular stereo matching method based on lightweight cost volume proposed in this invention has a multi-scale decoding coupling mechanism that is beneficial for recovering parallax details in edges, thin objects, smooth surfaces and weak texture areas.

[0027] 4. The binocular stereo matching method based on lightweight cost volume proposed in this invention has stronger adaptability to different data domains and different scene structures because the two cost volumes provide complementary geometric representations and are fused at multiple scale levels.

[0028] 5. The binocular stereo matching method based on lightweight cost volume proposed in this invention, compared with the scheme that relies on large cost volume and deep 3D convolution stacking, reduces the overall inference latency while maintaining high accuracy, and is more suitable for deployment-oriented binocular depth estimation applications. Attached Figure Description

[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the embodiments will be briefly introduced below. The features and advantages of the present invention can be more clearly understood by referring to the accompanying drawings. The accompanying drawings are schematic and should not be construed as limiting the present invention in any way. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0030] Figure 1 This is a flowchart of the binocular stereo matching method based on lightweight cost body proposed in this invention.

[0031] Figure 2 This is a network architecture diagram of the binocular stereo matching method based on lightweight cost volume proposed in this invention.

[0032] Figure 3 This is an example diagram showing the results of using the binocular stereo matching method based on lightweight cost body of the present invention. Detailed Implementation

[0033] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other.

[0034] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and therefore the scope of protection of the invention is not limited to the specific embodiments disclosed below.

[0035] like Figure 1 As shown, the binocular stereo matching method based on lightweight cost body of the present invention mainly includes the following process: Step 1: Input Data and Preprocessing The processor first acquires the left and right images after machine calibration and epipolar correction, causing corresponding points to shift primarily along the horizontal direction, thus transforming the binocular matching problem into a one-dimensional disparity search problem. The input image can be an RGB color image, or it can be extended to a grayscale image, a multispectral image, or special imaging data from industrial vision.

[0036] In some embodiments, the left and right images are two-dimensional image data of the same scene acquired from different perspectives. They are aligned in the vertical direction, so that the matching search of corresponding pixels is mainly carried out in the horizontal direction.

[0037] The input image is first normalized, and then data augmentation operations such as scaling, random cropping, and color perturbation are performed based on the training or deployment environment. This preprocessing step is not limited to any specific method, as long as it ensures the input conforms to the tensor format of the neural network and maintains epipolar geometric consistency between the left and right images.

[0038] In some embodiments, the processor denotes the left image as I l The right view image is denoted as I r Subsequently, the processor performs normalization processing on the input image to reduce the impact of brightness and contrast variations under different acquisition conditions on the feature extraction results. Normalization processing can include pixel value scaling, mean-variance standardization, or other data preprocessing methods suitable for neural network inputs.

[0039] In some embodiments, the processor may also resize the image to match the input resolution with the network's preset input size; or, padding may be applied without changing the original aspect ratio to meet the network's computational requirements for tensor size divisibility.

[0040] Step 2: Shared Feature Extraction (1) After the input preprocessing is completed, the processor calls the feature extraction network to extract features from the left and right images respectively, and obtains the basic features of the left image and the basic features of the right image.

[0041] The left and right images share the same two-dimensional convolutional feature extraction network to ensure that the features of the left and right branches are located in the same embedding space. Preferably, a residual convolutional network similar to the PSMNet backbone can be used, with a semi-dilation setting to balance local texture extraction and receptive field expansion. To reduce complexity, spatial pyramid pooling modules can be omitted, and high-level features at one-quarter resolution of the input image can be directly output.

[0042] In some embodiments, the feature extraction network may include multi-layer two-dimensional convolution, batch normalization, non-linear activation, and downsampling units to progressively expand the receptive field and extract multi-scale information.

[0043] The feature maps from the last three scales of the feature extraction network are concatenated to obtain a fused feature map. This introduces a richer multi-scale context and provides a unified feature foundation for constructing different forms of cost volumes.

[0044] In some embodiments, the processor not only outputs the final features at a single level, but also retains feature representations at at least two levels of different depths. For example, the processor outputs shallow features and deep features, where the shallow features contain more texture, edge, and local structural information, and the deep features contain more semantic and contextual information. Unlike existing technologies that use only a single high-level feature to construct the cost volume, this invention utilizes the differences and complementarities between features at different levels in subsequent processing, providing a foundation for the construction of a dual-branch cost volume and feature coupling.

[0045] (2) After obtaining the basic features of the left and right images, the processor performs branching processing to generate upper branch features and lower branch features.

[0046] In some embodiments, the processor performs channel transformation and grouping processing on the original features to form upper branch features suitable for group correlation modeling; at the same time, the processor performs channel compression or lightweight mapping on the original features to form lower branch features suitable for norm correlation, splicing or difference calculation.

[0047] Preferably, the upper branch features retain strong multi-channel expressive power to characterize the similarity relationship between left and right features at the grouping level; the lower branch features appropriately reduce the channel dimension to reduce the storage and computational pressure brought by subsequent cost volume construction and 3D convolution aggregation. By assigning different functional requirements to different branches, this invention avoids the contradiction caused by a single path in the prior art having to bear both rich expression and lightweight constraints.

[0048] Step 3: Construct the cost bodies for the upper and lower branches. (1) Upper branch cost body The processor constructs the cost body of the upper branch based on the features of the upper branch.

[0049] In some embodiments, the upper branch cost body is a group correlation cost body, whereby the processor divides the feature channels into multiple groups and calculates the correlation response between the corresponding groups at each candidate disparity location. Specifically, the processor calculates the correlation response between the corresponding groups for each candidate disparity index. Offset the branch features on the right image horizontally. At each pixel location, a group-by-group similarity calculation is performed with the upper branch features of the left image to obtain the upper branch cost body composed of disparity dimension, spatial dimension, and group dimension. This process can preserve the structured matching information between channel groups with low redundancy. It can be represented as:

[0050] in, Indicated in the disparity index Pixel position and the The group-related value of each group. Indicates the candidate disparity value. Represents the normalization coefficient. Indicates the left image features at pixel locations Place, No. Feature vectors in each group Indicates the right image features at pixel locations Place, No. Feature vectors in each group.

[0051] Compared to single-channel correlation maps, group correlation cost volumes can preserve finer-grained local similarity patterns. Essentially, they characterize the internal structural differences of features using multiple correlation channels without fully restoring the massive dimensionality of the original concatenated cost volume. Preferably, group correlations can take the form of inner products. Essentially, this involves performing group-level information modeling on high-dimensional features to enhance the expressive power of multi-channel matching relationships. It can be represented as:

[0052] in, Indicated in the disparity index Pixel position and the The group difference correlation value of each group. Indicates the left image features at pixel locations Place, No. Feature vectors in each group Indicates the right image features at pixel locations Place, No. Feature vectors in each group This represents the difference between the corresponding group features of the left and right images.

[0053] Spatial resolution optimization and features of group-related cost bodies Figure 1 The resolution should be one-quarter of the input image's resolution; the disparity dimension is preferably one-quarter of the maximum disparity range. Number of groups. N g It can be configured according to hardware resources, target accuracy, and real-time requirements.

[0054] Unlike the single-entity correlation in existing technologies, this invention enables the processor to control the computational scale while retaining strong distinguishing capabilities through a grouping mechanism, thereby reducing redundancy caused by high-dimensional fully connected correlation or full-channel splicing.

[0055] (2) Lower branch cost body The processor constructs the lower branch cost body based on the lower branch features (including the lower branch features of the left image and the lower branch features of the right image).

[0056] In some embodiments, the lower branch cost body is any one of norm-related cost body, spliced ​​cost body, or group difference cost body.

[0057] In an embodiment employing a norm-related cost body, the processor computes a normalized inner product of the left and right feature vectors at each candidate disparity location to obtain a similarity metric that is robust to magnitude changes. Specifically, the process involves: first, channel compression of the fused features is performed using convolutional layers, batch normalization layers, and nonlinear activation layers, reducing the high-dimensional features to a lower number of channels; then, at each disparity layer, a normalized inner product operation is performed on the compressed left and right features to obtain a compact correlation cost representation with single or low channels.

[0058] The advantages of norm-related cost volumes lie in their compact representation and low computational cost, making them suitable for providing global similarity cues with a relatively small size. Their disadvantage is that channel representations are relatively compressed, and detailed information is not as rich as that of group correlation. Therefore, this invention uses them in parallel with group correlation cost volumes to achieve complementary advantages. This can be represented as:

[0059] in, Indicated in the disparity index and pixel position The norm-related algebraic value at a given position, with the symbol ":" indicating the channel dimension output corresponding to that position. Indicates the left image features at pixel locations The feature vectors of all channels at that location. Indicates the right image features at pixel locations The feature vectors of all channels at that location. The normalized correlation coefficient is essentially a cosine similarity measure used to reduce the impact of feature magnitude variations on matching results.

[0060] In the embodiment employing a concatenation cost body, the processor concatenates the left image features with the right image features after disparity offset along the channel dimension, enabling the subsequent network to autonomously learn the matching pattern. This can be represented as:

[0061] in, Indicated in the disparity index and pixel position The splicing-related cost features at the location, Indicates the left image features at pixel locations The feature vector extracted at that location, Indicates the right image features at pixel locations The feature vector extracted at that location, This refers to a concatenation operation, which typically involves concatenating along the channel dimension to preserve the original information of the left and right features for subsequent network learning of matching relationships.

[0062] In embodiments employing a group difference classification cost body, the processor performs absolute difference, squared difference, or other distance metrics on the left and right features to explicitly reflect the degree of difference between candidate matching positions.

[0063] Overall, the lower branch cost volume has a more compact representation compared to the upper branch cost volume. The processor configures the lower branch with lightweight feature representations and low-redundancy cost construction methods, reducing computational resource consumption while preserving effective matching clues. In this way, the upper and lower branches respectively undertake the different tasks of "rich representation" and "compact representation," thus forming a dual-cost-volume collaborative modeling mechanism.

[0064] Step 4: Multi-scale Coupled Aggregation Network (1) Two-branch three-dimensional aggregation network The two constructed cost bodies are respectively fed into the upper branch 3D aggregation network and the lower branch 3D aggregation network. The aggregation network includes 3D convolutional units, normalization units, activation units, and residual connection structures, which are used to jointly extract context information along the disparity dimension and the spatial dimension.

[0065] The two aggregation networks preferably adopt a lightweight UNet-style three-dimensional hourglass structure, with each network containing an encoder and a decoder.

[0066] The encoder preferably includes three levels of downsampling units. Each downsampling unit may include: a 3D convolutional layer with a stride of 2, used to reduce the cost volume resolution and expand the receptive field; and a 3D convolutional layer with a stride of 1, used to further extract geometric and semantic features at this scale. Through layer-by-layer downsampling, the network can integrate disparity, context, and structural information over a wider spatial support range.

[0067] The decoder preferably includes three levels of upsampling units. Each level of upsampling unit may include: a 3D transposed convolutional layer with a stride of 2 for restoring resolution; and a 3D convolutional layer with a stride of 1 for refining the upsampled features. To enhance information fidelity, skip connections may also be set between the corresponding encoding and decoding scales.

[0068] In one embodiment, the processor first performs preliminary aggregation on the cost bodies of the two branches respectively to obtain the upper branch aggregation features and the lower branch aggregation features; then, a coupling module is set at at least one intermediate layer position to enable information exchange between the intermediate cost representations of the two branches.

[0069] Compared to existing single large cost volume schemes, this invention employs two parallel small 3D aggregation networks to process the two cost representations respectively. This avoids directly performing deep 3D convolutions on the large and redundant joint cost volume, and allows each branch to learn more targeted geometric encodings for its respective cost features.

[0070] (2) Multi-layer feature coupling processing To enhance the complementary effects between information from different branches and levels, the processor performs coupling operations during cost volume aggregation. Preferably, the coupling operations include performing convolutional transformations on features from lower-level or low-dimensional paths, fusing them element-wise with features from upper-level or high-dimensional paths, and then back-injecting them into the original path via residuals.

[0071] In some embodiments, the processor performs spatial convolution on the intermediate features of the lower branch to enhance the local context; then the processing result is summed with the intermediate features of the upper branch to obtain joint features containing both types of information; next, the processor performs convolutional integration on the joint features again and adds the original lower branch features to the output through residual connections to obtain coupled fused features.

[0072] In some embodiments, a coupling module is provided in the decoding stage to fully integrate the geometric information extracted from different cost volumes by the two branches. The coupling module receives geometric features from the current scale of the upper and lower branches, and preferably outputs fused geometric features consistent with the input dimension through a series of convolution operations and residual summation operations that are only perceptible in the spatial dimension.

[0073] The coupling module can be represented as:

[0074] in, This represents the fused feature tensor output by the coupling module. This represents the input feature tensor from the lower branch or lower-level path, preserving local detail information, edge information, and texture information. This represents the input feature tensor from the upper branch or higher-level path, containing semantic and contextual information. This indicates a convolution operation with a kernel size of 1×3×3, used to extract and fuse neighborhood information in the spatial dimension while maintaining the scale of the disparity dimension. Indicates to Perform a 1×3×3 convolution operation. This indicates that the result of convolutional transformation of low-level features is combined with high-level features. Element-by-element addition is performed to enable information exchange between different levels of features. This indicates a residual connection, which directly adds the original low-level features back to the output to enhance detail preservation and improve gradient propagation.

[0075] First, a spatial convolution transformation is performed on the lower branch features, and then added to the upper branch features. Next, spatial convolution is performed on the summed result, and finally, residual fusion is performed with the original lower branch features to obtain the fused result. This design has two functions: first, it uses convolution to reshape the geometric representation of the two branches in the local neighborhood; second, it preserves the residual path, avoiding excessive smoothing of effective information during the fusion process.

[0076] Coupler modules are preferably placed at multiple scales of the decoder, for example, after the first, second, and third upsampling layers. As the spatial resolution is restored step by step, coupling operations at different scales allow the network to repeatedly exchange and correct geometric information in the process from coarse to fine, which is more conducive to the parallax recovery of edges, thin rods, vehicle contours, texture repetition areas, and weak texture areas.

[0077] The above processing enables the processor to achieve synergistic utilization of shallow detail information and deep semantic information, as well as compact and rich representation information, without significantly increasing the parameter scale. Unlike the simple concatenation or addition methods in the prior art, this invention makes information exchange more targeted and learnable through a coupled module with convolutional transformation and residual path.

[0078] Step 5: Parallax Regression After completing the two-branch aggregation and coupling, the outputs of the upper and lower branches can be summed to obtain a unified aggregation cost representation. Then, disparity regression is performed to obtain a low-resolution initial disparity map.

[0079] To reduce computational load, regression can be performed at a quarter resolution first.

[0080] In some embodiments, Top-k regularization is performed on the aggregate cost volume before disparity regression, retaining only the most probable position for each pixel. k A number of candidate disparity responses are used to reduce the computational burden on subsequent operations. k The preferred value is 2, but it is not limited to this.

[0081] In some embodiments, the processor maps aggregated features into cost or probability values ​​distributed along the disparity dimension through 3D convolution and a classification head; then, it performs softmax normalization on the disparity dimension to obtain the probability distribution of each pixel at different candidate disparities; finally, the processor outputs the final disparity map using soft-argmin, expectation regression, or equivalent disparity regression.

[0082] In some embodiments, the disparity map can also be used for subsequent tasks such as 3D reconstruction, depth estimation, robot navigation, autonomous driving perception, or industrial inspection.

[0083] Step 6: Upsampling The initial low-resolution disparity map can be further upsampled to full resolution using a weighted averaging method based on "superpixel" neighborhoods. This upsampling strategy, compared to simple bilinear interpolation, better utilizes local structural similarity to maintain clear boundaries and detail continuity. The final output full-resolution disparity map can be directly used for depth restoration, 3D reconstruction, or downstream perception tasks.

[0084] Step 7: Training Methods and Loss Function When the present invention is run in training mode, the processor also performs a supervised learning process. Specifically, the processor acquires the true disparity labels and calculates the error between the predicted disparity output by the network and the true disparity to obtain the training loss. Preferably, the processor calculates the loss for intermediate scale prediction and final scale prediction separately, and then obtains the total loss by weighted summation.

[0085] In some embodiments, end-to-end supervised learning can be used for training. Preferably, supervision terms are set for both the low-resolution initial disparity map and the full-resolution final disparity map simultaneously to improve training stability and enhance the consistency of multi-stage outputs.

[0086] The total loss function consists of two smoothed L1 losses: the first part constrains the deviation between the quarter-resolution disparity prediction and the corresponding scale ground truth; the second part constrains the deviation between the full-resolution output and the original ground truth disparity. The two parts are weighted and summed to obtain the total loss.

[0087]

[0088] in, This represents the weighting coefficient of the first loss term, used to control the proportion of the quarter-resolution disparity prediction supervision term in the total loss; This represents the weighting coefficient of the second loss term, used to control the proportion of the full-resolution disparity prediction supervision term in the total loss; This represents the initial disparity prediction result at one-quarter resolution output by the network; This represents the final disparity prediction result at full resolution output by the network; This indicates the true disparity map. According to The true disparity at one-quarter resolution obtained by downsampling at the same resolution; This represents the true parallax label at the original resolution.

[0089] In terms of training strategy, pre-training can be performed using a large-scale synthetic stereo dataset, followed by fine-tuning on a real-world dataset to balance convergence stability and real-world generalization ability. The optimizer can be Adam or its equivalent variant, and the learning rate can employ piecewise decay, cosine annealing, or other conventional deep learning training strategies.

[0090] Through the training process described above, the processor enables the dual-branch cost body structure to gradually learn a representation that is more suitable for matching complex scenarios, thereby improving the final inference accuracy.

[0091] Figure 3 Specifically, the disparity map generated from the left and right images is shown using the binocular stereo matching method based on lightweight cost volume of the present invention.

[0092] In summary, the binocular stereo matching method based on lightweight cost volume proposed in this invention has the following advantages: (1) The processor does not just construct a single cost body, but constructs two-branch cost bodies with different information expression capabilities, so as to make the matching clues more complete; (2) The processor does not rely solely on a single level of features, but achieves multi-level information collaboration through coupling modules during the aggregation process, thereby improving the matching quality of edges and weak texture regions; (3) The processor reduces overall computational complexity and storage overhead while maintaining high accuracy by designing a branch as a relatively lightweight and compact expression path; (4) The processor improves robustness in occluded regions, repetitive texture regions and parallax mutation regions through dual cost body co-aggregation.

[0093] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0094] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can include direct contact between the first and second features, or contact between the first and second features through another feature between them. Furthermore, "above," "over," and "on top" of the second feature includes the first feature directly above or diagonally above the second feature, or simply indicates that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature includes the first feature directly below or diagonally below the second feature, or simply indicates that the first feature is at a lower horizontal level than the second feature.

[0095] In this invention, the terms "first," "second," "third," and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance. The term "multiple" refers to two or more unless otherwise expressly defined.

[0096] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A binocular stereo matching method based on lightweight cost volume, characterized in that, Includes the following steps: Step 1: Acquire the left and right images after camera calibration and epipolar correction, and perform preprocessing; Step 2: Input the left and right images into the feature extraction network, select features at least three scales at the end of the feature extraction network and splice them together to form upper branch features and lower branch features respectively; Step 3: Construct the upper branch cost body and the lower branch cost body based on the upper branch features and the lower branch features, respectively; Step 4: Input the upper branch cost body and the lower branch cost body into the multi-scale coupled aggregation network to obtain the coupled upper branch fused geometric features and lower branch fused geometric features; The multi-scale coupled aggregation network consists of two parallel three-dimensional aggregation networks and a coupling module located after at least one upsampling layer. Step 5: After summing or equivalently combining the upper branch fusion geometric features and the lower branch fusion geometric features, perform disparity regression to obtain a low-resolution initial disparity map; Step 6: Based on the superpixel neighborhood weighted upsampling strategy, restore the low-resolution initial disparity map to a full-resolution disparity map, which will be used as the final matching result.

2. The binocular stereo matching method according to claim 1, characterized in that, The left and right images are RGB color images, grayscale images, multispectral images, or special imaging techniques used in industrial vision.

3. The binocular stereo matching method according to claim 1, characterized in that, The preprocessing in step 1 includes normalization, scaling, random cropping, or color perturbation.

4. The binocular stereo matching method according to claim 1, characterized in that, The feature extraction network is a weighted convolutional network, including multi-layer two-dimensional convolution, batch normalization, non-linear activation, and downsampling units.

5. The binocular stereo matching method according to claim 1, characterized in that, The upper branch cost body is constructed using a group correlation matching mechanism, while the lower branch cost body is constructed using a norm correlation matching mechanism, a splicing mechanism, or a group difference mechanism.

6. The binocular stereo matching method according to claim 5, characterized in that, The upper branch cost body is constructed using a group-related matching mechanism, and is represented as follows: in, Indicated in the disparity index Pixel position and the The group-related value of each group. Indicates the candidate disparity value. Represents the normalization coefficient. Indicates the left image features at pixel locations Place, No. Feature vectors in each group Indicates the right image features at pixel locations Place, No. Feature vectors in each group.

7. The binocular stereo matching method according to claim 5, characterized in that, The lower branch cost body adopts a norm-related matching mechanism, and is expressed as follows: in, Indicated in the disparity index and pixel position The norm-related algebraic value at a given position, with the symbol ":" indicating the channel dimension output corresponding to that position. Indicates the left image features at pixel locations The feature vectors of all channels at that location. Indicates the right image features at pixel locations The feature vectors of all channels at that location. This represents the normalized correlation coefficient.

8. The binocular stereo matching method according to claim 5, characterized in that, The lower branch cost body adopts a splicing mechanism and is represented as follows: in, Indicated in the disparity index and pixel position The splicing-related cost features at the location, Indicates the left image features at pixel locations The feature vector extracted at that location, Indicates the right image features at pixel locations The feature vector extracted at that location, This indicates a concatenation operation.

9. The binocular stereo matching method according to claim 1, characterized in that, The three-dimensional aggregation network includes three-dimensional convolutional units, normalization units, activation units, and residual connection structures; the upper branch cost volume and the lower branch cost volume are respectively input into two three-dimensional aggregation networks, and a coupling module set after at least one upsampling layer enables information exchange between the intermediate cost representations of the two branches.

10. The binocular stereo matching method according to claim 1, characterized in that, The coupling module is represented as follows: in, This represents the fused feature tensor output by the coupling module. This represents the input feature tensor from the lower branch or lower-level path, preserving local detail information, edge information, and texture information. This represents the input feature tensor from the upper branch or higher-level path, containing semantic and contextual information. This represents a convolution operation with a kernel size of 1×3×3. Indicates to Perform a 1×3×3 convolution operation.

Images