Self-adaptive binocular stereo matching method based on global four-dimensional graph volume attention
By using a global four-dimensional graph volume attention framework, multi-layer four-dimensional cost volume information is adaptively integrated, which solves the problem of insufficient robustness of traditional stereo matching methods in high dynamic environments, improves the accuracy and reliability of disparity estimation, and is applicable to scenarios such as three-dimensional reconstruction and autonomous driving.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHONGQING UNIVERSITY OF SCIENCE AND TECHNOLOGY
- Filing Date
- 2026-01-16
- Publication Date
- 2026-04-17
AI Technical Summary
Traditional stereo matching methods are not robust enough in high dynamic environments, especially in areas with large parallax variations, where they are prone to ghosting and parallax holes, affecting the accuracy and reliability of applications such as autonomous driving.
A global four-dimensional graph volume attention framework is constructed. Multi-scale four-dimensional cost volume is formed by multi-scale pyramid feature extraction and feature concatenation. Graph neural network nodes are used for cross-layer and intra-layer interaction aggregation. Multilayer perceptron is used to calculate channel-level attention weights for weighted fusion to achieve adaptive information integration.
It improves the robustness of disparity estimation in occluded, low-texture, and large disparity variation regions, enhances the model's adaptability and interpretability in complex scenes, and improves the accuracy and robustness of stereo matching.
Smart Images
Figure CN121883552A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning technology, and in particular to an adaptive binocular stereo matching method based on global four-dimensional graph volume attention. Background Technology
[0002] Stereo matching is a core task in computer vision, aiming to obtain scene depth information through pixel-level correspondences between binocular images. This technology is widely used in fields such as autonomous driving, 3D reconstruction, and robot navigation. With the development of deep learning, learning-based stereo matching methods have gradually become mainstream, among which cost aggregation is crucial for disparity estimation accuracy.
[0003] Traditional aggregation methods are mostly based on 3D cost volumes, while GC-Net introduced a 4D cost volume by introducing a disparity dimension, thus improving matching performance. Subsequently, methods such as PSMNet and SMDNet used multi-layer 4D cost volumes for fixed-structure aggregation. Their main drawback is that when dealing with complex scenes with large disparity changes, the receptive field of 3D convolution is relatively fixed, and the aggregation weights between layers cannot be dynamically adjusted. This makes it difficult for the model to simultaneously capture the details of nearby objects with large disparity and maintain semantic consistency with distant objects with small disparity. This leads to "ghosting" at object edges and in regions of disparity jumps, or disparity holes due to feature interference, limiting the robustness of stereo matching methods in highly dynamic environments such as autonomous driving. Summary of the Invention
[0004] The purpose of this invention is to provide an adaptive binocular stereo matching method based on global four-dimensional graph volume attention, which aims to solve the problem that traditional stereo matching methods based on PSMNet limit the robustness of stereo matching methods in high-dynamic environments such as autonomous driving.
[0005] To achieve the above objectives, this invention provides an adaptive binocular stereo matching method based on global four-dimensional graph volume attention, comprising the following steps: Based on the existing stereo matching framework, a multi-scale four-dimensional cost volume is constructed by multi-scale pyramid feature extraction and feature splicing. Using the multi-scale four-dimensional cost volume channel as graph neural network nodes, a graph structure is constructed that simultaneously includes cross-scale connections between front and rear layers and intra-layer scale connections. The nodes in the graph structure are interactively aggregated and updated to obtain node embedding features with global context information; The embedded features of the graph nodes are used to calculate channel-level attention weights through a multilayer perceptron, and the multi-scale four-dimensional cost volume is then weighted and fused according to the weights.
[0006] The graph structure is a channel-level graph structure, where nodes correspond to the channel dimensions of the multi-scale four-dimensional cost volume, and edges include both intra-layer edges and inter-layer edges.
[0007] The interactive aggregation update includes self-aggregation and cross-layer interaction.
[0008] The self-aggregation mechanism models the information relationship between nodes within a layer through a graph attention mechanism. The attention weight is calculated as the dot product of the value and the key. The cross-layer interaction mechanism performs forward propagation and backward propagation simultaneously to construct inter-layer attention relationships with the previous layer key and the subsequent layer value, and the subsequent layer key and the previous layer value, respectively.
[0009] The multilayer perceptron comprises three layers and employs a layer-by-layer compression structure for channel weight regression, with the output being scalar attention weights.
[0010] This invention presents an adaptive binocular stereo matching method based on global four-dimensional graph volume attention. Building upon existing stereo matching frameworks, it constructs a multi-scale four-dimensional cost body through multi-scale pyramid feature extraction and feature concatenation. Using the channels of this multi-scale four-dimensional cost body as nodes in a graph neural network, it constructs a graph structure that simultaneously includes cross-scale connections between layers and intra-layer scale connections. The nodes in this graph structure are interactively aggregated and updated to obtain node embedding features with global contextual information. These node embedding features are then used to calculate channel-level attention weights via a multilayer perceptron, and the multi-scale four-dimensional cost body is weighted and fused accordingly. This method adaptively integrates multi-layer four-dimensional cost body information from a global perspective. Inspired by the global information transmission characteristics of graph structures, a four-dimensional graph volume attention framework is constructed. This framework maps the channels of the multi-layer cost body to graph nodes and achieves global correlation modeling and adaptive fusion of multi-scale features through cross-layer interaction and intra-layer self-aggregation mechanisms on the graph. This method breaks through the limitations of traditional local convolution aggregation, and can model the dependencies between layers and channels in the global scope. It achieves adaptive weighted fusion of multi-layer four-dimensional cost volumes, significantly improves the robustness of disparity estimation in occlusion, low texture and large disparity variation regions, and enhances the adaptability and interpretability of the model in the process of integrating multi-layer cost volume information. It can be widely used in high-precision stereo vision systems in scenarios such as 3D reconstruction, autonomous driving, and robot navigation. Attached Figure Description
[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.
[0012] Figure 1This is a flowchart of the overall system framework of the present invention, showing the end-to-end process from left and right image input, pyramid feature extraction, multi-layer four-dimensional cost volume construction, global four-dimensional graph volume attention module processing, cost volume weighted fusion, to final disparity regression.
[0013] Figure 2 This is a schematic diagram of the global four-dimensional graph volume attention module structure in this invention, showing the construction methods of "self-aggregating subgraphs" and "cross-interaction subgraphs", as well as the operation flow of graph information interaction and attention weight regression (MLP+softmax) in this module.
[0014] Figure 3 The diagram shows the results of the ablation experiment, which demonstrates the performance of multiple components in the global four-dimensional graph volume attention proposed in this invention on the KITTI2015 and Middlebury Stereo Evaluation datasets.
[0015] Figure 4 This is a schematic diagram illustrating the performance of the present invention on the KITTI2012 dataset.
[0016] Figure 5 This is a schematic diagram illustrating the performance of the present invention on the KITTI2015 dataset.
[0017] Figure 6 This is a flowchart of an adaptive binocular stereo matching method based on global four-dimensional graph volume attention. Detailed Implementation
[0018] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.
[0019] Please see Figures 1-6 The adaptive binocular stereo matching method based on global four-dimensional graph volume attention provided by this invention includes the following steps: S1 constructs a multi-scale four-dimensional cost volume based on the existing stereo matching framework by using multi-scale pyramid feature extraction and feature splicing. Input left and right images and Utilizing deep convolutional feature extraction networks with shared parameters (such as ResNet containing spatial pyramid pooling) to extract... Feature maps at various scales .
[0020] Subsequently, at each scale The four-dimensional cost volume is obtained by constructing the feature concatenation. Used for subsequent graph structure construction: S2 uses the multi-scale four-dimensional cost volume channel as graph neural network nodes to construct a graph structure that simultaneously includes cross-scale connections between front and rear layers and intra-layer scale connections. In an embodiment of the present invention, a figure It is a set of nodes Sum of edges This structure possesses significant global representation capabilities. Unlike traditional spatial graph methods, this invention treats the channel dimension of the cost volume as nodes in a graph neural network to preserve the semantic and disparity structure information carried by each channel. Two distinct subgraphs are constructed: one for the front layer and one for the back layer. Volume of the front subgraph and for use in the back layer Volume of the back subgraph Among them, node features Each is composed of compression operations Post-cost body characteristics Initialization. Inner connection edge set. Connecting different channel nodes at the same scale within the same layer. Set of cross-layer connection edges. This connects nodes of the same passage at different scales between adjacent layers. The specific steps are as follows: Node mapping: mapping the cost body Each channel Consider as a node ,common Each node constitutes a graph node set. The node features are initialized using dimension-preserving pooling. By compressing the spatial and disparity dimensions of the four-dimensional cost volume, the global statistical attributes and semantic activation strength of each feature channel across the entire scene are extracted. This significantly reduces the computational complexity of graph operations while capturing long-distance semantic relationships between channels. This initialization process does not alter the original cost volume's data structure; it serves only as a computational guide signal. Subsequent attention weight generation mechanisms apply this signal to the original high-resolution cost volume, ensuring global context modeling is achieved without sacrificing disparity accuracy.
[0021] Feature compression: The embedding vector for each channel node is initialized using dimension-preserving pooling. in, This represents the maximum value of the parallax. and This indicates the height and width of the convolutional layer.
[0022] Graph structure definition: For the front layer and back layer Construct the front subgraph respectively and Subgraph , used for self-aggregation.
[0023] Connect the corresponding channel nodes between two subgraphs to construct a set of cross-graph interaction edges. This enables channel dependency modeling between the front and back diagrams.
[0024] S3 performs interactive aggregation and update on the nodes in the graph structure to obtain node embedding features with global context information; In this embodiment of the invention, the interactive aggregation update includes self-aggregation and cross-layer interaction. The self-aggregation mechanism models the information relationship between nodes within a layer through a graph attention mechanism. The attention weight is calculated in the form of the dot product of the value and the key. The cross-layer interaction mechanism performs forward propagation and backward propagation simultaneously, constructing inter-layer attention relationships with the previous layer key and the subsequent layer value, and the subsequent layer key and the previous layer value, respectively.
[0025] Self-aggregation mechanism In the figure and In this paper, for each node, an interactive update method based on graph structure and attention mechanism is designed. The query, key, and value are defined. , and ,in, : Calculate the attention coefficient: in, Indicates the length of the relative node feature sequence. Represents the number of nodes.
[0026] Finally, update the node features after self-aggregation. It was calculated that: Cross-layer interaction The graph information interaction includes a two-way cross-layer interaction mechanism guided by physical semantics: Forward propagation (f2b) transmits shallow cost volume information containing rich high-frequency spatial details to deep layers, and uses clear geometric boundary features to sharpen and compensate for the blurred features in deep layers, focusing on improving the accuracy of disparity prediction at the boundaries of object occlusion areas. Backpropagation (b2f) feeds back deep cost volume information with strong global contextual semantics to shallow layers, providing consistency constraints to guide local matching. This effectively suppresses matching ambiguity in low-texture and repetitive texture regions, enhancing the model's perception capability in large-scale environments. Cross-graph information propagation: Forward propagation (f2b): Backpropagation (b2f): Finally, after cross-interactions, node features and The calculation method is as follows: S4 calculates channel-level attention weights for the node embedding features using a multilayer perceptron, and then performs weighted fusion on the multi-scale four-dimensional cost volume according to the weights.
[0027] In this embodiment of the invention, after aggregating information from different aspects through self-aggregation or cross-interaction, it is necessary to infer and update the differential features along the channel dimension to generate a new state. A multilayer perceptron (MLP) is used to adaptively learn and infer and update the information for each channel, thereby generating updated node features. and : For the last layer of the Multilayer Perceptron (MLP), this invention uses a softmax function to normalize the output to balance the data distribution. The output here is the attention weights after multilayer graph interaction and feature updates. The MLP described above employs a multilayer structure, compressing dimensions layer by layer to enhance nonlinear fitting capabilities, as shown below (from the disparity dimension D to the scalar): Input layer: Size D Hidden layers: 48→12→6 Output layer: 1 Subsequently, cost characteristics Calculated as the original forward cost feature With backward cost characteristics Weighted sum: in, This represents the softmax operation. This represents the dot product operation along the channel dimension. and These represent the last layer node features of the front and back images, respectively.
[0028] Weighted fusion and disparity regression (e.g.) Figure 2 (as shown) The fusion cost volume is represented by the final attention coefficients along the channel node features at each scale. Corresponding node features Weighted average: Finally, the disparity is output using softmax weighted regression: in, This is the summation index variable in softmax normalization, used to iterate through all disparity hypotheses.
[0029] Using L1 light loss (smooth) Supervised training will be conducted. in, Indicates the total number of valid pixels. To correspond to the true parallax.
[0030] Compared with existing technologies, this invention has the following advantages: By constructing a channel-level graph structure, it achieves adaptive modeling of the contribution of multi-layer four-dimensional cost volumes, overcoming the limitations of fixed aggregation structures in handling large disparity variations. The introduced bidirectional cross-layer interaction mechanism can collaboratively utilize spatial details and global semantics, significantly improving the disparity estimation accuracy and model interpretability in highly challenging scenarios such as occlusion boundaries and low-texture regions.
[0031] The above-disclosed embodiments are merely preferred embodiments of the adaptive binocular stereo matching method based on global four-dimensional graph volume attention in this application, and should not be construed as limiting the scope of this application. Those skilled in the art can understand that all or part of the processes for implementing the above embodiments, and equivalent variations made in accordance with the claims of this application, still fall within the scope of this application.
Claims
1. An adaptive binocular stereo matching method based on global four-dimensional graph volume attention, characterized in that, Includes the following steps: Based on the existing stereo matching framework, a multi-scale four-dimensional cost volume is constructed by multi-scale pyramid feature extraction and feature splicing. Using the multi-scale four-dimensional cost volume channel as graph neural network nodes, a graph structure is constructed that simultaneously includes cross-scale connections between front and rear layers and intra-layer scale connections. The nodes in the graph structure are interactively aggregated and updated to obtain node embedding features with global context information; The node embedding features are used to calculate channel-level attention weights using a multilayer perceptron, and the multi-scale four-dimensional cost volume is then weighted and fused according to the weights.
2. The adaptive binocular stereo matching method based on global four-dimensional graph volume attention as described in claim 1, characterized in that, The graph structure is a channel-level graph structure, where nodes correspond to the channel dimensions of the multi-scale four-dimensional cost volume, and edges include both intra-layer edges and inter-layer edges.
3. The adaptive binocular stereo matching method based on global four-dimensional graph volume attention as described in claim 1, characterized in that, The interactive aggregation update includes self-aggregation and cross-layer interaction.
4. The adaptive binocular stereo matching method based on global four-dimensional graph volume attention as described in claim 1, characterized in that, The self-aggregation mechanism models the information relationship between nodes through a graph attention mechanism. The attention weight is calculated in the form of the dot product of the value and the key. The cross-layer interaction mechanism performs forward propagation and backward propagation simultaneously, constructing attention relationships with the previous layer key and the next layer value, and the next layer key and the previous layer value, respectively.
5. The adaptive binocular stereo matching method based on global four-dimensional graph volume attention as described in claim 1, characterized in that, The graph node embedding features are used to calculate channel-level graph attention weights through a multilayer perceptron, which consists of three layers and uses a layer-by-layer compression structure for channel weight regression, with the output being scalar attention weights.