A high-precision stereo matching method based on a double cross-attention mechanism

Abstract

This invention designs a high-precision stereo matching method based on a dual cross-attention mechanism. The steps are as follows: First, extract corrected left and right view features using a shared-parameter hourglass-shaped network structure to calculate the matching cost. Second, calculate the Key of the attention mechanism using the left view features as input, calculate the Query of the attention mechanism using the right view features as input, and calculate the Value of the attention mechanism using voxels concatenated by channels of the left and right view features as input. Third, calculate the weights applied to the Value using the Key and Query, and solve for the matching cost. Fourth, generate a disparity map estimate using a multi-scale cost aggregation network based on 2D convolution. Finally, use a residual correction network based on left-right consistency checks to restore the disparity map resolution to the input resolution, while refining the matching details. This invention avoids the need for pre-specifying the disparity range. When the disparity range changes, disparity estimation results for a larger disparity range can be obtained without retraining the network, improving the convenience of disparity estimation.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method and optimization strategy for acquiring depth information with adaptive parallax range, which can be used for efficient depth information acquisition under binocular input. Background Technology

[0002] In the field of stereo vision, given the disparity of a pixel, its depth z can be obtained by solving z = bf / d. Here, b is the baseline length of the binocular camera system, f is the focal length of the camera system, and d is the disparity corresponding to that pixel. Therefore, the depth information estimation task can be transformed into the disparity estimation task of the corresponding pixel. Given a pair of calibrated binocular images, in the absence of occlusion, a point in a real scene will be simultaneously projected into the images of the left and right viewpoints. The difference in the horizontal coordinates between these two corresponding pixel pairs located at different viewpoints is called the "disparity." Typically, this disparity estimation task is decomposed into four steps: 1) feature extraction; 2) cost calculation; 3) cost aggregation; and 4) disparity estimation. For traditional algorithms, these four steps are performed sequentially, and the techniques for each step can usually be applied independently.

[0003] In recent years, with the development of deep learning technology, the completion of each step in disparity estimation has been gradually replaced by neural networks, and the independence between steps has weakened. Disparity estimation tasks are increasingly being completed by an end-to-end network. In these networks, cost calculation is typically performed using two methods: feature concatenation and convolution. For the former, the inputs from the left and right viewpoints pass through the same feature extraction structure with shared parameters. The resulting features are then directly concatenated by channel and used as input to subsequent cost aggregation and disparity estimation networks. This method typically relies on 3D convolution in the cost aggregation part, thus incurring a significant computational cost. Moreover, this direct concatenation cost calculation method is less effective than the cost matrix obtained through convolution. The convolution-based cost calculation method calculates the similarity of features from the left and right viewpoints within a given disparity range through dot products, placing the results of different point pairs on different channels. Typically, the number of channels in the cost matrix under this method is equal to the manually set disparity range. For most databases, this disparity range is specified as 192 pixels.

[0004] While this correlation-based computation method can yield a better cost matrix than direct concatenation, it must be applied to a fixed disparity range. When this disparity range changes, the network structure needs to be retrained, and the training time increases significantly with the increase in disparity range. Furthermore, in some application scenarios with large disparity range variations, if the network is not specifically retrained, its performance will be greatly reduced, or even fail. For example, in the field of autonomous driving, the algorithm needs to accurately identify distant and nearby targets, and the disparity difference between these two is huge. Traditional algorithms with a fixed disparity range are computationally very difficult, and their performance deteriorates significantly with increasing disparity range.

[0005] Besides these two convolutional neural network-based disparity estimation structures, there is also a network structure based on an attention mechanism. This method treats the stereo matching task as a sequence computation task similar to natural language processing, simultaneously completing cost calculation and cost aggregation tasks through the interweaving of cross-attention and self-attention mechanisms. Although this method can also achieve disparity range adaptation, its computational cost is huge and its performance is inferior to mainstream convolutional neural network-based stereo matching algorithms. Summary of the Invention

[0006] The purpose of this invention is to address the limitations imposed on algorithm application and performance by the above-mentioned variable disparity range applications, and to design an efficient stereo matching method based on a dual cross-attention mechanism for high-precision disparity estimation under variable disparity range conditions.

[0007] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0008] An efficient stereo matching method based on a dual cross-attention mechanism is characterized by its independence from disparity range constraints in the application scenario. It can adaptively adapt to disparity requirements under the application scenario without secondary training, achieving high-precision disparity estimation. The overall network structure of the method is as follows: Figure 1 As shown, the left and right viewpoint images are processed by a feature extraction module to learn features. Then, the features from both viewpoints are processed by a dual-cross-attention matching module to obtain matching costs. Finally, a disparity map estimate is obtained through a cost aggregation module. The method includes the following steps:

[0009] Step 1: Process the input data, dividing it into training and testing sets as needed. The training set can be further divided into a training set and a validation set. The validation set contains ground truth disparity values, but these values ​​are only used for evaluation and not involved in the loss calculation during training. Write relevant data reading functions that specify the paths to the input binocular images and the ground truth disparity images, used to read the corresponding inputs according to these paths during training. Perform data augmentation operations on the input images, including vertical translation and rotation, RGB brightness transformation, Gaussian noise, or brightness-contrast transformation.

[0010] Step 2: Use funnel-shaped feature extraction network to calculate features. The specific steps are as follows:

[0011] Step 2a: Extract features from the input and compress the spatial size using 5 downsampling units, numbered l = {1, 2, 3, 4, 5}. Each downsampling unit consists of: a) a convolutional layer with a stride of 2, a kernel size of 3x3, and padding of 1; b) batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; e) batch normalization; f) Leaky ReLU with a slope of 0.1; in sequence.

[0012] Step 2b: Extract features and restore spatial dimensions using two upsampling units, numbered l = {6, 7} respectively. Each upsampling unit consists of: a) a deconvolutional layer with a stride of 2, a kernel size of 3x3, and padding of 0; b) batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; e) batch normalization; and f) Leaky ReLU with a slope of 0.1; in sequence. The outputs of downsampling unit 4 and upsampling unit 6 are concatenated as the input of upsampling unit 7.

[0013] Step 2c: The outputs of downsampling unit 3 and upsampling unit 7 are concatenated as input to the feature pyramid. The feature pyramid includes: a) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; b) three parallel average pooling layers with kernel sizes and strides of (2,2), (4,4), and (8,8), respectively; c) three parallel convolutional layers with a kernel size of 1x1 and padding of 0, following the preceding pooling layers; d) three parallel batch normalization layers, following the preceding convolutional layers; e) three parallel Leaky ReLU layers, following the preceding normalization layers; and f) three parallel interpolation layers, following the activation function layers, used to restore the spatial resolution to 1 / 8 of the original size.

[0014] Step 3: The output of the first layer of the feature pyramid is concatenated with the outputs of the last three layers, serving as the input to the cost calculation module of the dual attention mechanism. The cost calculation module is composed of a dual attention mechanism. Considering that matched points will only appear on the same kernel line in both left and right viewpoints, cost calculation only requires one-dimensional matching. Therefore, given a position in a viewpoint, we only need to perform matching calculations along the same kernel line. The specific calculation method for dual cross-attention is as follows... Figure 2 As shown, the specific steps include:

[0015] Step 3a: Using the left viewpoint features as input, learn the key through one fully connected layer; using the right viewpoint features as input, learn the query through one fully connected layer; using the cascade of left and right viewpoint features as input, learn the value through one fully connected layer. If a multi-head attention mechanism is applied, the neurons in the above fully connected layers will be divided into... Each portion is applied to On each channel, achieve - Head attention mechanism.

[0016] Step 3b: Calculate the attention weight α for the calculated key and query. h The formula is: Where Q represents query, K represents key, and T represents transpose operation; C represents the number of feature channels in a single head. f This is the total number of feature channels; This is the number of the header to which it belongs.

[0017] Step 3c: Considering the shift invariance of the attention mechanism, positional encoding information needs to be added to the above method to assist in distinguishing different matching positions. Relative positional encoding is used here because absolute positional encoding can be recovered from relative positional encoding. The specific encoding style is as follows: Figure 2As shown, both the horizontal and vertical axes represent the image width. Therefore, the attention weights described above can be expressed as: Where F L and F R W represents the features of the left and right figures respectively. Q W K，F W K，r These represent the weights between query, key-feature, and key-position, respectively. δ This represents the relative position encoding, where δ represents the difference in the x-coordinates of the key and query, and u h These are vectors learned individually for each head.

[0018] Step 3d: Since the position of the same target in the right image must appear to the left of the target's position in the left image, i.e., the x-coordinate... r <x l Therefore, the attention mechanism only needs to consider x when calculating. l The position to the left is sufficient. This mask will be applied to the attention mechanism weights, resetting weights that do not meet the constraints to None, so they will not participate in subsequent calculations.

[0019] Step 3e: Multiply the attention weights with positional encoding by their corresponding values, concatenate the results from all heads, and input them into the linear mapping layer to obtain the final output. The specific formula is: O = W O Cascade (α) h ×V h )+b O The multiplication here is matrix multiplication. Where O represents the matched output; W... O and b O V represents the learnable parameters of the output layer. h This is the Value corresponding to the h-th head.

[0020] Step 3f: The output of the cascaded left image features, right image features, and cost calculation of the double cross attention mechanism is passed through a convolutional layer and used as the input for cost aggregation, denoted by parameter C.

[0021] Step 4: The cost aggregation part adopts a multi-scale cost aggregation structure based on 2D convolution, mainly including two aggregation modes: intra-scale and inter-scale. During cost aggregation, the two modes are used alternately and the aggregation results are shared. The specific steps are as follows:

[0022] Step 4a: Perform multi-scale intra-scale aggregation on the input. Aggregation primarily employs two neighborhood locations: fixed neighborhood and adaptive neighborhood. The fixed neighborhood follows an 8-neighborhood of 1 points with a displacement of 1 from the center position, sampling a total of 9 points. The adaptive neighborhood has a variable displacement, calculated by the neural network, sampling 9 points along the kernel line. Cost aggregation between sampled points is handled by a shared parameter w. l and specific parameter m l The cost is determined jointly, where the subscript 'l' represents a different sampling point number. The formula for cost aggregation is: Where M 2 Represents the total number of sampling points. This represents the output cost matrix, where p is the position and i is the channel number. l Δp is the fixed coordinate offset, and Δp is the adaptive coordinate offset.

[0023] Step 4b: Then, perform inter-scale aggregation on the cost output from the previous step. Taking a scale of 3 as an example, when the target scale is smaller than the current scale, the cost matrix at the current scale will be downsampled through a convolutional layer with a stride of 2 until the spatial size is consistent with the spatial size of the target scale; if the target scale is larger than the current scale, it will be upsampled through interpolation until the spatial size is consistent with the spatial size of the target scale; if the target scale is equal to the current scale, no transformation is required.

[0024] Step 4c: Repeat steps 4b and 4c several times, finally performing cost aggregation through a convolutional layer. The output feature channel number of the final aggregation layer is 1, resulting in the estimated disparity map.

[0025] Step 5: The resolution restoration and detail reconstruction module takes the disparity map estimated in Step 3 as input and gradually restores the resolution by passing it through the resolution restoration module three times. The structure diagram is as follows. Figure 3 As shown, this part uses five variables—left viewpoint image, right viewpoint image, left viewpoint disparity estimate, left viewpoint image transformed to the right viewpoint image based on the left viewpoint disparity estimate, and the difference between the right viewpoint image and the transformed right viewpoint image—as input to the structure, cascaded in channels. Each resolution restoration module consists of four ResNet modules and one deconvolution with a stride of 2. After passing through a 1x1 convolutional layer, the output of each module can be restored as the disparity map residual at that resolution. Finally, the residual is added to the disparity map estimate upsampled to the same resolution to obtain the disparity map at that spatial resolution.

[0026] Step 6: Calculate the loss by comparing the disparity map and the ground truth disparity map using the loss function, and then use backpropagation to guide the update of the network model parameters. The resulting fixed network model parameters can then be used for disparity map inference.

[0027] Compared with existing binocular stereo matching methods, this invention has the following advantages:

[0028] 1. This invention designs a dual-cross attention cost calculation structure, which, in conjunction with a specially designed positional encoding and attention mask, searches for matching points along the epipolar line. This method calculates weights by combining the key learned from the left viewpoint and the query learned from the right viewpoint, and applies these weights to the feature matrices of the left and right viewspoints to obtain the matching cost. This overcomes the limitation of traditional convolution-based cost calculation methods, which are constrained by manually set disparity ranges, making the method more conveniently applicable to different disparity estimation scenarios.

[0029] 2. This invention employs a convolutional neural network as the basic structural component for feature extraction, cost aggregation, and disparity estimation. In the feature extraction stage, by cascading features at different levels and scales, feature information that simultaneously encompasses both long and short-range vision is obtained, which helps address feature extraction problems at different scales. In the cost aggregation and disparity estimation stages, matching costs are aggregated using both fixed and adaptive neighborhood sampling strategies, further improving matching accuracy and disparity reliability at the spatial level.

[0030] 3. This invention employs a residual correction and resolution restoration module designed based on the left-right consistency verification theory. This module takes the left image, right image, and disparity map estimation as inputs, and gradually restores the resolution of the disparity map to the input spatial resolution through a deconvolution structure. Simultaneously, it progressively refines the details of the disparity map, further improving the accuracy of disparity estimation. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the disparity estimation network structure proposed in this invention.

[0032] Figure 2 The diagram shows the matching structure and attention masking of the dual cross-attention mechanism proposed in this invention.

[0033] Figure 3 This is a schematic diagram of the residual correction network structure proposed in this invention.

[0034] Figure 4 (a)-(f) show the comparison between the results of this invention and existing disparity estimation algorithms on the KITTI database.

[0035] Figure 5 (a)-(g) show the comparison between the results of this invention and existing disparity estimation algorithms on the Middlebury database. Detailed Implementation

[0036] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0037] Given a corrected database image as input, the present invention includes the following steps:

[0038] Step 1: Write a data processing function tailored to the target database's data format, specifying the paths to the stereo images and the ground truth values ​​for the training set, for network training. Taking the SceneFlow database as an example, the training and test sets together contain 39,000 images. In this method, 80% of the training set is used for network training, the remaining 20% ​​is used for cross-validation during training, and the entire test set is reserved for testing purposes only, without participating in any form of network training or fine-tuning. The training set data will undergo data augmentation operations, specifically including:

[0039] Step 1a: Simultaneously perform vertical displacement and rotation transformations on the left and right images. The displacement is within ±1.5 pixels, with the specific displacement amount randomly determined. The rotation occurs within ±0.5 degrees, with the specific rotation amount randomly determined. The random numbers for the displacement and angle follow a uniform distribution within the range [0,1).

[0040] Step 1b: Perform independent brightness transformation on the three RGB channels with a 50% probability. All channel changes follow a uniform distribution within the range of (-20, 20). The transformation occurs on the value of each pixel. There is a 20% probability that different transformation values ​​will be applied to the left and right images.

[0041] Step 1c: Subsequently, the algorithm has a 50% probability of applying Gaussian white noise or a brightness / contrast transformation to the input. The Gaussian noise follows a Gaussian distribution with a mean of 0 and a variance in the range (10, 50), where the variance follows a uniform distribution in the range [0, 1). The brightness / contrast transformation applies a brightness and contrast transformation to the image within the range (-0.2, 0.2). Similarly, there is a 20% probability that different transformation values ​​will be applied to the left and right images.

[0042] Step 2: Write the network structure code for the feature extraction part. The code for the feature extraction module can be mainly broken down into the definitions of downsampling units, upsampling units, and pyramid units. Note that the layers in each unit cannot be shared in multiple applications; specific layers need to be defined separately. The left and right images share the parameters of the feature extraction part, and the final output is the extracted features of the left and right images. Feature extraction specifically includes the following steps:

[0043] Step 2a: Define 5 downsampling units, numbered l = {1, 2, 3, 4, 5}. Each downsampling unit consists of: a) a convolutional layer with a stride of 2, a kernel size of 3x3, and padding of 1; b) channel-wise batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; e) channel-wise batch normalization; f) Leaky ReLU with a slope of 0.1; in sequence. The number of input and output channels of the downsampling units, from unit 1 to unit 5, are (3, 32), (32, 64), (64, 128), (128, 256), and (256, 512), respectively.

[0044] Step 2b: Define two upsampling units, numbered l = {6, 7} respectively. Each upsampling unit consists of: a) a deconvolutional layer with stride of 2, kernel size of 3x3, and padding of 0; b) channel-wise batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with stride of 1, kernel size of 3x3, and padding of 1; e) channel-wise batch normalization; and f) Leaky ReLU with a slope of 0.1. Upsampling unit 6 uses the output of downsampling unit 5 as its input, and upsampling unit 7 uses the cascaded outputs of downsampling units 4 and 6 as its input. Therefore, the number of input and output channels of the upsampling units, from unit 6 to unit 7, are (512, 256) and (256 + 256, 256) respectively.

[0045] Step 2c: Define the feature pyramid, including: a) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; b) three parallel average pooling layers with kernel sizes and strides of (1,1), (2,2), and (4,4), respectively; c) three parallel convolutional layers with a kernel size of 1x1 and padding of 0, following the preceding pooling layers; d) three parallel channel-wise batch normalization layers, following the preceding convolutional layers; e) three parallel LeakyReLU layers with a slope of 0.1, following the preceding normalization layers; and f) three parallel interpolation layers, following the activation function layers, used to restore the spatial resolution to 1 / 8 of the original size. The outputs of downsampling unit 3 and upsampling unit 7 are concatenated as the input to the feature pyramid. Therefore, the input and output channels of each part of the feature pyramid are as follows: the first total convolutional layer (256+128,128), and the other three scales are (128,128).

[0046] Step 3: The cost calculation section uses the outputs of the feature pyramid at each of the three scales as input. For simplicity, H and W are used to represent the height and width of the feature in spatial dimensions. For different scales, H and W are the corresponding downsampling results (1 / 8, 1 / 16, or 1 / 32 of the original input image). As mentioned earlier, for the corrected image, the matching positions will only appear on the same epipolar line. Therefore, before cost calculation, the left and right viewpoint features are converted from NxCxHxW format to WxNHxC format, where N is the number of images used in each batch. For the SceneFlow dataset, the input image size is 960x540, and the training process uses a batch size of 1. Taking the 1 / 8 scale as an example, the corresponding feature size is 1x128x64x120, and the transformed size is 120x64x128. Based on this, the cost calculation is then performed. The size calculations below all use the 1 / 8 scale as an example; the same applies to other scales. The specific steps are as follows:

[0047] Step 3a: Using the left viewpoint features, right viewpoint features, and the concatenated features from the left and right viewpoints as inputs, learn the corresponding key, query, and value through three fully connected layers. The fully connected layers here use the linear function from the nn module in PyTorch. Then, following the multi-head attention mechanism, transform the key, query, and value from WxNHxC to... Where W is the feature width, with a value of 120; NH is the batch size multiplied by the height, which equals 1 x 64 = 64; C1 is the number of headers, with a value of 8; C2 is the number of channels under each header, which is 128 / 8 = 16. Therefore, the dimensions of key, query, and value are all 120x64x8x16.

[0048] Step 3b: Based on the calculated key(K) and query(Q), calculate the attention weights using the formula. Where T is the transpose operation. C represents the number of feature channels in a single head. f This is the total number of feature channels; This is the ID of the head it belongs to. The calculated attention weight size is... Where NH is the batch size multiplied by the height, and its value is 1 x 64 = 64; is the number of heads, with a value of 8; W is the feature width, with a value of 120. Therefore, the attention weight dimension is 64x8x120x120.

[0049] Step 3c: Add relative positional encoding information to the above attention weights. First, generate relative positional encoding based on the width W, with an encoding size of (2W-1)xC. Here, W is the width, equal to 120; C is the number of feature channels, equal to 128. The generation method is to generate 2W-1 values ​​evenly within the interval (-W+1, W-1), meaning one value per unit position. Then, the results of sin and cosine calculations are interspersed and placed at the corresponding positions. Finally, the attention encoding matrix is ​​converted to a form of size WxWxC, i.e., 120x120x128. Using this as input, and employing a method similar to step 3a, with the positional encoding as input, qr and kr are obtained, and these are used with the key and query respectively to calculate the attention weights with positional encoding. Finally, the three attention weights are added together to obtain the attention weight α with all encoded information. h The dimensions are 64x8x120x120.

[0050] Step 3d: Since the position of the same target in the right image must appear to the left of the target's position in the left image, i.e., the x-coordinate... r <x l Therefore, the attention mechanism only needs to consider x when calculating. l The position to the left is sufficient. Based on this principle, the attention weight α is... h The positions that do not meet the condition are set to None. Specifically, an upper triangular matrix is ​​generated, and the α values ​​corresponding to positions with a value of 1 are set to None. h All channel values ​​are set to None and will not participate in subsequent calculations. The generated mask size is WxW, i.e., 120x120.

[0051] Step 3e: Finally, multiply the attention weights with positional encoding by the corresponding values ​​to obtain the matching results. First, change the dimension of the value from... become α h Dimensions from become Then, matrix multiplication is performed on the two to obtain the output from the dimension. The transformation is performed to dimension WxNHxC. Then, the multi-head value is learned through a fully connected layer, and the final output size is WxNHxC, which is 120x64x128.

[0052] Step 3f: After converting the matching result to an NxCxHxW size, concatenate it with the features from the left and right viewpoints to obtain an Nx3CxHxW matrix. This matrix is ​​then fed into a convolutional layer with a kernel size of 1x1 and a stride of 1. The matrix is ​​integrated according to channel information, ultimately yielding an output of size NxCxHxW, which serves as the matching cost. Specifically, the output size is 1x128x64x120.

[0053] Step 4: Using the output from Step 3 as input to the cost aggregation part, the final disparity map estimate is obtained by alternating between intra-scale and inter-scale aggregation calculations. The specific steps are as follows:

[0054] Step 4a: First, perform intra-scale aggregation on the input. The aggregation formula is as follows: Where, p l Δp and Δp represent the fixed displacement and the learnable displacement, respectively, and the definitions of other parameters are the same as described above. Here, a three-layer convolutional architecture is used, with kernel sizes of 1x1, 3x3, and 1x1. The second layer is a deformable convolution. Before and after the convolution calculation, the spatial dimensions of the matrix remain unchanged, at 1 / 8, 1 / 16, and 1 / 32, respectively. Except for the first group, the number of channels decreases by half for each aggregation group, that is, the number of input and output channels is (128,128), (128,64), (64,32), (32,16), (16,8), and (8,4) at all three scales.

[0055] Step 4b: Perform inter-scale aggregation on the output of Step 4a. Based on the input-output scale relationship, the following cases can be considered: From scale 1 (1 / 8) to scale 2 (1 / 16), and from scale 2 to scale 3 (1 / 32), a convolutional layer with a kernel size of 3x3 and a stride of 2 is used for transition; from scale 1 to scale 3, two convolutional layers with a kernel size of 3x3 and a stride of 2 are used for transition; from scale 2 to scale 1, from scale 3 to scale 1, and from scale 3 to scale 2, bilinear interpolation is first used for upsampling, followed by a convolutional layer with a kernel size of 1x1 and a stride of 1 for transition; from scale 1 to scale 1, from scale 2 to scale 2, and from scale 3 to scale 3, no changes are made. Then, the three mapping results at this scale are added by channel and normalized.

[0056] Step 4c: Repeat steps 4b and 4c a total of 6 times, then feed the result into a single convolutional layer with a kernel size of 1x1 and a stride of 1 to complete cost aggregation. The final aggregation layer has 4 input feature channels and 1 output feature channel, and the output result is the disparity map estimate at this scale.

[0057] Step 5: The inputs to the residual correction module are: left image, right image, the result of warping the left image to the right viewpoint based on the left viewpoint disparity map, the pixel difference between the two right images, and the estimated left viewpoint disparity map. Warping means moving the corresponding RGB values ​​to their disparity-shifted positions while keeping the ordinate unchanged. The occlusion region value is 0. The five inputs are concatenated by channel; for example, the total number of channels for an RGB image is 3+3+3+3+1=13. This is then fed into a convolutional layer with a kernel size of 3x3, a stride of 1, and padding of 1, resulting in 64 output channels. Afterward, the residual correction unit restores the resolution. Each residual correction unit consists of four ResNet modules and one deconvolution with a stride of 2. The ResNet modules use default parameters, and the deconvolution kernel size is 3x3. All convolutional layers output 64 channels. The output of the unit is fed into a convolutional layer with a kernel size of 1x1 and a stride of 1 to obtain the disparity map estimation residual at that resolution. The resolutions are 1 / 4, 1 / 2, and 1. The disparity map estimation residuals at different resolutions are added to the disparity map upsampled to the same resolution by bilinear interpolation to obtain the residual-corrected disparity map estimate.

[0058] Step 6: Write the loss function. This invention employs a smoothed L1 loss, which is applied to disparity maps at all scales. When calculating the loss, the ground truth disparity map is downsampled to the same spatial size as the estimated disparity map. Considering the importance and number of pixels at different spatial sizes, this invention uses a weighted summation method for the loss at each scale. The specific weights are: 1 (disparity estimation network, 1 / 8 size), 0.2 (disparity estimation network, 1 / 16 size), 0.2 (disparity estimation network, 1 / 32 size), 1 (residual correction network, 1 size), 0.5 (residual correction network, 1 / 2 size), 0.5 (residual correction network, 1 / 4 size). The training epochs are set to 40, trained using four NVIDIA P100 GPUs. Parameter optimization uses the weighted Adam method, and the learning rate is halved whenever the loss converges. After each epoch of training, the network parameters are tested and used for cross-validation. The final network parameters are then used as the fixed model.

[0059] The technical effects of this invention will be further explained below with reference to experimental simulation:

[0060] 1. Simulation conditions

[0061] The PyTorch framework was used, with four NVIDIA P100 graphics cards. The pre-training database was SceneFlow, with 80% of the training data reserved for training and the remaining 20% ​​for validation. The network was trained with a learning rate of 0.0001, employing the AdamW parameter update strategy, and a batch size of 2. Training lasted for 40 epochs, with the learning rate halved at epochs 20 and 30. Fine-tuning was performed on the KITTI dataset, with 85% of the training data used for training and the remaining 15% for validation. Training was first performed on the KITTI mixed dataset for 200 epochs. Fine-tuning was then performed on the KITTI2012 and KITTI2015 datasets, with 100 epochs trained on each dataset. The learning rate started at 0.0001, and an exponential update strategy was used each epoch.

[0062] 2. Simulation Content and Result Analysis

[0063] Simulation 1 tested the invention on the test set of the KITTI2015 database and compared it with other related studies. Experimental results are as follows: Figure 4 As shown in the diagram. Column (a) represents the left viewpoint image input, columns (b)-(e) are four methods used for comparison, and column (f) shows the result of this method. The first and third rows are the input image or disparity map estimation, and the second and fourth rows are the error map. Darker colors indicate more accurate regions, while lighter colors indicate greater errors. Pure black areas have no ground truth. It can be seen that within the area enclosed by the white rectangle, this method provides significantly more accurate estimation results.

[0064] Simulation 2 tested the invention on the Middlebury2014 dataset and compared it with other related studies. Note that the model used here was trained on the SceneFlow dataset, without any training or optimization on the Middlebury dataset, thus demonstrating the adaptability of the method. Experimental results are as follows... Figure 5 As shown in the diagram, column (a) represents the left viewpoint image input, columns (b)-(f) represent the methods used for comparison, and column (g) represents the results of this method. The first and second rows show two input scenes. It can be seen that within the areas enclosed by white or black rectangles, this method yields significantly more accurate results.

[0065] Simulation 3 tested the performance of the present invention under different disparity ranges on the SceneFlow dataset and compared it with relevant research results. Except for the present invention and the STTR method, other methods (if possible) were retrained under a maximum disparity of 480 to ensure fairness in the comparison. It can be seen that the present invention effectively improves accuracy, and the improvement is more significant under large disparity ranges. Here, 3-pixel error represents the percentage of pixels with an error greater than 3 pixels. The present method has an error of 1.53%, nearly double that of other algorithms. EPE represents End-point error, which is the average pixel error. The present method has an error of 0.66, also significantly higher than other algorithms. The experimental results are shown in Table 1. Table 1 compares the performance of the present invention with existing disparity estimation algorithms under different disparity ranges on the SceneFlow database.

[0066]

Claims

1. A high-precision stereo matching method based on a dual cross-attention mechanism, characterized in that: Includes the following steps: Step 1: Process the input data, dividing it into training and testing sets as needed. The training set is further divided into a training set and a validation set. The validation set contains ground truth disparity values, but these values ​​are only used for evaluation and do not participate in the loss calculation during training. Write relevant data reading functions that indicate the paths of the input binocular images and the ground truth disparity images, which are used to read the corresponding inputs according to these paths during training. Perform data augmentation operations on the input images, including vertical displacement and rotation, RGB brightness transformation, Gaussian noise, or brightness-contrast transformation. Step 2: Use funnel-shaped feature extraction network to calculate features; Step 3: The output of the first layer of the feature pyramid is concatenated with the output of the last three layers as the input to the cost calculation module of the dual attention mechanism; the cost calculation module is composed of the dual attention mechanism; the matched points will only appear on the same kernel line in the left and right viewpoints, and the cost calculation only needs to perform one-dimensional matching; given the position in a viewpoint, matching calculation is performed along the same kernel line; Step 4: The cost aggregation part adopts a multi-scale cost aggregation structure based on 2D convolution, including two aggregation modes: intra-scale and inter-scale. During the cost aggregation process, the two modes are used alternately and the aggregation results are shared. Step 4a: Perform multi-scale intra-scale aggregation on the input; the aggregation uses two neighborhood locations: fixed neighborhood and adaptive neighborhood; the fixed neighborhood follows an 8-neighborhood of 1 points with a displacement of 1 starting from the center position, for a total of 9 points sampled; the displacement of the adaptive neighborhood is not fixed and is calculated by the neural network, sampling 9 points along the kernel line; the cost aggregation between sampled points is handled by shared parameters. and specific parameters The decision is made jointly, where the subscript 'l' represents different sampling point numbers; the formula for cost aggregation is: ,in Represents the total number of sampling points. Represents the output cost matrix. Let i be the location, and i be the channel number. For fixed coordinate offset, This is an adaptive coordinate offset. Step 4b: Then, scale aggregation is performed on the cost output from the previous step; when the target scale is smaller than the current scale, the cost matrix at the current scale is downsampled through a convolutional layer with a stride of 2 until the spatial size is consistent with the spatial size of the target scale; if the target scale is larger than the current scale, it is upsampled through interpolation until the spatial size is consistent with the spatial size of the target scale; if the target scale is equal to the current scale, no transformation is required. Step 4c: Alternate between steps 4b and 4c several times, and finally complete the cost aggregation through a convolutional layer; the output feature channel number of the last aggregation layer is 1, and the estimated disparity map is obtained; Step 5: The resolution restoration and detail reconstruction module takes the estimated disparity map as input and gradually restores the resolution by passing it through three resolution restoration modules. The module takes five variables as input, cascaded in channels: left viewpoint image, right viewpoint image, left viewpoint disparity estimate, left viewpoint image transformed to the right viewpoint image according to the left viewpoint disparity estimate, and the difference between the right viewpoint image and the transformed right viewpoint image. Each resolution restoration module consists of four ResNet modules and one deconvolution with a stride of 2. The output of each module is passed through a 1x1 convolutional layer to restore the disparity map residual at that resolution. Finally, the residual is added to the disparity map estimate upsampled to the same resolution to obtain the disparity map at that resolution. Step 6: Calculate the loss by comparing the disparity map with the ground truth disparity map using the loss function, and use backpropagation to guide the update of the network model parameters; finally, the fixed network model parameters are obtained for disparity map inference.

2. The high-precision stereo matching method based on a dual cross-attention mechanism according to claim 1, characterized in that: Step 2 further includes: writing the network structure code for the feature extraction part; the code of the feature extraction module is broken down into the definition of downsampling units, upsampling units and pyramid units; note that the layers in each unit cannot be shared when applied multiple times, and specific layers need to be defined multiple times; the left and right images share the parameters of the feature extraction part, and the final output is the extracted features of the left and right images.

3. A high-precision stereo matching method based on a dual cross-attention mechanism according to claim 1 or 2, characterized in that: Step 6 further includes: the loss function is written using smoothed L1 loss, which is applied to disparity maps at all scales; when calculating the loss, the ground truth of the disparity map is downsampled to the same spatial size as the estimated disparity map; the importance and number of pixels at different spatial sizes are weighted and summed for the loss at each scale.

4. A high-precision stereo matching method based on a dual cross-attention mechanism according to claim 1 or 2, characterized in that: Step 1 further includes: Step 1a: Simultaneously perform vertical displacement and rotation transformations on both the left and right figures; wherein, the displacement is in The displacement occurs within a pixel range, with the specific amount of displacement determined randomly; rotation occurs within... The specific rotation amount is randomly determined within the range of degrees; the random numbers for the above displacements and angles follow a uniform distribution within the range of [0,1). Step 1b: Perform brightness transformation independently on the three RGB channels with a 50% probability. All channel changes follow a uniform distribution within the range of (-20, 20). The transformation occurs on the value of each pixel. There is a 20% probability that different transformation values ​​will be applied to the left and right images. Step 1c: Subsequently, the algorithm has a 50% probability of choosing to apply Gaussian white noise or brightness / contrast transformation to the input; where Gaussian noise follows a Gaussian distribution with a mean of 0 and a variance in the range of (10, 50), and the variance is chosen to follow a uniform distribution in the range of [0, 1); brightness / contrast transformation is to perform brightness and contrast transformation on the image brightness in the range of (-0.2, 0.2); similarly, there is a 20% probability that different transformation values ​​are applied to the left and right images.

5. A high-precision stereo matching method based on a dual cross-attention mechanism according to claim 1 or 2, characterized in that: Step 2 further includes: Step 2a: Extract features from the input and compress the spatial size using 5 downsampling units, numbered l = {1, 2, 3, 4, 5} respectively; each downsampling unit consists of: a) a convolutional layer with a stride of 2, a kernel size of 3x3, and padding of 1; b) batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; e) batch normalization; f) Leaky ReLU with a slope of 0.1; in sequence. Step 2b: Extract features and restore spatial dimensions using two upsampling units, numbered l = {6, 7} respectively; each upsampling unit consists of: a) a deconvolutional layer with stride of 2, kernel size of 3x3, and padding of 0; b) batch normalization; c) Leaky ReLU with a slope of 0.1; d) a convolutional layer with stride of 1, kernel size of 3x3, and padding of 1; e) batch normalization; and f) Leaky ReLU with a slope of 0.1; in sequence. Step 2c: Define the feature pyramid, including: a) a convolutional layer with a stride of 1, a kernel size of 3x3, and padding of 1; b) three parallel average pooling layers with kernel sizes and strides of (1, 1), (2, 2), and (4, 4), respectively; c) three parallel convolutional layers with a kernel size of 1x1 and padding of 0, following the preceding pooling layers; d) three parallel channel-wise batch normalization layers, following the preceding convolutional layers; e) three parallel LeakyReLU layers with a slope of 0.1, following the preceding normalization layers; and f) three parallel interpolation layers, following the activation function layers, used to restore the spatial resolution to 1 / 8 of the original size.

6. A high-precision stereo matching method based on a dual cross-attention mechanism according to claim 1 or 2, characterized in that: Step 3 further includes: Step 3a: Using the left viewpoint features as input, learn the key through one fully connected layer; using the right viewpoint features as input, learn the query through one fully connected layer; using the cascade of left and right viewpoint features as input, learn the value through one fully connected layer; if a multi-head attention mechanism is applied, the neurons in the fully connected layer will be divided into... Each portion is applied to 1 / On each channel, achieve - Head attention mechanism; Step 3b: Calculate the attention weights for the calculated key and query. The formula is: Where Q represents query, K represents key, and T represents transpose operation; The number of feature channels representing a single head. This is the total number of feature channels; This is the number of the header to which it belongs; Step 3c: The attention mechanism has shift invariance, requiring the addition of positional encoding information to aid in distinguishing different matching positions; a relative positional encoding form is used, with the absolute positional encoding recovered from the relative positional encoding; where both the horizontal and vertical coordinates represent the image width; the attention weights are expressed as: ,in and These represent the characteristics of the left and right images, respectively. , , These represent the weights between query, key-feature, and key-position, respectively. Represents relative position encoding, where This represents the difference in the x-coordinates of the key and query. These are vectors learned individually for each head. Step 3d: Since the position of the same target in the right image must appear to the left of the target's position in the left image, i.e., on the horizontal axis... Therefore, when calculating the attention mechanism, only the following needs to be considered. The position to the left; applied to the attention mechanism weights, resets weights that do not meet the constraints to None, and they do not participate in subsequent calculations; Step 3e: Multiply the attention weights with positional encoding by their corresponding values, concatenate the results from all heads, and input them into the linear mapping layer to obtain the final output; the specific formula is as follows: The multiplication is matrix multiplication; where, For matching output; and These are the learnable parameters for the output layer; The value corresponding to the h-th header; Step 3f: The output of the cascaded left and right image features and the cost calculation of the double cross-attention mechanism is passed through a convolutional layer and used as the input for cost aggregation, with parameters... express.

Images