Transform and lightweight 3D convolution fused structural member target parallax estimation method

Abstract

The invention discloses a parallax estimation method of a structural member target fusing Transform and lightweight 3D convolution, and belongs to the field of computer machine vision. Collecting left and right view images of the structural member scene, and respectively extracting two-dimensional basic features; an improved Transform global information extraction module is utilized to capture a long-distance dependency relationship and local detail features, and global features are obtained; the two-dimensional basic features and the global features are fused, and a weighted 4D cost body is constructed in combination with YOLOv5s target detection prior and a grouping distance mapping strategy; a lightweight deformable 3D convolution cost aggregation model is adopted to optimize the cost body; and finally, a continuous disparity map with sub-pixel precision is generated based on probability distribution regression, and accurate three-dimensional positioning of the structural member target is realized. The method has excellent global information capturing capability and local detail keeping capability in a complex environment, and is outstanding in parallax estimation precision and real-time performance.

Description

TECHNICAL FIELD

[0001] The application relates to a structure target parallax estimation method fusing a Transformer and a lightweight 3D convolution, and belongs to the field of computer machine vision. BACKGROUND

[0002] With the continuous expansion of industrial construction scale and the continuous development of automation technology, the structure parts and peripheral equipment (such as industrial boilers, ships, bridges, etc.) of major equipment are widely used in the engineering field. In the intelligent construction scene of welding, spraying and other large-scale structure parts, high accuracy and efficiency are required for target recognition and three-dimensional positioning. However, the construction site environment is complex, often accompanied by strong light, object occlusion, surface reflection and component morphology, etc. Challenges. Traditional manual detection methods have been difficult to meet the actual needs of automated construction in terms of accuracy, stability and efficiency.

[0003] At present, although the target detection and stereo matching technology based on deep learning has made significant progress, it still faces many bottlenecks in the above-mentioned complex industrial scene:

[0004] 1. Lack of global information: Traditional stereo matching algorithms and early convolutional neural networks (CNN) have limited receptive fields when extracting features, making it difficult to effectively capture long-range dependencies and global structure information in images, resulting in poor matching results in weak texture and repetitive texture areas.

[0005] 2. Insufficient positioning accuracy: Most methods only provide two-dimensional detection boxes, which cannot meet the demand for three-dimensional information of target space position in the construction process; and the stereo matching method based on CNN often has problems such as edge blurring and detail loss in disparity estimation, making it difficult to achieve sub-pixel level accurate positioning.

[0006] 3. Difficulty in balancing computational efficiency and accuracy: Although the model introducing 3D convolution for cost aggregation can improve accuracy, the computational load and memory consumption are huge, making it difficult to meet the real-time requirements in practical applications while ensuring high accuracy.

[0007] 4. Poor environmental adaptability: Existing methods lack robustness to complex environments (such as occlusion and light changes), and lack the use of prior knowledge of specific targets (such as structure parts), resulting in weak generalization ability in complex industrial scenes.

[0008] Therefore, a new method is needed that can achieve high-precision and high-efficiency three-dimensional positioning in complex environments to overcome the problems of insufficient global information capture, weak detail preservation ability and large computational overhead. SUMMARY

[0009] In order to solve the problem that the traditional stereo matching algorithm is difficult to accurately identify and locate the target part due to the lack of global structure information capture, the present application proposes a disparity estimation method for structure part target fusing Transformer and lightweight 3D convolution.

[0010] To solve the above technical problems, the present application provides the following technical solutions:

[0011] In the first aspect, the present application provides a disparity estimation method for structure part target fusing Transformer and lightweight 3D convolution, comprising the following steps:

[0012] Step S1: global feature extraction based on weight sharing of Transformer

[0013] S1.1, collect left view images and right view images of the structure part scene, and input them into a convolutional neural network respectively to extract two-dimensional basic features;

[0014] S1.2, input the two-dimensional basic features of the left and right views into the improved Transformer global information extraction module respectively to extract global features;

[0015] Step S2: cost volume construction fusing target priori and grouping strategy

[0016] S2.1, fuse the two-dimensional basic features extracted in step S1.1 with the global features extracted in step S1.2;

[0017] S2.2, based on the fused features, construct an initial 4D cost volume, and adopt a grouping distance mapping strategy in the construction process;

[0018] S2.3, use a YOLOv5s target detection network to generate a weight mask matrix with the same size as the left view image; multiply the initial 4D cost volume and the weight mask matrix element by element to obtain a final weighted 4D cost volume;

[0019] Step S3: cost volume aggregation based on lightweight deformable 3D convolution

[0020] S3.1, design a lightweight deformable 3D convolution cost volume aggregation model

[0021] S3.2, input the final weighted 4D cost volume into the designed lightweight deformable 3D convolution cost volume aggregation model to obtain an optimized three-dimensional cost volume;

[0022] Step S4: disparity map generation based on probability distribution regression

[0023] S4.1, apply a Softmax function to the optimized three-dimensional cost volume to obtain a disparity probability distribution;

[0024] S4.2, according to the disparity probability distribution, the expected value of each pixel is calculated, and a continuous disparity map with sub-pixel accuracy is generated.

[0025] Preferably, the improved Transformer global information extraction module comprises:

[0026] An improved multi-head self-attention mechanism is used to enhance the network's ability to capture global context;

[0027] And a gated dilated convolution feedforward network is used to improve the network's extraction effect of local detailed features;

[0028] Both of them work together to improve the overall feature expression ability of the network;

[0029] Wherein, the improved multi-head self-attention mechanism uses a depth separable convolution with a dilation rate of 2 when generating query, key and value matrices, and introduces a channel interaction mechanism in attention calculation.

[0030] In view of the problem that the traditional attention mechanism lacks effective information interaction between channels, the channel dimension of the query matrix Q and the key matrix K is spliced to capture more dimensional context information. At the same time, a 1x1 convolution layer is used to further linearly transform the channel dimension of the spliced features, so as to realize rich information interaction between Q and K in a higher dimensional space, and to enhance the global context awareness ability of the model. Subsequently, Q and K are normalized by the Softmax function, and V is calculated by multiplication to obtain the attention map.

[0031] In order to further refine the attention feature representation, an attention correction strategy is also designed, that is, the Q-K channel interaction information and the attention map are fused and corrected with the original features through dot product, so as to more accurately highlight the key feature information,

[0032] Preferably, the gated dilated convolution feedforward network inputs the input features into two branches, and each branch uses a depth separable convolution with a dilation rate of 2; the output features of one branch are subjected to a Gaussian error linear unit activation function and then matrix dot product with the output features of the other branch, so as to realize fine local feature extraction and effective information flow control.

[0033] Preferably, the grouping distance mapping strategy comprises:

[0034] The fused features are evenly divided into multiple groups;

[0035] The matching cost of each group is calculated independently under each disparity;

[0036] The matching costs of each group are aggregated to construct an initial 4D cost volume.

[0037] Preferably, the weight mask matrix is generated in the following manner:

[0038] The YOLOv5s target detection network is used for the left view image to identify the structural part target and output the boundary box position thereof;

[0039] According to the boundary box position, a weight greater than 1 is given to the target region pixel, and a reference weight 1 is given to the non-target region pixel.

[0040] Preferably, the lightweight deformable 3D convolution cost volume aggregation model comprises:

[0041] A plurality of lightweight deformable aggregation blocks connected in series, the aggregation block adopts a residual connection structure; the aggregation block comprises:

[0042] Firstly, the input feature passes through the deformable depth separable 3D convolution module, which introduces a trainable 3D dynamic offset on the basis of the standard 3D convolution;

[0043] Then, the channel-by-channel 3D convolution, the point-by-point 1x1x1 convolution and the 3D attention module are sequentially executed;

[0044] Finally, the input feature and the output feature of the 3D attention module are fused by element-by-element addition.

[0045] The set of sampling positions of the standard 3D convolution is defined as (i, j, k respectively traverse the range of the convolution kernel in the D, H, W dimensions), and the convolution calculation of the output feature at the position (d, x, y) is

[0046] After introducing the deformable convolution, each sampling point (i, j, k) has a learnable offset Δ(i, j, k), and the convolution calculation becomes:

[0047]

[0048] wherein, is the offset learned at the output position (d, x, y) for the convolution kernel sampling point (i, j, k).

[0049] Preferably, the disparity probability distribution is represented as:

[0050]

[0051] wherein, h, w represent the spatial dimension pixel position, d represents the disparity dimension coordinate, D max represents the maximum range of disparity prediction, and C(h, w, d) represents the optimized three-dimensional cost volume.

[0052] Preferably, the disparity expectation value is represented as:

[0053] wherein, sigma(·) represents an activation function.

[0054] Preferably, the L1 loss function is used to supervise the training of the predicted disparity and the real disparity.

[0055] In a second aspect, the application provides a structure target disparity estimation system fusing a Transformer and a lightweight 3D convolution, for executing the method, comprising:

[0056] a binocular image acquisition module, configured to acquire a left-view image and a right-view image of a structure scene;

[0057] a feature extraction and fusion module, configured to extract two-dimensional basic features of the left-view image and the right-view image, and extract global features, and fuse the two-dimensional basic features and the global features;

[0058] a cost volume construction and weighting module, configured to obtain a weighted 4D cost volume;

[0059] a cost volume aggregation module, configured to obtain an optimized 3D cost volume based on the weighted 4D cost volume;

[0060] a disparity map generation module, configured to generate a continuous disparity map with sub-pixel accuracy.

[0061] The application has the following beneficial effects: the IMSA and GDFN modules are used to extract global context and high-frequency detail features from left and right images, effectively capturing long-distance dependencies in structure image, and enhancing the perception ability of the network to complex texture regions; the Struct-YOLO detection prior and the Transformer feature are spliced to construct a fine cost volume model, and then a lightweight deformable 3D convolution cost aggregation module is used to combine multi-dimensional attention for cost aggregation, realizing efficient processing and accurate aggregation of the cost volume; finally, a loss function is constructed, and a high-precision disparity map is generated through multi-scale supervision, realizing accurate three-dimensional positioning of the structure target.

[0062] Specifically: (1) synergistic enhancement of global perception and local detail preservation ability: by introducing an improved Transformer module (including an improved multi-head self-attention IMSA and a gated dilated convolution feedforward network GDFN), long-distance dependencies and global context information in the image are effectively captured, and the feature extraction ability for edges, textures and other local details is strengthened by using the gating mechanism and the dilated convolution, fundamentally improving the matching robustness of the model in difficult areas such as weak texture and repetitive texture.

[0063] (2) High matching accuracy, more accurate positioning: by fusing the YOLOv5s target detection prior, the cost volume weighting is performed on the target area of the structural part, so that the model pays attention to the key target, and the background interference is effectively suppressed; combined with the disparity map generation method based on probability distribution regression (Soft-argmin), continuous disparity with sub-pixel accuracy can be output, which significantly improves the accuracy of three-dimensional space positioning.

[0064] (3) High algorithm efficiency, good real-time performance: a lightweight deformable 3D convolution cost aggregation model is designed, which combines depth separable convolution and deformable convolution to greatly reduce the computational complexity and memory occupation while realizing adaptive and high-precision cost aggregation, so that the application has better inference speed while maintaining high accuracy, meeting the real-time requirements of industrial sites.

[0065] (4) Strong environmental robustness, high practical value: the fusion strategy used in the application has stronger adaptability to complex environmental factors such as occlusion and light changes. Through verification on public data sets such as Scene Flow, it is shown that the method is superior to mainstream methods in multiple key indicators such as endpoint error (EPE), and has higher practical value and generalization ability in complex industrial scenes. BRIEF DESCRIPTION OF DRAWINGS

[0066] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0067] Among them:

[0068] Figure 1 It is a flowchart of a structural part target disparity estimation method fusing Transformer and lightweight 3D convolution.

[0069] Figure 2 It is a structural diagram of an improved Transformer global information extraction module.

[0070] Figure 3 It is a group distance mapping network structure diagram

[0071] Figure 4 It is a structure diagram of a lightweight deformable 3D convolution cost volume aggregation model.

[0072] Figure 5 It is a Scene Flow data set example.

[0073] Figure 6For different network effects of Scene Flow dataset comparison. DETAILED DESCRIPTION

[0074] In order to make the objects and characteristics of the present application more obvious and easy to understand, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0075] In the following description, a lot of specific details are set forth in order to facilitate a thorough understanding of the present application, but the present application can also be implemented in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the concept of the present application, therefore the present application is not limited to the specific embodiments disclosed below.

[0076] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the present application does not mean the same embodiment, nor is it an embodiment that is separate or alternative to other embodiments.

[0077] Reference Figures 1-5 For the first embodiment of the present application, the embodiment provides a disparity estimation method for structural object fused with Transformer and lightweight 3D convolution, which is based on the left and right view image pairs of the structural scene collected by the binocular camera to collect two-dimensional basic features, designs an improved Transformer global information extraction module to block process and extract long-distance dependent global features of the two-dimensional features; fuses the global Transformer features with YOLO detection priori to construct a 4D cost volume; designs a lightweight deformable 3D convolution cost volume aggregation model to obtain an optimized three-dimensional cost volume model; generates accurate disparity prediction results based on the probability distribution regression method, designs a loss function for optimization constraint, and realizes accurate three-dimensional positioning of the structural object. Finally, through the Scene Flow synthetic dataset, combined with the comparison of different algorithms, the superiority of the present application is verified according to the endpoint error evaluation index.

[0078] As Figure 1 shown, a disparity estimation method for structural object fused with Transformer and lightweight 3D convolution is mainly realized through the following steps:

[0079] Step 1: Transformer-based weight sharing feature extraction

[0080] S1.1, image acquisition and basic feature extraction:

[0081] Firstly, the scene of the structural part is captured by a binocular camera to obtain the left view image and the right view image which have been rectified. Then, the left and right view images are respectively input into a convolutional neural network (CNN) to extract two-dimensional basic features (weight sharing is a feature of CNN itself, which is applied to the left and right view images in this embodiment).

[0082] S1.2, global feature extraction:

[0083] The left and right view Figure Two feature maps obtained in S1.1 are respectively input into an improved Transformer global information extraction module which also shares weights to capture long-distance dependency and local detail features, so as to obtain global features.

[0084] Among them, the improved Transformer global information extraction module is stacked by an improved multi-head self-attention mechanism (IMSA) and a gated dilated feed-forward network (GDFN), the IMSA is mainly used to enhance the network's ability to capture global context, and the GDFN focuses on improving the network's extraction effect of local detail features, both of which work together to improve the network's overall feature expression ability.

[0085] As shown in the figure, the specific implementation of the improved Transformer global information extraction module is as follows: Figure 2

[0086] a) Improved multi-head self-attention mechanism (IMSA):

[0087] a.1) After the given input feature map , first, the features are preprocessed by layer normalization (LN) to speed up network training and improve generalization performance;

[0088] a.2) Then, a 3x3 depth separable convolution with a dilation rate of 2 is used to generate a query matrix (Q), a key matrix (K) and a value matrix (V) respectively; (this dilated convolution is used to expand the receptive field when generating QKV, which improves the spatial local feature capture ability without significantly increasing the parameter amount), the specific expression is as follows:

[0089] Q = Conv d3×3 (LN(X)), K = Conv d3×3 (LN(X)), V = Conv d3×3 (LN(X))

[0090] In the formula, Conv​d3×3 denotes a 3x3 depthwise separable convolution operation with dilation rate of 2;

[0091] a.3) Then, concatenate the Q matrix and the K matrix along the channel dimension, and use a 1x1 convolution to linearly transform the concatenated features to realize high-dimensional information interaction between Q and K;

[0092] a.4) After normalizing the Q matrix and the K matrix by the Softmax function, the attention map is calculated by multiplying the normalized Q matrix and the V matrix, and the expression is as follows:

[0093]

[0094] In the formula, d k denotes the dimension of the input picture, which is 512 in the embodiment;

[0095] a.5) Finally, the Q-K channel interaction information (output of the 1x1 convolution) in a.3) is matrix multiplied with the normalized attention output feature in a.4) to correct and refine the attention feature; the specific expression is as follows:

[0096] X att =Conv 1×1 (Concat(Q,K))⊙Attention(Q,K,V)+K

[0097] wherein Concat(Q,K) denotes the concatenation operation of Q and K along the channel dimension, denotes matrix multiplication, and X att denotes the feature output after the enhanced attention mechanism.

[0098] b) Gated dilated convolution feedforward network (GDFN):

[0099] b.1) First, perform layer normalization on the feature X att output by the IMSA module;

[0100] b.2) The normalized feature is input into two branches in parallel, and each branch uses a 3x3 depthwise separable convolution with a dilation rate of 2, and is respectively denoted as:

[0101] G1=Conv d3×3 (LN(X att )), G2=Conv d3×3 (LN(X att ))

[0102] b.3) apply a Gaussian Error Linear Unit (GELU) activation function to the output features of the first branch as a gating signal;

[0103] b.4) perform matrix point multiplication between the gating signal obtained in b.3) and the output features of the second branch to achieve fine extraction of local detail features and control of information flow;

[0104] Finally, the output of the GDFN, i.e., the output expression of the improved Transformer global information extraction module, is as follows:

[0105] X ffn = GELU(G1) ⊙ G2 + X att

[0106] Step S2: Fusion of target prior and construction of cost volume based on grouping strategy

[0107] S2.1, Feature fusion:

[0108] The two-dimensional basic features extracted in S1.1 are fused with the global features extracted in S1.2 to obtain the final feature map for matching.

[0109] S2.2, Initial cost volume construction (Grouped Distance Mapping, GDM):

[0110] As shown in FIG. 2, based on the fused features of S2.1, an initial 4D cost volume is constructed. In the construction process, a Grouped Distance Mapping (GDM) strategy is adopted: Figure 3

[0111] The fused features of S2.1 (assuming the number of channels is C) are evenly divided into G groups, each containing C / G channels;

[0112] At each disparity d, the matching cost (L1 norm average value within the group) of each group is independently calculated:

[0113]

[0114] wherein, represents the feature vector obtained at pixel (i, j) after feature extraction of the left view image;

[0115] represents the feature vector at pixel (i, j-d) of the right view image; |G k | = C / G is the number of channels per group;

[0116] The matching costs of the G groups are aggregated (averaged) to obtain the initial 4D cost volume C(i, j, d) at disparity d.​

[0117] S2.3, cost volume weighting (fusion of YOLOv5s prior):

[0118] In parallel, the left-view image is processed using a YOLOv5s object detection network to identify the structural object and output its bounding box position;

[0119] According to the bounding box position, a weight mask matrix W(i,j) of the same size as the image is generated: for pixels (i,j) located in the target area, a higher weight greater than 1 is assigned; for non-target area pixels, a reference weight of 1 is assigned.

[0120] The constructed initial 4D cost volume C(i,j,d) obtained in S2.2 is element-wise multiplied with the weight mask matrix W(i,j) to obtain the final weighted 4D cost volume:

[0121] C'(i,j,d) = C(i,j,d) x W(i,j).

[0122] Step S3: cost volume aggregation based on lightweight deformable 3D convolution

[0123] S3.1, design of lightweight deformable 3D convolution cost volume aggregation model:

[0124] As shown in Figure 4 , the core structure of the aggregation model is composed of a series of stacked "lightweight deformable aggregation blocks". Each aggregation block adopts a residual connection structure to ensure deep information flow. On its internal main path, the input features first pass through a deformable depth separable 3D convolution module; to reduce the amount of calculation while achieving adaptive sampling, the module learns an additional trainable 3D dynamic offset at the sampling position of the standard 3D convolution, allowing it to adaptively adjust the sampling grid according to the geometric shape (such as edges) of the structural object in the cost volume, and then use channel-wise 3D convolution (with offset) and point-wise 1x1x1 convolution to complete the depth separable operation to achieve more accurate cost aggregation. Next, the output features of the previous step are sent to a 3D attention module, which recalibrates the channel of the aggregated feature map to enhance effective feature responses and suppress unreliable matches. Finally, to achieve residual connection and avoid gradient vanishing, the input features of the aggregation block are fused with the output features of the 3D attention module through element-wise addition as the final output of the aggregation block.

[0125] S3.2, input the weighted 4D cost volume obtained in S2.3 into a lightweight 3D convolution cost volume aggregation model to regularize the cost volume and aggregate contextual information.

[0126] S3.3, output:

[0127] The polyphase model finally outputs an optimized 3D cost volume C(h, w, d) (dimension: height x width x disparity range).

[0128] Step S4: Disparity map generation based on probability distribution regression

[0129] S4.1, Probability distribution conversion:

[0130] The Soft-argmin (i.e. probability distribution regression) method is used to generate continuous disparity.

[0131] First, the optimized 3D cost volume output by S3 is applied with a Softmax function along the disparity dimension.

[0132]

[0133] where h, w represent spatial dimension pixel positions, d represents disparity dimension coordinates, D max represents the maximum range of disparity prediction;

[0134] S4.2, Disparity expectation value calculation:

[0135] According to the obtained disparity probability distribution, the predicted disparity value D p (h, w) of each spatial position can be represented as the expectation value calculation along the disparity dimension:

[0136]

[0137] where σ(·) represents an activation function;

[0138] S4.3, Output:

[0139] The expectation value D p (h, w) is the final predicted continuous disparity value with sub-pixel accuracy, thereby forming an accurate disparity map.

[0140] In order to train the model, a supervision signal is introduced, i.e. a real disparity map as a target label, and an L1 loss function is used to optimize and constrain the error between the predicted disparity and the real disparity.

[0141]

[0142] where D g (h, w) is the real disparity map, is the predicted disparity map.

[0143] To verify the effectiveness of the proposed algorithm, the network model is deployed on the NVIDIA RTX 3090 GPU platform, and the PyTorch 1.10 deep learning framework is used for training and testing. During training, the Adam optimizer is used, the initial learning rate is set to 1×10 -4 , the batch size is 8, the total training Epoch is 100, and the data augmentation strategy of random cropping, random color disturbance and random horizontal flipping is used.

[0144] Figure 5 is an example of the Scene Flow dataset, which is a large-scale synthetic stereo vision dataset containing left and right view image pairs and pixel-level dense disparity annotations. The dataset consists of multiple subsets (e.g., FlyingThings3D, Monkaa, and Driving), providing approximately 39,000 pairs of synthetic color stereo images with a resolution of about 960×540. Since it is generated by computer graphics, Scene Flow can provide complete and accurate disparity and optical flow ground truth for each pixel, unaffected by occlusion. This feature makes it one of the first large-scale densely annotated datasets for stereo vision and optical flow estimation research. The algorithm effectiveness verification experiment uses this dataset.

[0145] To objectively evaluate the stereo matching performance of the network model proposed in this chapter, the end-point error (EPE) is selected as the main evaluation indicator. The end-point error (EPE) is defined as the average Euclidean distance error between the predicted disparity value and the true disparity value at the pixel level, and its calculation formula is defined as:

[0146]

[0147] where (x1, y1) represents the pixel position corresponding to the predicted disparity value, and (x2, y2) represents the pixel position corresponding to the true disparity value. The EPE indicator can comprehensively measure the difference between the predicted disparity map and the true disparity map, and is a widely used basic evaluation indicator in the field of stereo matching.

[0148] On the Scene Flow dataset, not only the end-point error (EPE) is used for performance measurement, but also the proportion of pixels with an end-point error greater than 1 pixel (1px), 3 pixels (3px), and 5 pixels (5px) is calculated to more accurately depict the model's performance at different error levels. In addition, the root mean square error (RMSE) is used as an auxiliary evaluation indicator to depict the overall error distribution of the predicted disparity map. At the same time, the prediction time (Time) of the model on a single image is also calculated to comprehensively evaluate the real-time performance of the model.

[0149] The method proposed in the application is compared with DispNet, GC-Net and PSMNet on the Scene Flow dataset, and the comparison results are shown in the following table.

[0150] Table 1 Comparison of different methods on Scene Flow test set

[0151]

[0152] The above table lists the quantitative results of each method on the above dataset, including error rate under strict error threshold (>1px, >3px, >5px), average endpoint error / root mean square error, and running time. It can be seen that the proposed method achieves the smallest EPE and error rate on synthetic data. The EPE of the proposed method is 0.80, which is significantly lower than 1.09 of PSMNet and 1.68 of DispNet; the >3px error rate is only 4.5%, which is reduced by about 2.4 percentage points compared with PSMNet. It is worth noting that the Transformer and YOLO prior make the proposed method maintain high precision matching even in dynamic synthetic scenes. Although GA-Net and other methods do not directly report Scene Flow results, their excellent accuracy on KITTI shows that the EPE of such methods on Scene Flow is expected to be about 1 pixel. In summary, the proposed method achieves the current optimal level on Scene Flow, and the inference speed reaches 0.35 seconds per frame, which is better than the inference time of more than 0.5 seconds of some early methods.

[0153] Figure 6 In order to analyze the instance results of some different methods for Scene Flow dataset, the left image of the stereo image pair, GC-Net, PSMNet and the method of the present application are shown from top to bottom. From the results, it can be seen that the present application can also restore the gap between the leaves at a relatively long distance, and can smoothly predict the relatively thin connecting rod, which illustrates the superiority of the proposed method.

Claims

1. A method for disparity estimation of structural targets that integrates Transformer and lightweight 3D convolution, characterized in that, Includes the following steps: Step S1: Global Feature Extraction Based on Transformer Weight Sharing S1.1 Acquire the left and right view images of the structural component scene, and input them into a convolutional neural network to extract two-dimensional basic features; S1.2 Input the two-dimensional basic features of the left and right views into the improved Transformer global information extraction module to extract global features; Step S2: Constructing the cost matrix that integrates target priors and grouping strategies S2.

1. Fuse the two-dimensional basic features extracted in step S1.1 with the global features extracted in step S1.2; S2.2 Based on the fusion features, an initial 4D cost volume is constructed, and a group distance mapping strategy is adopted during the construction process; S2.

3. Use the YOLOv5s object detection network to generate a weight mask matrix of the same size as the left view image; multiply the initial 4D cost volume element-wise with the weight mask matrix to obtain the final weighted 4D cost volume. Step S3: Cost Volume Aggregation Based on Lightweight Deformable 3D Convolution S3.1 Design a lightweight, deformable 3D convolutional cost volume aggregation model S3.2 Input the final weighted 4D cost volume into the designed lightweight deformable 3D convolutional cost volume aggregation model to obtain the optimized three-dimensional cost volume; Step S4: Generation of disparity map based on probability distribution regression S4.1 Apply the Softmax function to the optimized 3D cost volume to obtain the disparity probability distribution; S4.

2. Based on the disparity probability distribution, calculate the expected disparity value of each pixel and generate a continuous disparity map with sub-pixel accuracy.

2. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, The improved Transformer global information extraction module includes: An improved multi-head self-attention mechanism is used to enhance the network's ability to capture global context; And gated dilated convolutional feedforward networks are used to improve the network's ability to extract local detail features; The two work together to enhance the overall feature representation capability of the network; The improved multi-head self-attention mechanism employs a depthwise separable convolution with an expansion rate of 2 when generating the query, key, and value matrices, and introduces a channel interaction mechanism in the attention calculation.

3. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 2, characterized in that, The gated dilated convolutional feedforward network inputs the input features into two branches, each branch employing a depthwise separable convolution with a dilation rate of 2; the output features of one branch are applied with a Gaussian error linear unit activation function and then multiplied by the output features of the other branch.

4. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, Group distance mapping strategies include: The fusion features are evenly divided into multiple groups; Calculate the matching cost for each group independently under each disparity; The initial 4D cost body is constructed by aggregating the matching costs of each group.

5. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, The weight mask matrix is ​​generated as follows: The YOLOv5s object detection network is used on the left view image to identify structural object targets and output their bounding box positions. Pixels in the target region are assigned a weight greater than 1 based on their bounding box position, while pixels in the non-target region are assigned a baseline weight of 1.

6. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, Lightweight deformable 3D convolutional cost volume aggregation models include: Multiple lightweight deformable polymer blocks connected in series, the polymer blocks employing a residual connection structure; each polymer block includes: First, the input features are processed through a deformable depthwise separable 3D convolution module, which introduces trainable 3D dynamic offsets on top of standard 3D convolution. Then, the channel-wise 3D convolution, the point-wise 1x1x1 convolution, and the 3D attention module are executed sequentially. Finally, the input features and the output features of the 3D attention module are fused by element-wise addition.

7. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, The disparity probability distribution is expressed as: Where h,w represents the spatial dimension pixel position, d represents the disparity dimension coordinates, and D max denoted by , where represents the maximum range of disparity prediction, and C(h,w,d) represents the optimized 3D cost volume.

8. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 7, characterized in that, The expected value of disparity is expressed as: Where σ(·) represents the activation function.

9. The disparity estimation method for structural targets that integrates Transformer and lightweight 3D convolution as described in claim 1, characterized in that, The L1 loss function is used to supervise the training of predicted disparity and true disparity.

10. A disparity estimation system for structural targets that integrates Transformer and lightweight 3D convolution, for performing the method according to any one of claims 1-9, characterized in that, include: A binocular image acquisition module is used to acquire left and right view images of the structural component scene; The feature extraction and fusion module is used to extract the two-dimensional basic features of the left and right view images, extract global features, and fuse the two-dimensional basic features and global features. The cost volume construction and weighting module is used to obtain the weighted 4D cost volume; The cost volume aggregation module is used to obtain an optimized 3D cost volume based on the weighted 4D cost volume; The disparity map generation module is used to generate continuous disparity maps with sub-pixel accuracy.

Images