A feature extraction method for stereo matching by combining CNN with transformer

Abstract

The application relates to the technical field of stereo matching, in particular to a feature extraction method for stereo matching by combining CNN and transformer. The steps are as follows: S1: obtaining left and right images required for stereo matching; S2: image enhancement; S3: obtaining local features of the images by using a CNN branch; S4: obtaining long-distance dependency between features by using a transformer branch; S5: fusing features of the CNN branch and the transformer branch by using a feature fusion module; and S6: inputting the fused features into a subsequent stereo matching module. The feature extraction method for stereo matching by combining CNN and transformer can reduce the stereo matching error rate, and compared with simple CNN feature extraction or transformer feature extraction, the improved features have locality and globality, can make the features more robust, the information more abundant, enhance the effect of the subsequent module, and reduce the stereo matching error rate.

Description

Technical Field

[0001] This invention relates to the field of stereo matching technology, and in particular to a feature extraction method for stereo matching using CNN-fused Transformer. Background Technology

[0002] CNN feature extraction is a technique that uses deep learning models such as CNNs to learn feature representations from images. These models can be trained on large-scale datasets to learn high-level features;

[0003] While CNNs can effectively capture local feature information, they are not good at capturing global dependencies, which is not conducive to processing images with no texture or weak texture.

[0004] Transformer feature extraction is a method that divides an image into fixed-size patches, takes the pixel values ​​of each patch as input, and then uses the Transformer's attention mechanism to capture the relationships between the patches.

[0005] In visual transformers, cascaded self-attention modules can capture long-range dependencies, but they encounter difficulties in acquiring local feature information, which weakens the feature details of the image.

[0006] To address this issue, we designed a feature extraction method that fuses CNN and Transformer for stereo matching, providing an alternative technical solution. Summary of the Invention

[0007] Therefore, it is necessary to provide a feature extraction method for stereo matching using CNN-fused Transformer to address the aforementioned technical problems and solve the technical issues raised in the background section.

[0008] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0009] A feature extraction method for stereo matching using CNN-fused Transformer, comprising the following steps:

[0010] S1: Obtain the left and right images required for stereo matching;

[0011] S2: Image enhancement;

[0012] S3: Use CNN branches to obtain local image features;

[0013] S4: Use the transformer branch to obtain long-distance dependencies between features

[0014] S5: Use the feature fusion module to fuse features from the CNN branch and the transformer branch;

[0015] S6: Input the fused features into the subsequent stereo matching module.

[0016] In a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, in step S2, image data enhancement is performed through random offset, and the specific steps are as follows:

[0017] First, obtain the cropping position of the image, and determine the size and position information of the cropped image [(x1,y1),(x2,y2)] using coordinates;

[0018] Shift the cropping position randomly to the right by d∈[1,20] pixels to obtain the cropping position [(x1+d,y1),(x2+d,y2)];

[0019] Increase the value of d in the left disparity map accordingly.

[0020] In a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, in step S3, local image features are obtained using CNN branches, and a convolutional feature extraction module is built using a convolutional neural network.

[0021] As a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, the convolutional feature extraction module includes:

[0022] The first step is a convolutional layer, which obtains the initial feature information of the image;

[0023] Then there is a residual layer containing three residual blocks to obtain deeper feature information.

[0024] As a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, the convolutional feature extraction module further includes:

[0025] A residual layer contains three residual blocks. Compared to the previous residual layer, this layer has double the number of channels and halved the size of the feature map, thus obtaining features at different scales.

[0026] As a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, in step S4, a transformer branch is added to the feature extraction module to obtain the long-distance dependencies between features.

[0027] As a preferred embodiment of the CNN-fused Transformer feature extraction method for stereo matching provided by the present invention, the transformer feature extraction module includes the following structure:

[0028] First, there is an embedding layer that transforms the input image into the required vector sequence;

[0029] Then there is a cascaded encoder layer, each encoder layer consisting of two sub-layers: a self-attention layer and a feedforward neural network layer.

[0030] Finally, there is an output layer that transforms and maps the obtained features.

[0031] As a preferred embodiment of the CNN-Transformer fusion feature extraction method for stereo matching provided by the present invention, in step S5, the features of the CNN branch and the transformer branch are fused using a feature fusion module, and the steps are as follows:

[0032] Convert the CNN branch and the transformer branch to Where C is the number of feature channels, and (H,W) is the size of the feature map;

[0033] The transformer features are Where N is the number of patches, i.e. the number of feature vectors, and O is the feature dimension;

[0034] Mapping the feature dimension O to the channel dimension C yields...

[0035] Mapping the number of feature vectors to two dimensions yields Where N h and N w Let N be the number of patches for row and column divisions, respectively, and let N = N h ×N w ;

[0036] The feature map is obtained by interpolating the transformer feature map.

[0037] The processed transformer features and convolutional features are added together with the same weights to obtain the final features.

[0038] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0039] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0040] This invention provides a feature extraction method for stereo matching that combines CNN and Transformer. Through the improvement of the method, the stereo matching error rate can be reduced. At the same time, compared with simple CNN feature extraction or Transformer feature extraction, the improved extracted features have both local and global characteristics, which can make the features more robust, richer in information, enhance the effect of subsequent modules, and reduce the stereo matching error rate.

[0041] The CNN and Transformer features obtained by the improved method are fused by the CCT module and transformed into the current mainstream feature map format, which can be easily integrated into other stereo matching backbone networks, making it highly portable.

[0042] The features extracted by this method have both local and global characteristics, which greatly reduces the stereo matching error rate; compared with the baseline model of CNN feature extraction, the matching error rate is reduced by 20%. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the feature extraction method for stereo matching using CNN fusion with Transformer according to the present invention;

[0045] Figure 2 This is a schematic diagram of the random offset data augmentation method of the present invention;

[0046] Figure 3 This is a schematic diagram of transformer feature extraction according to the present invention;

[0047] Figure 4 This is a schematic diagram of the CNN feature fusion transformer features of the present invention. Detailed Implementation

[0048] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0049] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0050] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0051] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0052] Reference Figures 1-4 A feature extraction method using CNN-fused Transformer for stereo matching, comprising the following steps:

[0053] Step 1: Obtain the left and right images required for stereo matching; specifically, this can be done by using commonly used public datasets such as KITTI and SceneFlow, which can be downloaded from their respective official websites.

[0054] Step 2, image enhancement, includes color transformation, random cropping, and random offsetting;

[0055] To enhance the model's generalization ability and improve its robustness, data augmentation methods were used during model training. Common data augmentation methods such as color transformations (CTs) and random cropping (RC) are already widely used.

[0056] By using random offsets (RTTs) for data augmentation, it can simulate real-world scenes over a range of disparities, rather than being limited to a single disparity value. This is a realistic data augmentation method, crucial for stereo matching.

[0057] Specifically, RTTs methods are as follows: Figure 2 As shown. First, the cropping position of the image is obtained. The size and position information of the cropped image are determined by using the coordinates of the top left corner (x1, y1) and the bottom right corner (x2, y2). The left image and the disparity map are cropped using the above cropping position [(x1, y1), (x2, y2)]. Then, for the right image, the above cropping position is randomly shifted to the right by d∈[1,20] pixels to obtain the cropping position of the right image [(x1+d, y1), (x2+d, y2)]. Finally, the value in the left disparity map is increased by d accordingly.

[0058] Step 3: Use CNN branches to obtain local image features;

[0059] A common method for feature extraction is convolutional neural networks (CNNs), which can effectively acquire local feature information. A convolutional feature extraction module was built using CNNs.

[0060] The convolutional feature extraction module contains the following structure:

[0061] The first step is a convolutional layer, which obtains the initial feature information of the image;

[0062] Then there is a residual layer, which contains three residual blocks to obtain deeper feature information;

[0063] Finally, there is a residual layer containing three residual blocks. This layer doubles the number of channels and halves the feature map size compared to the previous layer, thus obtaining features at different scales.

[0064] Step 4: Use the transformer branch to obtain long-distance dependencies between features;

[0065] A transformer branch was added to the feature extraction module to obtain long-distance dependencies between features.

[0066] Figure 3 This is a schematic diagram of the feature extraction process. The transformer feature extraction module used contains the following structure:

[0067] First, there is an embedding layer that transforms the input image into the required vector sequence;

[0068] Then there is a cascaded encoder layer, each of which consists of two sub-layers: a self-attention layer and a feedforward neural network layer.

[0069] a. In the self-attention layer, the self-attention mechanism captures the relationships between elements in the sequence by calculating the attention weights between each element and other elements in the input sequence. This allows each element to focus on information from related elements, thus better capturing contextual information.

[0070] b. In the feedforward neural network layer, the layer performs non-linear transformations and feature extraction on the representation of each element. It changes the dimension of the input vector through multiple layers of linear transformations and activation functions, and reduces overfitting through dropout operations.

[0071] Then there is a cascaded encoder layer, each of which consists of two sub-layers: a self-attention layer and a feedforward neural network layer.

[0072] Finally, there is an output layer that transforms and maps the obtained features.

[0073] Similarly, in order to obtain multi-scale features, multiple transformer blocks are constructed by designing different patches to obtain multi-scale transformer features.

[0074] Step 5: Use the feature fusion module to fuse the features of the CNN branch and the transformer branch;

[0075] Because the obtained convolutional features and transformer features differ in form and semantics, a feature fusion module was designed to better fuse these two types of features. To unify the formal differences between these two features, they are uniformly transformed into feature maps, i.e., converted... Where C is the number of feature channels, and (H, W) is the size of the feature map. The transformer feature is... Where N is the number of patches, i.e., the number of feature vectors, and O is the feature dimension. First, the feature dimension O is mapped to the channel dimension C, resulting in... To obtain the feature map, the number of feature vectors is mapped to two dimensions, resulting in... Among them NhN h and NwN w Let N be the number of patches for row and column divisions, respectively, and let N = N h ×N w Then, interpolation operations are used to map the transformer features to obtain the feature map. Finally, the processed transformer features and convolutional features are added together with the same weights to obtain the final features. The feature fusion process is as follows: Figure 4 As shown

[0076] Step 6: Input the fused features into the subsequent stereo matching module.

[0077] By converting features into a common feature map format, the obtained features can be directly passed into subsequent modules without any changes, which is simple to operate and effectively improves the overall performance of the model.

[0078] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A feature extraction method for stereo matching using CNN-fused Transformer, characterized in that, The steps are as follows: S1: Obtain the left and right images required for stereo matching; S2: Image enhancement; S3: Use CNN branches to obtain local image features; S4: Use the transformer branch to obtain long-distance dependencies between features S5: Use the feature fusion module to fuse features from the CNN branch and the transformer branch; S6: Input the fused features into the subsequent stereo matching module; In step S2, image data enhancement is performed through random offset, and the specific steps are as follows: First, obtain the cropping position of the image, and then determine the size and position information of the cropped image using coordinates. ; Randomly shift the cutting position to the right. Pixels, to obtain the cropping position of the right image in step S1. ; Increase the value in the left disparity map accordingly. Numerical value; In step S4, a transformer branch is added to the feature extraction module to obtain long-distance dependencies between features; The transformer feature extraction module contains the following structure: An embedding layer transforms the input image into the desired vector sequence; A cascaded encoder layer, each encoder layer consists of two sub-layers: a self-attention layer and a feedforward neural network layer; An output layer transforms and maps the obtained features; In step S5, the feature fusion module is used to fuse the features of the CNN branch and the transformer branch. The steps are as follows: Convert the CNN branch and the transformer branch to ;in, The number of feature channels, The size of the feature map; The transformer features are ;in This refers to the number of patches, i.e., the number of feature vectors. For feature dimensions; The feature dimensions Mapping to channel dimension ,get ; Mapping the number of feature vectors to two dimensions yields ,in and These are the number of patches for row and column splits, respectively, and they have... ; The transformer feature map Fc∈R is obtained by using interpolation operations. ; The processed transformer features and convolutional features are added together with the same weights to obtain the final features.

2. The feature extraction method for stereo matching using CNN-fused Transformer according to claim 1, characterized in that, In step S3, a CNN branch is used to obtain local image features, and a convolutional feature extraction module is built using a convolutional neural network.

3. The feature extraction method for stereo matching using CNN-fused Transformer according to claim 2, characterized in that, The convolutional feature extraction module includes: The first step is a convolutional layer, which obtains the initial feature information of the image; Then there is a residual layer containing three residual blocks to obtain deeper feature information.

4. The feature extraction method for stereo matching using CNN-fused Transformer according to claim 3, characterized in that, The convolutional feature extraction module also includes: A residual layer contains three residual blocks, doubling the number of channels compared to the previous residual layer, and halving the feature map size, thus obtaining features at different scales.

Images